Three papers I keep re-reading about inference cost
Paged attention, speculative decoding, and 4-bit weights — what each one actually buys.
inference
What actually fits on an 8 GB card
The weights fit. The context is what gets you.
Paged attention, speculative decoding, and 4-bit weights — what each one actually buys.
The weights fit. The context is what gets you.