Three papers I keep re-reading about inference cost
Paged attention, speculative decoding, and 4-bit weights — what each one actually buys.
Paged attention, speculative decoding, and 4-bit weights — what each one actually buys.
A composite index looked like the obvious fix. It made p99 four times worse for a week before I noticed.