I spent a week doing capacity planning for a service that spends most of its money on generated tokens, and reread three papers that keep coming up. These are notes for myself, not a survey.
PagedAttention (Kwon et al., 2023)
The observation is simple: a serving engine allocates KV cache in contiguous blocks sized for the worst case, so most of the memory is reserved and empty. The fix borrows virtual memory — split the cache into fixed-size blocks, keep a block table per sequence, allocate on demand.
What it buys is throughput, not latency: two to four times more concurrent sequences at the same footprint, because you stop wasting memory on reservations. What it costs is a layer of indirection in the attention kernel and a scheduler that has to think about block tables.
The part I keep coming back to is the framing. Batching is a memory management problem before it is a compute problem. If you are tuning batch size by watching GPU utilization, you are already downstream of the real constraint.
Speculative decoding (Leviathan et al., 2023)
A small draft model proposes k tokens, the big model verifies all of them in one forward pass, and rejection sampling keeps the output distribution identical to the big model alone. When the draft is right, you get several tokens for the price of one.
The catch is in the acceptance rate. On summarization and code completion, where the next token is often predictable, I have seen 2–3× on greedy decoding. At temperature 1 with open-ended prompts, acceptance drops and the gain mostly evaporates — you pay for the draft and get little back. It is a latency optimization for workloads with low entropy, and a trap for workloads without it.
Also: the draft model has to be genuinely cheap. If it takes a third of the time of the target, you need an acceptance rate above roughly 50% just to break even.
AWQ (Lin et al., 2023)
Weight-only 4-bit quantization. The insight is that not all weights matter equally — protecting the ~1% of salient channels, chosen by activation magnitude rather than weight magnitude, keeps quality much closer to fp16 than naive rounding.
For single-stream decoding this is the highest-leverage change available, because decode is memory-bandwidth-bound: every token requires reading the entire weight matrix, and halving the bytes roughly doubles the tokens per second. For batched prefill, where the math is compute-bound, it does much less.
The failure mode is calibration. Quantize with a calibration set that looks nothing like your traffic and the damage shows up in the long tail — rare intents, unusual formatting, non-English input — not on the benchmark you ran.
The model I use now
Prefill is compute-bound. Decode is memory-bandwidth-bound. Every optimization above attacks one of those two facts, and none of them creates throughput out of nothing.
Our mix is about 80% short prompts under 400 tokens, so batching and paged cache did most of the work; quantization helped the single-stream tail. If your mix were long documents and short outputs, the ranking would be different. Papers are cheap. Measuring your own mix is the expensive part.