The model was not the problem. The context window was.
I have a 4060 with 8 GB and no intention of buying anything else, so every few months I try to run a local model on it, and every few months I learn the same lesson in a slightly different way. This time I wrote the numbers down before closing the terminal.
A 7B model at Q4_K_M is about 4.9 GB on disk, and roughly the same in VRAM once llama.cpp puts every layer on the GPU with -ngl 99. That leaves a little under 3 GB, minus whatever the desktop is already using. On Windows that is not nothing: a browser with thirty tabs will hold 600–800 MB of VRAM and will not give it back politely.
At 4k context the model generated at 42 tokens/s. At 8k it managed 41. At 16k it either failed to allocate or — worse — kept going at 6 tokens/s, because Windows had quietly moved part of the model into shared memory. There is no error message for that. You just notice the fan and the sudden patience.
The KV cache is the whole story
Weights are a fixed cost. The KV cache grows with every token you keep in context, and for a Llama-3-8B-shaped model the arithmetic is:
32 layers × 8 KV heads × 128 head_dim × 2 (K and V) × 2 bytes = 128 KiB per token
So:
| context | KV cache, fp16 | weights | fits in 8 GB? |
|---|---|---|---|
| 4k | 512 MiB | 4.9 GB | yes, barely |
| 8k | 1 GiB | 4.9 GB | yes, if the browser is closed |
| 16k | 2 GiB | 4.9 GB | no |
| 32k | 4 GiB | 4.9 GB | no |
The table is the post. Everything else is footnotes.
Two flags bought back some room. --flash-attn cuts the temporary attention buffers, and --cache-type-k q8_0 --cache-type-v q8_0 halves the cache itself. With the quantized cache, 16k fits and generation stays around 38 tokens/s. I did not measure long-context recall carefully enough to tell you what the quantization costs, so I will not pretend I did. For reading logs and summarizing a file it has been fine.
The parts that are not about memory
Prompt processing is fast — around 250 tokens/s on this card — so a 3,000-token prompt still costs a dozen seconds before the first output token. That is usually the number people actually feel, not the generation rate.
Batching did not help. -np 4 gave me roughly the same total throughput as one stream, because the card is compute-bound long before it is memory-bound at batch size four. If you have a 4090 or two of them, the calculus changes; on a 4060 it does not.
The other surprise was thermals. The card sits in a small case, and after twenty minutes of generation it settles about 8% lower than the first-minute number. Every benchmark you run in the first minute is optimistic.
What I would tell someone with the same card
8 GB is enough for a 7–8B model at Q4 with 8k context, one user, no browser, no games. It is not enough for 14B, and it is definitely not enough for 32k context unless you quantize the cache and accept that you are now benchmarking a compromise.
The model is the cheap part. The context is the bill.