Three papers I keep re-reading about inference cost
Paged attention, speculative decoding, and 4-bit weights — what each one actually buys.
Posts
Notes, mostly written for future me.
Reading TCP socket states without netstat
netstat is missing from most minimal images. Everything it knew is still in /proc.
What actually fits on an 8 GB card
The weights fit. The context is what gets you.
The eval set was in the training data
0.92 on the holdout, 0.71 in production. The holdout was the problem.
A cron job that ran twice
Nothing in the logs said overlap. The second copy just started, and both wrote the same file.
When a Postgres index made the query slower
A composite index looked like the obvious fix. It made p99 four times worse for a week before I noticed.