The eval set was in the training data

0.92 on the holdout, 0.71 in production. The holdout was the problem.

May 9, 2026 · 3 min · 619 words · mn