Evals are the release gate
A 600-case regression set, model rubrics calibrated against human labels, and 23 regressions caught before they shipped.
Context: a customer-facing agent with 13+ tools, memory and scheduled actions, serving many tenants.
Problem: prompt and model changes that looked fine in a demo broke real conversations, and nobody could say which change did it.
Action: we built a 600-case regression set from production conversations, ran every prompt, model and tool change against it before release, fed each week's failures and handoffs back into it, and graded with model rubrics calibrated against human labels (0.81 agreement). Models are picked per task from the same sets on quality, latency and cost.
Outcome: 23 regressions caught before they shipped in the first quarter; answer accuracy up 31%; swapping a model became a routine change.
Lesson: "better" means nothing about an agent until an eval says so.