Writing / Tag / 10 of 68
Evals
Every essay tagged Evals, newest first.
Most eval setups I see answer one question, whether the final output matched, and then stop. The essays here are about everything that question misses: the agent that got the right answer by calling six tools instead of two, the judge that quietly rewards longer answers, the frozen golden set that stopped describing the product weeks ago, the 90% pass rate that hides how often the same input flips.
The position across all of them is that an eval is a release gate, not a report. It has to run in CI, it has to grade the path and not just the destination, and it has to be fed from production traces so it keeps up with what users actually do.
Begin with LLM evals in CI for the basic gate: a golden dataset, assertions, a judge, a blocked merge. Then trajectory evaluation for grading the steps an agent took, and judge bias and calibration so the number you gate on means what you think it means.
October 2026
Your Agent Decides at 0.8 Confidence. You Never Measured 0.8.
A model returns 0.9 and your pipeline treats it as ninety percent sure, so it auto approves, routes, or flags on that number without anyone checking what the number means.Your Evals Are Green and the Product Is Getting Worse
A frozen eval suite stops describing your product within weeks.Your Agent's One Reliability Number Is Hiding the Failures That Matter
A single pass rate on a dashboard tells you an agent is "99% reliable" and tells you nothing about which 1% is on fire.
September 2026
Your Agent Got the Right Answer the Wrong Way
Your eval checks the final answer and goes green.Your Agent Would Rather Guess Than Admit It Doesn't Know
Most agents never say "I don't know." They were trained on benchmarks that reward a confident guess over an honest abstention, so in production they invent a policy, a number, or a citation and say it with a straight face.Your Agent Scores 90% on Evals and Still Fails Customers
A 90% pass@1 eval score feels like a passing grade, and then production hands you a stream of complaints anyway.A New Model Dropped This Week. Don't Just Bump the String.
Four frontier models shipped in the last 72 hours and your instinct is to change one env var and redeploy.