When agents write code, evals become the quality gate

A reviewer reads a diff an agent wrote, line by line, and it looks fine. The trouble is in what the code does when it runs. Its output is plausible and wrong, or right in the demo and wrong on the rare inputs. Nothing in the diff shows it, so the reviewer has nothing to flag.

Code review exists because writing code is where mistakes enter. So we put a gate right there, another set of eyes on the diff before it ships. When agents write a growing share of the code, that gate stops catching what breaks. The failures live in behaviour, and a line-by-line read can't see behaviour.

So the checkpoint shifts to behaviour. The question becomes whether the output is correct across the inputs that matter. An eval is how you answer it. It's a set of cases run against the output, with a pass mark set in advance, and you don't ship past a failing gate.

Who does the checking is a routing decision, and the rule is cheapest reliable check first. Rules settle shape, format and thresholds. They go first because a rule can read 10,000 outputs and a person can't. A model handles what a rule can't, and it's cheap enough to run on every change. A person comes last, for the calls that set a standard.

Setting the standard is also where agents stop. An agent can run the check and tell you 6 outputs look off. It doesn't set the threshold. It doesn't decide that a failing case is acceptable this once. The check is delegated, and the standard stays with a person.

The sign you've under-invested is confidence that comes from how it felt when someone tried it. Code quality was in that state before code review, and we didn't accept it there.

Deciding what to evaluate, and what threshold counts as good enough, is one of the most valuable calls a builder makes now. The eval suite is where your standard for quality gets written down, and from then on the system enforces it. Start by writing the cases and the pass mark before the first run, and keep the threshold with the person who owns the standard.