The Never-Ending Eval

Someone asked me last week how they should "add evals" to their agent product. It's a completely reasonable sentence and I've said it myself, but it's become about as specific as "add tests" — it can mean four different things depending on which year of AI the speaker is standing in.

Because the thing being evaluated keeps changing shape. Five years ago it was a string. Today, at the frontier of what people are actually shipping, it's a business process that runs for forty minutes, calls nine tools, reads a company's internal policy document, hands a draft to a human, and then waits six weeks for the physical world to reveal whether it was right.

You cannot grade that with a test set. And nobody has fully worked out what you grade it with instead — which is, I think, the most interesting open problem in applied AI right now. Here's where the thinking is, how it got here, and what "tuning" even means when the system is this shape.

Four eras of the eval

The useful way to track this isn't by technique. It's by asking: what is the unit being evaluated? Every time that answer has changed, the whole toolkit has had to be thrown out and rebuilt.

What's actually in flight on one agent run

Take a concrete one. An agent decides how much of a product to reorder from a supplier. Here is the full list of things that can be the reason it was wrong:

• Input freshness. Was the sales data complete through yesterday, or did a sync stall on Thursday?
• Input completeness. Did it see all three warehouses or two of them?
• Retrieval. Did it pull the right rule out of the company's policy doc — and only the relevant ones?
• Tool choice. Of the eleven tools available, did it pick the right one?
• Tool truthfulness. Did the tool return the truth? Tools lie, quietly, more often than models do.
• Model and version. Which one ran, and at what settings, and was that the same one that ran last week?
• Decomposition. Did it break the job into the right steps, or invent a shortcut?
• Sequencing and gates. Did a step run on inputs it should have refused?
• The human edit. What did the approver change before sending, and why?
• The world. Six weeks later: did it stock out, or sit in a warehouse accruing fees?

Ten variables. One output. And the failure mode that matters most — a confident, well-formatted, wrong number — throws no error and looks exactly like a success until the calendar catches up with it.

The unit of evaluation is a step with a contract

The only method I've found that survives contact with a real multi-step system is boring, and it starts before any model is involved: decompose the process into named steps, and give every step a contract.

A step isn't a prompt. It's a declared thing with six fields:

1. Entry condition — what must be true before this is allowed to run at all.
2. Inputs, with a freshness requirement — not "sales data" but "order history complete through T-1, no gaps over six hours." Freshness as a contract, not an assumption.
3. Permitted evidence — what this step may reason from. A step that can reach anything cannot be diagnosed when it's wrong.
4. The assertion it produces — one falsifiable, typed claim. Not prose.
5. The exit gate — what must hold for this to pass downstream. Hard gates stop the run; soft gates continue with degraded confidence and a flag.
6. Who owns ground truth — arithmetic, a human, or the world six weeks from now. This is the field everyone forgets, and it's the one that determines everything about how the step can be graded.

Do this and something quietly enormous happens to your bug reports.

The second thing that falls out is that most steps turn out not to need a model at all. In a typical process I'd expect over half the steps to be deterministic — arithmetic, constraint checks, completeness checks — and deterministic steps are testable the old, boring, permanent way. The judgment steps, where a language model genuinely earns its seat, are usually two or three out of ten. Those are the ones that deserve expensive evaluation. Knowing which is which is the decomposition.

Chain math, and why your suite goes quiet exactly when you need it

Ten steps at 95% accuracy each is a 60% run. Twenty steps is 36%.

This isn't an argument against long chains — long chains are the whole point, they're what makes agents worth more than autocomplete. It's an argument for knowing which chain you're in, as a number rather than a feeling. Once you can see it, you have three real moves: shorten the chain, make a step deterministic, or put a human at the weakest link. Without per-step scores you can't tell which move you need, so you reach for the fourth one, which is rewriting the prompt and hoping.

And there's a sharper version of the problem. Per-step accuracy is correlated in ways the multiplication hides — a stale input poisons every downstream step at once, so a bad day isn't 60%, it's a cluster of total failures that all look like different bugs. This is why input-freshness gates repay their cost faster than any model work you will ever do.

Three clocks of truth

Ground truth arrives at three different speeds, and a real evaluation system needs all three. Most teams build exactly one of them — usually the fast one, because it's the one that fits in CI.

Deterministic truth (seconds). Did it parse. Did it respect the stated constraint. Did the arithmetic hold. Was the data fresh. Did it cite a rule that actually exists. Unglamorous, catches most regressions, cheap forever. Nobody has ever regretted building these, and roughly seventy percent of a healthy suite should be here.

Human truth (hours). The approver's diff. If a human reviews the agent's work before it ships — and in any consequential domain one does — then every difference between what was drafted and what was sent is a free labeled example. Most products are already collecting this and throwing it away.

The critical detail: capture the edit and a one-tap reason. An edit with no reason is noise; an edit with a reason code is a test case. And the capture has to cost the reviewer nothing — if logging a correction takes thirty seconds, you will get no corrections, and you'll conclude your system is doing great.

World truth (weeks). Did the thing that was predicted happen. This is the only clock that pays the bills and the only one you can't summon on demand.

The prediction ledger

Slow truth can't be handled with a test suite, because a test suite is a photograph and this is a process with a clock in it. What it needs is an accounting structure.

Every consequential decision writes a dated, falsifiable claim at the moment it's made — "this will not run out before October 12 at this quantity" — and a settlement job scores it when reality arrives. Append-only. A correction is a new entry, never an edit.

That last constraint is the whole thing. The common failure in "outcome evals" is grading in hindsight with a model that already knows how it turned out, which produces numbers that are both excellent and meaningless. A ledger written before the fact can't cheat. It's the same discipline as a forecaster publishing a probability with a timestamp, and for the same reason.

The compounding property is worth noticing: a ledger makes your product's own telemetry into your eval set. It grows per customer, it grows while you sleep, and it's the one asset in this whole stack a competitor cannot buy or copy — it only accrues to whoever was in the loop when the decisions were actually made.

Grade the trajectory, not just the endpoint

Long runs fail by wandering long before they fail by being wrong — and wandering is visible immediately, while wrongness takes six weeks. So score the path: how many steps, how many tool calls, did it call the same tool four times, did it go fetch data it already had, did it revisit a decision it had already made.

Path metrics are cheap, they're available in real time, and they're a leading indicator for both of the expensive failures: a run that wanders is usually about to be wrong, and it's definitely about to be expensive. Of all the things on this list, trajectory logging has the best ratio of insight to engineering effort, and it's the one I see skipped most often.

Two-axis judging

When a model does judge another model's work, the single most useful structural change is to split the question in two:

• Was the evidence any good? Complete, fresh, relevant, sufficient to decide.
• Was the judgment any good, given that evidence?

These fail for completely different reasons and have completely different fixes, and collapsing them into one score makes the result unactionable — you learn that something is wrong but not which team should care. In my experience a large share of apparent model failures are evidence failures wearing a model-failure costume: a stale sync, a missing source, a retrieval that pulled the wrong policy. If your judge can't tell you that, it's telling you to retrain a model that was reasoning perfectly well about garbage.

The corollary rule, which I'd defend harder than anything else in this post: every AI-produced number ships with an explainable chain, or it doesn't ship. Provenance is usually framed as a trust feature for customers, and it is. But it's also the debugging surface and the eval trace. You build it once and get three things. Skip it and you get a system that can't be improved, only replaced.

So what does "tuning" even mean now?

Here's the part that surprised me most as these systems got bigger. When a process-shaped agent is underperforming, the fix is very rarely a prompt change. The moves that actually work, roughly in order of how often I reach for them:

1. Move a gate. A step was running on inputs it should have refused. This is the most common real fix and it involves no AI at all.
2. Make a judgment step deterministic. Half of what gets handed to a model is arithmetic wearing a sentence.
3. Split a step. A step that's wrong 12% of the time is usually two steps, one of which is wrong 24% of the time and one of which is fine.
4. Shorten the chain — or accept the chain and put a human at the weak link.
5. Change who owns ground truth. Sometimes the fix is realizing a step can't be graded by the world in any reasonable timeframe, so it needs a human proxy.
6. Fix retrieval. Boring, measurable to two decimals, and routinely the highest-ROI item on the list.
7. And then, finally, the prompt or the model.

Six structural moves before you get to the thing everybody starts with. That inversion is really the whole argument: you are not tuning a model, you're tuning a process graph. The model is one node in it, and usually not the broken one.

Which also explains why replay is the highest-leverage piece of infrastructure to build early. Record every run as a fixture; when a new model version lands, replay five hundred real historical runs against it before it touches a customer. Given the rate the frontier is moving, "should we move to the new model" is a question you'll answer every few weeks for the rest of the product's life. Make it a ten-minute test rather than a debate and you've bought yourself years.

Era four: the eval becomes the product

The direction this goes, I think, is that the scoring layer stops being internal tooling and surfaces as the thing customers actually buy.

Three ways that shows up. First, a "how your system did" report — decisions made, approved versus edited, predictions settled, money saved — which is the honest answer to the question every AI product gets at month four: is this doing anything? Second, the customer's own documented process becomes measurable: which rules fired, which have never once been retrieved, which ones the human overrode six times. Telling a customer their process is wrong, with receipts, is a genuinely new product category and it requires the eval layer to exist first.

Third, and most importantly: the score becomes the autonomy dial. Per step, per customer, measured confidence decides what runs unattended and what waits for a human. You never have to promise autonomy or argue about it. It's earned visibly, in bands, and the customer watches the dial move. That's a better product, a cleaner pricing ladder, and a far more honest story than "trust us, it's very capable."

Why it's never-ending

Every part of this decays. The models change underneath you every few weeks. The customer's process drifts. Seasonality invalidates last year's fixtures. New tools change the decomposition. The golden set you hand-labeled in March encodes an assumption that stopped being true in July.

So an eval suite that's "done" is an eval suite measuring last quarter's system. The deliverable was never a set of tests — it's a loop: every human correction becomes a labeled example, every consequential decision becomes a dated claim, every claim settles, every settlement updates what you trust enough to automate.

That's the whole thing. Not a test suite you finish. A ledger you keep.