The teams that ship.
They have a representative test set - sometimes only fifty cases, often a few hundred - covering the realistic distribution of work the agent has to handle. Each case has an expected behaviour, defined in business terms: this email should be classified as “complaint, regulated”; this document should yield these eight extracted fields; this customer question should be answered from policy section 4.2.
They run this test set on every prompt change, every model upgrade, every configuration tweak. They run it on a schedule in production. They watch two things: aggregate quality (overall score, by category, by confidence band), and the specific cases that regressed. They treat a regression on a single case the same way they’d treat a failing unit test.
The teams that don’t.
They prompt and pray. The prompt that worked in the demo gets tweaked by the project lead the night before a stakeholder meeting. The new version is “better” according to whoever last looked at it. Nobody’s quite sure if the change improved anything or just shifted the failure modes. The model provider quietly upgrades the underlying model two months later, and the system silently gets worse without anyone noticing.
In practice.
What an eval harness looks like in practice depends on the work. For classification, it’s straightforward: labels and a confusion matrix. For drafting, it’s more interesting, because you need rubric-based grading, sometimes by humans, sometimes by another model with a carefully constructed prompt, sometimes both. For retrieval, it’s faithfulness-versus-source, citation correctness, and answer quality graded against a reference. For multi-step workflows, it’s all of the above plus end-to-end success rate on a scenario.
The thing that’s nearly universal: build the eval set before you build the agent. Build it from real cases (anonymise where you need to). Get it reviewed by the business owners who’ll judge the live system. And make running it a hard prerequisite for any change to anything in the stack: prompts, models, retrieval configurations, tool schemas, all of it.
A team without evaluations is a team that ships a system and then hopes. Hope is not a strategy in operations.