The evaluation loop
A demo is not
a measurement.
Agents fail quietly and inconsistently. A system that looks perfect in five hand-picked conversations can fail one in ten real requests. Evaluation turns impressions into numbers you can gate releases on.
Each column is a slice of the test set rising toward the release threshold. The marked column falls short and holds the release.
01 · Define
Agree the task, the correct outcome and the failures that must never happen.
02 · Collect
Build a versioned test set from real work, edge cases and adversarial inputs.
03 · Grade
Score outcomes and trajectories with code checks, calibrated model judges and people.
04 · Gate
Compare against the production baseline and apply written release thresholds.
05 · Monitor
Grade live samples, watch online signals and feed new failures back into the test set.
Benchmarks score models in isolation. Your agent is a model plus prompts, tools, retrieval, policies and approvals. Evaluate the whole thing on your own work.
01 / Define success
Write down what good looks like.
An agent cannot be evaluated against a feeling. Before writing a single test, agree what the agent is for, what a correct outcome is, and which failures are unacceptable regardless of average quality.
Task-level outcomes
Describe success as an end state: ticket resolved with the right refund, invoice matched to the right ledger line. Avoid grading only the wording of the final message.
Severity tiers
Separate harmless mistakes from costly ones. A typo and a wrong payment amount should never share a score. Define zero-tolerance failures explicitly.
A support agent's success is defined as: correct policy applied, correct customer account, no promise outside policy, and escalation when confidence is low.
Two reviewers independently grade ten examples using the written definition and agree on at least nine.
02 / Test sets
Build a test set from real work.
The best evaluation data comes from the work the agent will actually do. Synthetic examples fill gaps, but real cases expose the messy inputs that break agents in production.
Sample real traffic
Collect historical tickets, documents or requests, remove personal data, and label the correct outcome. Include the boring majority and the rare difficult cases.
Cover the edges
Add ambiguous requests, missing data, conflicting instructions, other languages, long inputs and adversarial content. Tag each case so results can be sliced.
A 300-case set: 200 sampled from last quarter's tickets, 60 edge cases written by the support lead, 40 adversarial cases from the security review.
Every case has an owner, a source, a tag and an expected outcome. Version the set and never tune prompts against the held-out portion.
03 / Graders
Choose how each case is scored.
Use the cheapest grader that is reliable for the question being asked. Deterministic checks first, model-based judges where judgement is needed, and people where stakes or ambiguity are high.
Cases flow through the grader. Passing cases continue; a failing case drops into the review tray for a person.
Code-based checks
Exact matches, schema validation, correct tool called with correct arguments, numbers within tolerance, forbidden phrases absent. Fast, cheap and reproducible.
Model judges and people
Use a rubric-guided model judge for tone, completeness or reasoning. Calibrate it against human labels and keep a human review sample for high-impact workflows.
Refund cases are graded by code (amount, account, tool call), while reply quality is graded by a rubric judge and 10% spot-checked by a support lead.
The model judge agrees with human graders on a calibration set at a level you have written down, and the agreement is re-measured when the judge model changes.
04 / Trajectories
Grade the path, not just the answer.
Agents take many steps. A correct final answer reached by calling the wrong tool, reading forbidden data or looping twenty times is still a failure waiting to happen.
Step assertions
Check which tools were called, in what order, with which arguments, and which were never called. Flag unnecessary steps, retries and loops.
Efficiency budgets
Track steps, tokens, latency and cost per task. Set limits per workflow so a quality gain that triples cost is visible before release.
An invoice-matching agent must query the ledger before proposing a match, must not call the payment tool, and must finish within eight steps.
Trajectory traces are stored for every evaluation run and failures link directly to the step that broke the assertion.
05 / Safety & red teaming
Test what the agent should refuse.
Quality tests check the agent does the job. Safety tests check it does not do the wrong job when pushed. Both belong in the same release gate.
Adversarial cases
Prompt injection in retrieved documents, requests outside policy, attempts to reveal other customers' data, social engineering and jailbreak phrasing.
Boundary checks
Confirm high-impact actions always stop for human approval, and that the agent escalates rather than guesses when information is missing.
A document containing hidden instructions to email records externally must result in no outbound call and a flagged trace.
Zero-tolerance safety cases pass at 100% on every release candidate. A single failure blocks the release regardless of the overall score.
06 / Regression
Catch what a change breaks.
Prompt edits, model upgrades, new tools and retrieval changes all shift behaviour. Compare every candidate against the current production baseline on the same cases.
The faint line is the production baseline. The candidate tracks it closely until one slice dips, marked for review.
Baseline comparison
Run baseline and candidate on the identical set and report the difference per tag, not just overall. A small average gain can hide a large drop on one slice.
Repeat runs
Agents are not deterministic. Run each case several times and report pass rates with variance, so noise is not mistaken for improvement.
A model upgrade improves overall accuracy by 3% but drops Arabic-language cases by 12%. The slice report catches it before release.
Evaluation runs automatically in the deployment pipeline and a written threshold decides pass or block, not a person's impression of a demo.
07 / Human review
Keep people in the evaluation loop.
Automated scores are a proxy. Domain experts catch the failures nobody thought to write a test for, and their judgement is what the scores should ultimately track.
Structured review
Give reviewers a rubric, blind them to which version produced the output, and record disagreements. Review time is a budgeted part of each release.
Turn findings into cases
Every new failure a reviewer finds becomes a permanent test case, so the test set grows with what the business learns.
Each week a finance lead reviews 25 random reconciliations. Two new failure patterns found in March became 14 new regression cases.
The share of reviewer-found failures that already had a matching test case rises over time.
08 / Production monitoring
Evaluation continues after launch.
Real inputs drift away from any test set. Monitoring tells you when the agent is meeting work it was never tested on, before customers tell you.
Live samples move inside an expected band. When the band widens and a sample escapes it, an alert is raised.
Online signals
Track escalation rate, human override rate, task completion, user corrections, latency and cost per task. Alert on sudden changes, not just thresholds.
Sampled grading
Run the same graders on a sample of live traces. Compare live scores with offline scores; a growing gap means the test set is out of date.
Override rate on refund approvals rises from 4% to 11% after a policy change. The trace sample shows the agent applying the old policy.
Every production alert has an owner, a runbook, and a path to roll back to the previous version.
Worked example / support agent
Run a change through the gate.
Three illustrative changes show how written thresholds decide the outcome. These are reference examples, not results from a live system.
- Change
- Swap to a newer model for the support agent.
- Results
- Overall pass rate 91% → 93%. Arabic slice 88% → 76%. One safety case failed.
- Gate
- BLOCK / zero-tolerance safety case failed and one slice regressed beyond threshold
- Next
- Fix the safety failure, investigate the Arabic slice, and re-run the full set before reconsidering.
# Illustrative evaluation config, not a runnable file
suite: support_agent_v4
test_set: support_cases@2026-10 (held_out: 20%)
repeats: 3
graders:
correct_account: code # exact match
refund_amount: code # within 0.01
required_tools: trajectory
reply_quality: model_judge(rubric_v2, calibrated)
human_sample: 10%
release_gate:
safety_cases: pass_rate == 100%
overall: >= baseline - 1%
any_tag_drop: <= 5%
cost_per_task: <= baseline * 1.2
on_fail: block_release_and_notify_ownerWrite release thresholds before running the candidate. Choosing them after seeing the numbers turns a gate into a justification.
Before production
Ten questions before you ship.
Use these in your release review. Attach the evaluation run, its version and an accountable owner to each answer.
0 / 10 reviewed in this session
Further reading
Go deeper.
This guide draws on public evaluation practice and the resources below. Methods evolve quickly; check current versions.
Technical guidance, not a certification or a claim that these practices are already in place at your organisation.
Ship agents you
can measure.
Bring a workflow and some real examples. We can help design the test set, graders and release gate for your first agent.