Test the work your AI is meant to do
Build a repeatable test from real questions, expected results and past mistakes. See what changed before customers or colleagues depend on a new answer.
What an evaluation harness actually does
An evaluation harness runs a repeatable set of tasks and compares the results with the expected behavior. It helps the team see which answers improved, which actions failed and whether a change is ready.
Representative test set
Representative cases with expected outcomes, including both high-frequency work and known edge conditions.
Release gate
Thresholds that block deployment when accuracy, latency, or unsafe behavior exceed what the workflow can tolerate.
Operating loop
Review real mistakes and business-rule changes regularly, then add the cases your next release needs to pass.
The four layers of agent evaluation
1. Happy path
- Common case success rate
- Correct action or recommendation
- Expected time and cost envelope
2. Edge cases
- Missing fields and partial context
- Ambiguous instructions or conflicting evidence
- Rare but operationally meaningful exceptions
3. Prohibited behavior
- Blocked actions without approval
- Data handling outside policy
- Improper routing, hallucinated tools, or invented facts
4. Human review sampling
- Blind review of borderline cases
- Reviewer disagreement tracking
- Escalation reasons folded back into the golden set
What to include in a production-ready golden set
- Normal cases that reflect the highest-volume work the workflow will actually see
- Borderline cases where confidence, routing, or retrieval quality might drop
- Prohibited actions that should always trigger a stop, gate, or escalation
- Historical failures and postmortem examples so known regressions stay visible
- Reviewer labels that tie the expected result back to business policy, not taste
Release metrics that matter
- Task success rate
- Unsafe action rate
- Escalation rate
- Reviewer agreement rate
- Latency and cost envelope
Signals that trigger rollback
- Spike in critical errors or policy violations
- Drop in golden-set performance after a model or prompt change
- Material increase in reviewer overrides
- Unexpected tool misuse or route drift
How to keep the harness alive after launch
Per change
Run regressions whenever prompts, tools, policies, routing logic, or model versions change.
Weekly
Review drift signals, sampled cases, operator complaints, and threshold misses with the workflow owner.
Monthly
Refresh the golden set, retire stale cases, add new failure modes, and decide whether the workflow can expand in autonomy.
Evaluation questions teams ask most
What would your next release need to pass?
Use these questions to agree what your test should cover, who reviews uncertain cases and which failures stop a wider rollout.