Agent evaluation before deployment
Test trajectories, side effects and recovery—not just whether the final answer looks correct.
An agent can reach the right result through an unsafe path, or fail gracefully in a way a final-answer score misses. Evaluation has to inspect the work, not only the ending.
Define the unit of success
Begin with a task contract: initial state, allowed resources, required artifact, forbidden actions and completion checks. Outcome accuracy is only one dimension. Add policy compliance, tool correctness, evidence quality, cost, latency and human intervention. A useful evaluator can distinguish a correct answer reached with fabricated evidence from one reached through valid work.
For multi-step jobs, score checkpoints. Did the agent gather the necessary context before acting? Did it verify the target? Did it notice contradictory evidence? Checkpoints reveal where a run went wrong and make failures reproducible. They also show whether a model improvement changes reasoning quality or merely the final wording.