How to evaluate AI agents in production.

Production is where an agent meets the ambiguity, tool failures, user expectations, and edge cases that a curated evaluation set cannot fully represent. Production evaluation turns those traces into evidence without treating every observation as a verdict.

When this helps

Use it when an agent is already serving users and your team needs to understand reliability, quality, cost, latency, or behavior changes across real workflows.

Start with bounded, privacy-safe traces

Collect the smallest evidence needed to understand a behavior. Normalize provider data, redact sensitive content, preserve trace relationships, and prefer semantic summaries or IDs over raw prompts and responses.

Sample by risk, not just volume

The most common workflow is not always the most important one. Keep a broad sample for baseline health, then deliberately inspect high-value transactions, policy exceptions, tool failures, repeat attempts, and workflows where users cannot easily detect a wrong answer.

Document the sampling rule. Without it, a production score can look precise while quietly excluding the cases that would change the release decision.

Measure coverage before interpreting results

Separate eligible traces from assessed traces. Track missing identity, unknown outcomes, analysis errors, and unsupported paths so a high score cannot hide a narrow sample.

Compare against the agent’s own history

Use adjacent, time-scoped windows or a clearly defined baseline. A production metric is actionable when the team can see what moved, where it moved, and whether the change is meaningful for the affected workflow.

Separate observation from judgment

Production evidence tells you what users exercised and what the system did in those cases. It does not, by itself, tell you whether the behavior was intended or whether an unobserved path is safe. Keep the observed trace, the evaluation judgment, and the release recommendation as separate parts of the review.

Read next

For the distinction between runtime evidence and test-set judgment, read observability vs evaluation. For release review, use the AI agent release checklist.