AI Agent Evaluation: How to Test Agents Before They Reach Customers

An AI agent that answers correctly in a demo can still refund the wrong order on a Tuesday afternoon. Agents call tools, read data, and change records, and the same prompt can lead to a different path on every run. That is why a few manual test chats before launch are not enough. You need AI agent evaluation: a repeatable way to measure whether the agent reaches the right outcome, takes acceptable actions along the way, and keeps doing so after every prompt, model, or tool change.

This guide explains how to evaluate AI agents in practice: what to measure, which graders to use, how to build a test set from real traces, and how to turn it into a release gate. The examples use an ecommerce support agent, but the same approach works for internal operations agents, RAG assistants, and agents connected to business systems through MCP.

Why evaluating an AI agent is different from testing normal software

Traditional tests assume determinism: the same input gives the same output, so one passing test means the code works. Agents break that assumption in three ways.

• Outputs vary between runs. The same request can produce different wording, a different tool order, or occasionally a different decision. One successful run tells you very little.
• The path matters, not just the answer. An agent that gives the customer the right reply but called a refund tool twice along the way has failed, even if the final message looks perfect.
• Quality is partly subjective. Whether a reply was grounded in your returns policy, or whether the tone fits your brand, can't always be checked with an equality assertion.

Agents also change for reasons outside your codebase. A model provider updates a model, a teammate edits a system prompt, a tool's response format changes, or a knowledge base article is rewritten. Each of these can shift behavior without a single line of your application code changing. Evaluation is how you notice.

What to measure: outcome, trajectory, and side effects

A useful agent evaluation scores each test case on three levels. LangChain describes the same split in its guide to evaluating agents at the run, trace, and thread level.

1. The final outcome

Did the agent solve the task? For a support agent, that might mean the customer got the correct delivery status, or an eligible return was approved with the right label. Outcome is the metric business owners care about most, and it should carry the most weight.

2. The trajectory

The trajectory is the sequence of steps the agent took: which tools it called, with which arguments, and in what order. Useful trajectory checks include:

• Did it call the right tool with valid arguments, such as an order ID that actually belongs to the signed-in customer?
• Did it avoid forbidden tools for this kind of request?
• Did it take a reasonable number of steps, without loops or repeated identical calls?
• Did it ask for clarification when information was genuinely missing, instead of guessing?

Be careful not to make trajectory checks too rigid. Agents often reach a correct result through a valid path you didn't anticipate. Reserve strict, ordered tool-call matching for sequences that matter for correctness or safety, for example "verify identity before changing the shipping address". LangSmith's documentation on trajectory evaluations covers both strict matching and judge-based scoring.

3. State changes and side effects

For agents that write data, check the resulting state, not just the conversation. After a test run, query the sandbox: was exactly one refund created, for the right amount, on the right order? Was a note added to the ticket? Side-effect checks catch the failures that are most expensive in production.

Alongside these, track operational numbers for every run: latency, token usage, cost, and number of tool calls. A prompt change that improves accuracy slightly but doubles cost per conversation is a business decision, and you can only make it if you measure both.

Choosing graders: code, model judges, and humans

Every evaluation criterion needs a grader. There are three kinds, and good evaluation suites use all of them for different jobs. Anthropic's engineering write-up, Demystifying evals for AI agents, recommends deterministic graders where possible, model graders where necessary, and human graders judiciously for validation.

Code-based checks

Use plain code for anything objectively verifiable: the right tool was called, the output matches a JSON schema, a refund stayed under the allowed limit, an order ID exists, a required disclaimer appears, no email address leaked into a public reply. These checks are fast, cheap, and give the same answer every time, so run them on every test case.

Model-based judges (LLM-as-a-judge)

Use a separate model with a written rubric for semantic questions: was the answer supported by the retrieved policy, did it address what the customer actually asked, was the tone appropriate. A few practices make judges far more trustworthy:

• One question per judge. Grade groundedness, helpfulness, and tone with separate prompts rather than one judge scoring everything at once.
• Prefer pass/fail over 1–10 scales. Binary verdicts with a short written reason are easier to calibrate and compare than fuzzy numeric scores.
• Give the judge an exit. Allow an "unknown" or "not enough information" verdict so it doesn't invent a confident grade.
• Give it the evidence. A groundedness judge needs the retrieved documents and tool results, not just the final reply.

Human review

People remain essential for ambiguous cases, high-stakes decisions, and new kinds of failure. Their most important job, though, is calibrating the model judges. Keep a small set of traces labeled by someone who knows the business, run the judge on the same set, and look closely at every disagreement. Re-check that agreement whenever you change the judge's model or rubric; otherwise you may be gating releases on noise.

Building your first evaluation dataset

You do not need thousands of test cases to start. You need cases that reflect how the agent is really used and where it really fails.

1. Log complete traces. Capture the user input, system prompt version, every tool call and result, the final reply, and cost and latency. The OpenTelemetry semantic conventions for generative AI are a good vendor-neutral starting point for structuring this data.
2. Read the failures yourself. Go through a batch of real or pilot conversations and write down what went wrong in plain language: "quoted the old returns window", "called cancel_order on a shipped order", "didn't ask which of two orders the customer meant".
3. Turn each failure type into test cases. A test case is an input, any starting state the sandbox needs, and explicit success criteria for the outcome, trajectory, and side effects.
4. Add the happy paths and the edge cases. Include the most common requests, ambiguous requests, requests the agent must refuse, and adversarial inputs such as instructions hidden in a product review or email.
5. Write down what "correct" means. If two reasonable people on your team would grade a case differently, the success criteria are not clear enough yet.

A practical example for a Shopify support agent: "Customer asks to change the shipping address on order #1042, which has already been fulfilled." Success criteria might be that the agent looked up the order, recognized it had shipped, did not call the address-update tool, explained the options clearly, and offered to hand over to a person. Two of those are code checks, one is a side-effect check, and one is a judge question.

If the agent answers from your documentation, evaluate retrieval separately too. Our guide to building a RAG knowledge base from internal docs covers how retrieval quality affects the final answer.

Handling non-determinism: pass@k and pass^k

Because agents vary from run to run, run each test case several times and look at two different numbers:

• pass@k is the probability that at least one of k attempts succeeds. It is useful when a person reviews options or the agent can retry safely.
• pass^k is the probability that all k attempts succeed. It was popularized by the τ-bench benchmark for customer-service agents, and it is the stricter measure of consistency.

For anything customer-facing or anything that changes records, pass^k is usually the number that matters. An agent that handles a refund request correctly most of the time still produces a steady stream of mistakes at real traffic volumes. As k rises, the gap between the two numbers shows you how much of your apparent accuracy is luck.

Turning evaluation into a release gate

An evaluation suite is most valuable when it runs automatically. Treat it like the test suite for any other software:

• Run it in CI on every change to prompts, tool definitions, retrieval settings, or model versions. Open-source tools such as promptfoo and MLflow's GenAI evaluation can run test sets and compare versions, and model providers offer their own options such as the OpenAI Evals guide.
• Split the suite in two. A capability suite measures how well the agent handles hard tasks and is expected to improve over time. A regression suite contains cases the agent already handles and should stay close to 100 percent; any drop blocks the release.
• Compare versions side by side. When deciding between two prompts or models, pairwise comparison on the same cases is usually more reliable than comparing two absolute scores.
• Keep sampling production. Run code checks and a sampled set of judge checks on live traffic, and send new failure types back into the dataset. That closes the loop.

Common mistakes in AI agent evaluation

• Only grading the final message. The reply can look fine while the trajectory or the database tells a different story.
• Trusting an uncalibrated judge. A model judge that has never been compared with human labels can have systematic biases, such as favoring longer answers.
• Testing only happy paths. The expensive failures come from ambiguous requests, missing data, and adversarial input. Our article on prompt injection prevention for AI agents explains which attacks belong in your test set.
• Running each case once. A single pass hides inconsistency. Repeat cases and track consistency.
• Testing against production systems. Agents that write data should be evaluated against sandboxes or mocked tools with known starting state, so side effects can be checked and reset.
• Letting the dataset go stale. If your test cases never change, the suite stops reflecting how customers actually use the agent.

Practical recommendations

If you are starting from zero, keep it small and real:

1. Pick the one workflow where a mistake costs the most, such as refunds, order changes, or quotes.
2. Write a few dozen test cases from real conversations, each with explicit success criteria.
3. Automate the code checks first, then add one or two model judges for the questions code can't answer.
4. Label a small calibration set by hand and confirm the judges agree with it.
5. Run the suite several times per case in CI and block releases on regressions.
6. Review a sample of production traces every week and add new failure types to the dataset.

Agents built on MCP or other tool protocols benefit from the same discipline at the tool layer. Clear tool descriptions and strict input validation make agents easier to evaluate, which we cover in our guides to AI agents for ecommerce with MCP and building a remote MCP server for SaaS.

Frequently asked questions

What is AI agent evaluation?

AI agent evaluation is the practice of testing an agent against a set of realistic tasks with defined success criteria, and grading its final outcome, the steps it took, and any changes it made to data. It is repeated whenever the agent, its model, or its tools change.

How many test cases does an AI agent evaluation need?

Enough to cover the agent's main workflows and known failure types. Many teams start with a few dozen well-chosen cases drawn from real usage and grow the set as new failures appear. Coverage of real behavior matters more than raw count.

Is LLM-as-a-judge reliable?

It can be, when each judge answers one clear rubric question, returns pass/fail with a reason, sees the relevant evidence, and has been checked against human labels. Use code instead wherever the answer can be verified deterministically.

What is the difference between pass@k and pass^k?

pass@k measures whether at least one of k attempts succeeds, while pass^k measures whether all k succeed. pass^k is the better measure of consistency for agents that serve customers or change records.

Should evaluation stop once the agent is in production?

No. Production traffic shows failure types that pre-launch tests missed. Sampling live traces and feeding new failures back into the test set is what keeps an agent reliable over time.

Next steps

Agents become dependable when their behavior is measured the same way every time, not when the prompt finally looks right. Start with one high-risk workflow, write down what correct looks like, and make the evaluation suite part of every release.

If you are planning an AI agent for support, operations, or your Shopify store and want it built with evaluation and guardrails from day one, talk to CodeSapient about AI agent development or explore our software development services.