Eval Strategies for Agentic AI
An AI agent works in a loop. It reasons, calls tools, reads the results and keeps going until it decides it is done. Evaluating one means checking more than the final message. This post covers what an eval is, how graders work, how to decide what to test, how to run the tests and look inside a run, how different kinds of agents are evaluated, and how to keep evals useful over time.
Table of Contents
What an eval is
An eval is a test for an AI system. An input goes in, and grading logic is applied to the output to measure success. The focus here is on automated evals, which can run during development without real users. See Anthropic's explanation of agent evaluations.
The simplest form is a single-turn eval: a prompt, a response and grading logic. For earlier LLMs, this was the main way to evaluate. As models became more capable, multi-turn evals grew more common.
Agent evals are more complex. Agents use tools across many turns, change the state of their environment and adapt as they go, so a mistake early in a run can carry through and compound. Capable models can also find solutions that a fixed eval did not expect. In one τ²-bench problem about booking a flight, a model found a loophole in the policy. It failed the eval as written, but gave the user a better result.
The traditional approach to LLM evaluation has three steps: build a dataset of examples, write an evaluator, then run the application over the dataset and score the outputs. Every example is treated the same way. Agents break that pattern, because more than the final message needs testing, and the success criteria can be specific to each example, with checks on the agent's trajectory and state. This distinction is also explored in LangChain's evaluation of deep agents.
Manual testing, dogfooding and intuition can go a long way early in development. The breaking point often comes once an agent is in production and scaling, when users report that it feels worse after a change and the only way to verify is to guess and check. Without evals, debugging is reactive: wait for complaints, reproduce by hand, fix, and hope nothing else regressed.
Once evals exist, baselines come with them, since latency, token use, cost per task and error rates can be tracked on a fixed set of tasks. Evals also affect how quickly a new model can be adopted, because a team with evals can see a new model's strengths and adjust in days instead of weeks.
Checking what the agent did, not what it said
Two terms matter most here. A transcript (also called a trace or trajectory) is the complete record of a run: outputs, tool calls, reasoning and intermediate results. The outcome is the final state of the environment when the run ends.
The two can disagree. A flight-booking agent might end its transcript with "Your flight has been booked," but the outcome is whether a reservation exists in the environment's database. The same applies to an order-cancellation agent. Useful checks include whether it selected the correct order and whether it waited for the cancellation tool to succeed before telling the user the order was canceled.
For this reason, domain experts reviewing an agent are better served by questions about results than about mechanics. "Has an appointment been made?" says more than "Did the tool call succeed?" Outcomes are checked wherever possible, and the transcript is used to understand how the agent got there.
Who does the checking
A few more terms are needed. A task (also called a problem or test case) is a single test with defined inputs and success criteria. Each attempt at a task is a trial. Because model outputs vary between runs, several trials are run per task. A grader is logic that scores some aspect of the agent's performance. A task can have several graders, and each grader can contain several assertions, sometimes called checks.
Graders come in three kinds, and most evals combine them. Each can look at the transcript, the outcome, or both.
Kind | Methods | Strengths | Weaknesses |
Code-based | String match, binary tests, static analysis, outcome verification, tool call verification, transcript analysis | Fast, cheap, objective, reproducible, easy to debug | Brittle to valid variations, little nuance, limited for subjective tasks |
Model-based | Rubric scoring, natural language assertions, pairwise comparison, reference-based evaluation, multi-judge consensus | Flexible, scalable, handles open-ended tasks | Non-deterministic, costs more than code, needs calibration with human graders |
Human | Expert review, crowdsourced judgment, spot checks, A/B testing, inter-annotator agreement | Gold standard quality, matches expert judgment, calibrates model graders | Expensive, slow, often needs experts at scale |
For each task, scoring can be weighted (the combined grader scores must reach a threshold), binary (all graders must pass), or a hybrid. Deterministic graders are chosen where possible, model graders where needed or for flexibility, and human graders sparingly.
A few habits make graders more dependable:
Grade what the agent produced, not the path it took. Checking for a very specific sequence of tool calls tends to give brittle tests, because agents regularly find valid approaches the eval designer did not anticipate.
Build in partial credit for tasks with several parts. A support agent that correctly identifies the problem and verifies the customer but fails to process a refund is meaningfully better than one that fails immediately.
Give a model grader a way out. An instruction to return "Unknown" when information is missing helps avoid made-up verdicts. Structured rubrics with each dimension graded by its own isolated judge work better than one judge grading everything.
On the form of scores, pass/fail is preferred over 1-to-5 ratings. The difference between adjacent points such as 3 and 4 is subjective and inconsistent across reviewers, detecting differences takes more data, and reviewers tend to drift toward the middle. A binary label forces a decision. To track gradual progress, a score can be split into small binary checks, such as "4 out of 5 expected facts included".
Many binary checks can be combined into one number with a pass-all rate: an example passes only if it passes every check, and the report can be broken down by check. Complicated weighted scores are discouraged, since a composite that moves from 3.2 to 3.7 does not show what improved for users. For more on this approach, see AI Evals: Everything You Need to Know.
Deciding what to test
Checking is only useful if the right things are checked. The consistent advice is to start from real data.
Error analysis is described as the most important activity in evals, because it determines which evals are worth writing at all. It runs in four steps:
Create a dataset. Representative traces of user interactions are gathered. Without data, synthetic data can fill the gap.
Open coding. A reviewer reads traces and writes open-ended notes about what looks wrong, much like journaling. At least 30 traces are annotated by hand before any suggestions from an agent are reviewed. At the start, only the first failure in a trace is noted, since upstream errors cause downstream ones.
Axial coding. The notes are grouped into a failure taxonomy, and the failures in each category are counted. This step is called the most important one.
Iterative refinement. An agent can cluster the data and search remaining traces for likely instances of the failures already described. Each suggestion is accepted or rejected, and the loop continues until new reviews stop revealing failure modes. Qualitative researchers call this theoretical saturation.
A working pool of roughly 100 diverse traces is a useful starting point. A minimal setup can be as small as 30 minutes spent reviewing 20 to 50 outputs after each significant change, with one domain expert as the quality decision-maker, sometimes called a benevolent dictator.
A few details matter. Anything that makes the product less useful is worth recording, even when the model is not at fault, and root-cause analysis is deferred until problems are prioritized. A rubric is better written after reviewing examples than before, because a rubric can narrow attention to its own items, and ideas about what counts as good shift while reading. This shift is called criteria drift.
In one example, a support agent handled refunds by escalating to a human per policy, and only after reading some interactions did it become clear that the process frustrated users. AI helps with parts of the work, such as a first pass at grouping notes after 30 to 50 traces have been coded by hand. Initial open coding, validating the taxonomy and ground-truth labeling stay with people.
When no production data exists yet, synthetic queries can be generated from a few dimensions that describe how queries vary. For a support bot, these might be issue type, customer mood and prior context. About 20 combinations are written by hand first. An LLM then generates more combinations, and in a separate step each combination is turned into a natural-language query, which avoids repetitive phrasing.
Synthetic data cannot show how common a failure is in production, and it can mislead in specialized or high-stakes domains, with low-resource languages, or for underrepresented user groups.
Review tooling matters too. A custom annotation interface can be built in hours with AI coding tools, shows all the context in one place, and renders data the way the product does.
Turning findings into tasks. A collection of tasks designed to measure a specific capability or behavior is an evaluation suite. A support suite might test refunds, cancellations and escalations. Large numbers are not needed at the start. Around 20 to 50 simple tasks drawn from real failures is a good beginning: early in development, each change tends to have a clear effect, so small samples are enough.
Evals also get harder to build the longer they are delayed, because early on product requirements translate naturally into test cases, and later the success criteria have to be reverse-engineered from a live system.
Sources for the first tasks include the manual checks already run before each release, common tasks users try, the bug tracker and the support queue, prioritized by user impact.
Several things keep tasks fair:
Unambiguous tasks. A good task is one where two domain experts would independently reach the same pass or fail verdict. In one benchmark audit, a task asked an agent to write a script without naming a file path, while the tests assumed a particular path, so the agent could fail through no fault of its own.
Reference solutions. A known working output that passes all graders shows that the task is solvable and the graders are set up correctly. With frontier models, a 0% pass rate across many trials usually points to a broken task and not an incapable agent.
Balanced sets. Cases where a behavior should occur and where it should not both need testing. In web search evals, one team included queries that should trigger a search (finding the weather) and queries that should be answered from knowledge ("who founded Apple?"), and balancing under- and over-triggering took many rounds. The same idea applies to abstention. A balanced set of answerable and unanswerable questions tests whether a model declines when it should. A pass requires answering the answerable ones and refusing the unanswerable ones, and a fabricated answer to an unanswerable question is a failure.
On effort, projects described in the AI Evals FAQ spent 60 to 80 percent of development time on error analysis and evaluation, mostly on looking at data and not on building automated checks. Not every failure needs an automated evaluator. Many turn out to be simple bugs or unspecified preferences that a prompt change fixes. Automated evaluators are worth building for failures that persist after those fixes.
Running the tests
Two more terms apply here. The evaluation harness is the infrastructure that runs evals end to end: it provides instructions and tools, runs tasks concurrently, records all the steps, grades outputs and aggregates results. The agent harness (or scaffold) is the system that lets a model act as an agent, processing inputs, orchestrating tool calls and returning results. Evaluating "an agent" means evaluating the agent harness and the model together.
The agent in the eval should behave about the same as the agent in production, and the environment should not add noise. Each trial starts from a clean environment, because shared state between runs (leftover files, cached data, resource exhaustion) can cause correlated failures that reflect infrastructure flakiness and not agent performance. Shared state can also inflate scores. In some internal evals, a model gained an unfair advantage by reading git history left over from earlier trials.
LangChain's team makes the same point for stateful deep agents: each eval run needs a fresh, clean environment to be reproducible. For one benchmark they run each task inside a dedicated Docker container or sandbox, and for their coding CLI they create a temporary directory for each test case.
For agents that call real services, runs against live APIs can be slow and expensive. One approach is to record HTTP requests to the filesystem and replay them during tests, using vcr in Python or by proxying fetch requests in JavaScript. See LangChain's evaluation learnings.
Several open-source and commercial frameworks can take care of parts of this infrastructure, including Harbor, Braintrust, LangSmith, Langfuse and Arize. Many teams also use their own scripts as a starting point. A framework is only as good as the tasks and graders run through it.
Looking inside a run
An agent can be run in three ways for testing:
Single step: the core agent loop is limited to one turn, to see the next action it decides on.
Full turn: the agent runs in its entirety on a single input, which can involve many tool-calling iterations.
Multiple turns: the agent runs several times in a row, often to simulate a back-and-forth conversation with a user.
For a full turn, three things can be tested:
Trajectory: the sequence of tools called and the arguments the agent generated.
Final response: the last message returned to the user.
Other state: other values the agent produced, such as files or other artifacts.
Single-step tests work like unit tests for decisions. About half of LangChain's test cases for its deep agents were of this kind. Typical questions are whether the agent called the right tool to search for meeting times, inspected the right directory or updated its memories. Regressions often occur at individual decision points, not across full runs. In LangGraph, a break point before the tools node makes it possible to run one step and inspect the tool call that was about to happen.
Full turns give a view of everything the agent does from start to finish. A common trajectory check is that a particular tool was called at some point, without caring exactly when, since an agent might need several calls to find a time slot that suits everyone. For open-ended tasks like coding and research, the quality of the final output often matters more than the path. Other state is examined much like the final response: for a coding agent, by reading and testing the files it wrote, and for a research agent, by asserting that it found the right links or sources.
Multiple turns need some care. If a sequence of user inputs is hardcoded and the agent deviates on turn one, later scripted inputs may stop making sense. One fix is conditional logic: run the first turn, check the output, continue if it was expected, and fail early if not. That avoids modeling every possible branch. To test turn two or three in isolation, a test starts from that point with the right initial state.
Checks are often written per case. LangChain's example is a calendar agent told, "never schedule meetings before 9am." A single test asserts three things: the agent called edit_file on its memories file, it told the user about the update in its final message, and the file contains the rule. The last check can be a regex looking for "9am" or an LLM judge with specific criteria.
Multiple graders can cover one task. A conversational example combines a check on the final state (the ticket is resolved and the refund processed), required tool calls (an identity check, then a refund within a limit, then a confirmation), a transcript limit (at most 10 turns) and a rubric for tone. Even with all of these checks available, grading what the agent produced is generally preferred over grading the exact path it took.
Different agents, different checks
Four kinds of agents are common today: coding, conversational, research and computer use. They can all be evaluated with similar techniques, and the methods below are starting points to extend to a particular domain.
Coding agents write, test and debug code. Deterministic graders fit naturally, because software is generally straightforward to evaluate: does the code run, and do the tests pass? SWE-bench Verified gives agents GitHub issues from popular Python repositories and grades solutions by running the test suite, so a solution passes only if it fixes the failing tests without breaking existing ones. LLMs went from 40% to over 80% on this eval in one year. Terminal-Bench tests end-to-end technical tasks, such as building a Linux kernel from source or training an ML model.
Once pass/fail tests cover the key outcomes, the transcript can be graded as well, for example with code quality rules or a rubric for how the agent called tools or interacted with the user. In practice, coding evals usually rely on unit tests for correctness and an LLM rubric for overall quality, with other graders added only as needed.
Conversational agents work in areas like support, sales and coaching. They keep state, use tools and take actions mid-conversation, and the quality of the interaction is itself part of what is evaluated. Success is multidimensional: was the ticket resolved (a state check), did it finish in under 10 turns (a transcript constraint), and was the tone appropriate (an LLM rubric)?
These evals often need a second LLM to simulate the user. τ-bench and its successor τ²-bench do this, simulating multi-turn interactions in domains like retail support and airline booking, where one model plays a user persona while the agent works through realistic scenarios. Model-based graders are typically used to assess communication quality and goal completion, since many tasks have more than one correct solution.
Research agents gather, synthesize and analyze information. Unlike coding, there is no binary signal from unit tests, and quality is judged relative to the task: what counts as comprehensive, well-sourced or correct depends on context. Experts may disagree about whether a synthesis is complete, ground truth shifts as sources change, and longer outputs leave more room for error.
One strategy is to combine grader types. Groundedness checks verify that claims are supported by retrieved sources. Coverage checks define key facts a good answer must include. Source quality checks confirm that the sources were authoritative and not just the first ones retrieved. Exact match works for questions with objectively correct answers, such as a company's quarterly revenue. BrowseComp is a benchmark built around questions that are easy to verify but hard to solve. Because quality is subjective, LLM rubrics need frequent calibration against expert human judgment.
Computer use agents work through screenshots, mouse clicks, keyboard input and scrolling, the way a person would, and can use any application with a graphical interface. They are evaluated by running them in a real or sandboxed environment and checking whether the intended outcome was achieved.
WebArena tests browser tasks, using URL and page-state checks plus backend verification for tasks that change data, such as confirming an order was actually placed and not just that a confirmation page appeared. OSWorld extends this to full operating system control, with scripts that inspect the file system, application configs, database contents and interface properties afterward.
Browser agents face a tradeoff between token use and latency. Reading the page structure is fast but uses many tokens, while screenshots are slower but use fewer. Summarizing a Wikipedia article suits text extraction, and browsing a shopping site suits screenshots. Evals can check that the agent picks the right approach for each context.
When results change from run to run
Agent behavior varies between runs, so one passing result says little. Each task has its own success rate, maybe 90% on one task and 50% on another, and a task that passed on one run might fail on the next.
Two measures capture this:
pass@k is the chance of at least one correct solution in k attempts. It rises as k grows, because more attempts mean more chances. A 50% pass@1 means the agent succeeds on half the tasks on its first try. It suits cases where one success is enough.
pass^k is the chance that all k trials succeed. It falls as k grows, because consistency across more trials is a harder bar. With a 75% success rate per trial and 3 trials, the chance of passing all three is about 42% (0.75 cubed). It suits customer-facing agents where users expect reliable behavior every time.
At k = 1, the two are identical. By k = 10, they point in opposite directions: pass@k approaches 100% while pass^k falls toward 0%. Which one to use depends on the product.
Beyond correctness, a fixed set of tasks also gives baselines for the number of turns, tool calls and tokens, and for latency measures such as time to first token and time to last token.
Can the judge be trusted?
Where code can check a condition directly, code is used. For example, an assistant that manages contacts can be tested by asking it to create one and then checking that exactly one matching record exists in the database. This needs no human labels. A judgment call is a different case, and an LLM judge can then act as a classifier that returns Pass or Fail for one specific error. It needs to be validated against human labels first.
The process described in the AI Evals FAQ runs as follows:
Labeled examples. For each failure mode, 100 to 200 examples are labeled by a trusted domain expert, with enough Pass and Fail examples to evaluate both classes.
Three sets. The labeled examples are split into a train set (10 to 20 percent, used as examples that can appear in the judge's prompt), a dev set (40 to 45 percent, used while refining the judge) and a test set (40 to 45 percent, kept aside for one final check). When possible, the dev and test sets each hold 30 to 50 Pass and 30 to 50 Fail examples.
Overfitting. Each time dev results change the prompt, information from those examples leaks into the judge. After many rounds, it may do well on the dev set and poorly on new examples. The test set is the check on data that has not guided any change.
Two rates. Here, "positive" means an error is present. The true positive rate shows how many real failures the judge catches (if people find 10 failures and the judge catches 8, the rate is 80%). The true negative rate shows how many good outputs the judge passes correctly (if people find 100 good outputs and the judge passes 95, the rate is 95%, and the other five are false alarms). Both are tracked, since catching more failures can bring more false alarms.
Accuracy alone can mislead. If an agent is right 99% of the time, a judge that always says "pass" is 99% accurate and catches nothing.
When a judge disagrees with human reviewers, at least 50 passing and 50 failing labeled examples help, and the disagreements are inspected for clues such as missing context or vague instructions. The most common mistake is asking one judge to catch too many kinds of errors, so a separate judge is built for each failure type. Each judge also gets only the part of the trace it needs, because extra context can make it worse, and removing one piece at a time and comparing against human labels shows what can be left out.
Starting with a capable model makes the judge's prompt easier to develop. Cheaper models can be tried once it works, measuring how much agreement is lost. Using the same model as the main application is usually fine.
Ready-made metrics such as helpfulness or coherence are not recommended as quality measures, because they can create false confidence. They can still help find traces worth reading. Similarity metrics such as BERTScore or ROUGE are not useful for most LLM output, but they do have a role in search and recommendation, including debugging retrieval.
Graders have bugs of their own, and they are easy to miss. A frontier model scored 42% on CORE-Bench until a researcher found rigid grading that penalized "96.12" when the expected answer was "96.124991…", ambiguous task specs, and tasks that could not be reproduced exactly. After fixes and a less constrained scaffold, the score was 95%.
METR found misconfigured tasks in its time horizon benchmark that asked agents to optimize to a stated score threshold while the grading required exceeding it, which penalized models that followed the instructions. Graders also need to resist bypasses, so that passing requires solving the problem and not exploiting a loophole. Model graders should be closely calibrated with human experts until little divergence remains. After that, occasional human review is enough.
Keeping evals useful over time
Reading transcripts is not a one-time step. When a task fails, the transcript shows whether the agent made a genuine mistake or the graders rejected a valid solution. Failures should feel fair, with a clear account of what the agent got wrong. Eval scores are not taken at face value until someone has read some transcripts.
Evals also wear out. An eval at 100% tracks regressions but gives no signal for improvement, which is called saturation. SWE-bench Verified, for example, started at 30% in the year the guide was written, and frontier models are now nearing saturation above 80%. Near the ceiling, large capability gains show up as small score increases.
The code review startup Qodo was initially unimpressed by a new model because its one-shot coding evals did not capture gains on longer, more complex tasks, so it built a new agentic eval that gave a clearer picture.
The AI Evals FAQ makes a related point: a 100% pass rate usually means the system is not being challenged enough, and a 70% rate may be a more meaningful evaluation.
Eval datasets also go stale as the product and users change. Error analysis is repeated after significant changes such as new features, prompt updates or model switches, with a goal of at least 100 fresh traces each review cycle, typically every 2 to 4 weeks. Evals that keep passing are run less often or retired, and expensive ones like LLM judges are phased out more readily than cheap ones. Scores from a changing eval set may no longer compare with older scores, and product metrics such as churn or active users give a steadier long-term view.
An eval suite is a living artifact, and it needs ownership. A setup that worked well was a dedicated evals team that owns the core infrastructure, while domain experts and product teams contribute most tasks and run the evals themselves. Owning evals becomes as routine as maintaining unit tests. Writing eval tasks also tests whether product requirements are concrete enough to build against, and product managers, customer success or sales colleagues can contribute a task as a pull request using a coding agent.
Evals are one method among several, and each has gaps:
Method | Strengths | Main drawbacks |
Automated evals | Fast, reproducible, no user impact, can run on every commit | Upfront investment and ongoing maintenance, and false confidence if they drift from real usage |
Production monitoring | Real behavior at scale, catches issues synthetic evals miss | Reactive, noisy, lacks ground truth for grading |
A/B testing | Measures real user outcomes, controls for confounds | Slow, needs traffic, only tests changes that are deployed |
User feedback | Surfaces unexpected problems, real examples | Sparse, self-selected, skews toward severe issues |
Manual transcript review | Builds intuition, catches subtle quality issues | Time-intensive, does not scale |
Systematic human studies | Gold-standard judgments, handles subjective tasks | Expensive and slow |
These map to stages of development. Automated evals work as a first line of defense before launch and in CI/CD, on each change and model upgrade. Production monitoring takes over after launch to detect drift and unexpected failures. A/B tests validate significant changes once traffic allows. Feedback triage and weekly transcript sampling are ongoing habits, and systematic human studies are reserved for calibrating LLM graders and for subjective outputs.
Like the Swiss cheese model in safety engineering, no single layer catches every issue, but failures that slip through one layer are caught by another.
Should the evals come first?
There are two views on timing.
One view is eval-driven development: build evals to define planned capabilities before the agent can fulfill them, then iterate until it performs well. It resembles test-driven development in software. Capability evals that start at a low pass rate make it visible which bets on new model abilities have paid off when a new model arrives, and evals written early settle ambiguity in the spec, since two engineers reading the same spec can handle edge cases differently. The same guide also notes that some teams create evals at the start and others add them once at scale.
The other view is that eval-driven development is generally not recommended, because LLMs have an unpredictable and very large surface area for failure, and evaluators written ahead of time can target errors that never happen. The suggestion is to write evaluators for errors discovered through error analysis. An exception is made for specific constraints where success is known exactly, such as "never mention competitors".
The two overlap more than they differ. In both, the first tasks come from real failures: the Anthropic guide suggests starting with 20 to 50 tasks drawn from real failures, and the AI Evals FAQ recommends evaluators that target errors found in the data. Both also stress weighing the cost of each eval.
Where to start
Read 50 to 100 real traces and write down how the agent fails, starting with the first failure in each trace.
Group the failures, count them, and turn the most common and costly ones into 20 to 50 tasks, each with a clear pass or fail and a reference solution.
Check outcomes first. Add strict checks on tool use only for rules that must hold, and add an LLM judge, validated against human labels, only for what code cannot check.
Run several trials per task, and look at pass^k as well as pass@k.
Start every trial in a clean environment, and record and replay calls to outside services.
Read the transcripts of failures to confirm they are fair, and fix grader bugs before changing the agent.
Run the suite on every change, add new failures from production to it, and replace evals once they saturate.
Sources
Anthropic Engineering, Demystifying evals for AI agents
Hamel Husain and Shreya Shankar, AI Evals: Everything You Need to Know
LangChain, Evaluating Deep Agents: Our Learnings
