hardeep.cv/Study/Agent evalsLesson 02
#evals#agents#article-notes

Grade what the agent did, not what it said

An agent eval runs a task several times as trials. Each trial leaves a transcript of what the agent said and an outcome in the world. Graders should score the outcome.

AnthropicResearch article · notes
October 21, 202610 min readJan 2026 article
Read the article

What you’ll learn

  1. 01What an agent eval isTasks, trials, transcripts, outcomes and graders.
  2. 02Which grader to useCode, model or human, and why the path doesn’t count.
  3. 03pass@k vs pass^kOne agent at 75% per trial scores 98% or 42%.
  4. 04How to startNine steps, from 20 tasks to a suite you trust.
× k acts on says ends grade this Taskbook a flight Trialsk = 3 runs Agentmodel + harness Environmentclean each trial Transcript“Flight booked!” Outcomereservation in DB Graders3 types Scorepass@k
  1. A task is one test with inputs and success criteria, here “book this flight.” Each attempt is a trial. Outputs vary between runs, so you run several trials.
  2. You evaluate the model and its harness together. Each trial starts from a clean environment, so files and cached data from one run can’t help or hurt the next.
  3. Every trial leaves two records. The transcript holds everything the agent said, called and reasoned. The outcome is the final state of the environment.
  4. Graders score the trial. The agent may say “Your flight has been booked,” but the grader checks whether the reservation exists in the SQL database.
Fig. 1One agent eval. The eval harness runs the trials, records every step, grades each trial and adds up the results.

Map of the article

The roadmap is the longest part, about 30% of the words

Anatomy11% Why9% Graders8% Agent types20% 5% Roadmap, steps 0 to 830% Layers10% End8% ▲ pass@k 0 words≈ 6,000 words
Fig. 2Section lengths by word count. This lesson skips most of the advice per agent type (coding, chat, research, computer use) and keeps what applies to every agent.

Graders

Use code where you can, a model where you must, and humans to calibrate

A grader scores one part of the transcript or the outcome. A task can have several graders. Per task, you can weight their scores against a threshold, require all of them to pass, or mix the two.

Where possible

Code-based

bookings tableafter the trial SELECT … WHERE id ✓ row exists

String checks, unit tests, static analysis, state checks. Fast, cheap, objective and easy to debug. Brittle when a valid answer doesn’t match the pattern.

Where necessary

Model-based

transcript + rubricone judge per dimension score or “Unknown” needs calibration

Rubrics, natural-language assertions, pairwise or reference comparison. Flexible and good with open-ended output. Non-deterministic and costlier than code.

To calibrate

Human

sample of trialsexpert review do two experts agree? gold standard

Expert review, spot checks, A/B tests, inter-annotator agreement. Matches expert judgment. Expensive, slow and hard to get at scale.

Fig. 3The article’s rule. Pick deterministic graders where possible, LLM graders where necessary, and human graders for extra validation.

Two tips for LLM judges. Give the judge a way out, such as returning “Unknown” when it lacks information, so it doesn’t make up a verdict. And grade each rubric dimension with its own isolated judge instead of one judge for everything.

What to grade

Grade what the agent produced, not the path it took

It is tempting to check that the agent called the right tools in the right order. Anthropic found this too rigid. Agents often find valid approaches the eval designer didn’t expect. Frontier models can even outgrow a static eval. Opus 4.5 solved a τ2-bench flight booking problem by finding a loophole in the policy. It “failed” as written, but its answer was better for the user.

Wrong fail

Tool-order check

expects read → edit → testagent: grep → edit → test tool_calls in order ✗ FAIL · valid fix

Punishes a creative path that worked.

Fair

Outcome check

same runany path tests pass? ✓ PASS

Checks the result the user cares about.

Partial credit

Score the parts

✓ found the problem✓ verified the customer✗ processed the refund 2 of 3 parts illustrative

Better than failing at once. The score should show it.

Fig. 4For tasks with several parts, build in partial credit. And make graders hard to cheat, so passing means the problem was solved.

Non-determinism

At three trials, the same 75% agent scores 98% or 42%

Agents behave differently from run to run. So the question is how often a task succeeds across trials. Two metrics answer two different questions. pass@k is the chance that at least one of k trials succeeds. pass^k is the chance that all k succeed.

100%75%50%25%0% 12345678910 trials per task (k) pass@k pass^k k = 1 · both 75% 98%42% ≈ 100%6%
  1. One trial per task. Both metrics equal the per-trial success rate, 75%.
  2. Three trials. pass@3 needs one success, so it is 1 − 0.25³ ≈ 98%. pass^3 needs three out of three, so it is 0.75³ ≈ 42%.
  3. Ten trials. pass@10 is about 100%. pass^10 is 0.75¹⁰, about 6%, and keeps falling toward zero.
Fig. 5The same agent at 75% per trial. More trials make pass@k rise and pass^k fall, so the two tell opposite stories by k = 10.
Use pass@k when one success is enough

Tools that propose options

If the user can pick any working answer, more shots on goal is fine. In coding, pass@1 often matters most, because the fix should work the first try.

Use pass^k when every run must work

Customer-facing agents

Users expect the same good result each time. A 75% agent fails at least one of three conversations more than half the time.

Two kinds of suite

Capability evals start low, then graduate into regression evals

graduates once it passes reliably Capability evals“What can it do well?”low pass rate, a hill to climb Regression evals“Can it still do it?”nearly 100%, catches drift ▲ a capability eval at 100% is saturated, so it tracks regressions but shows no room to improve
Fig. 6Run both. While you climb the capability hill, regression evals check that the change didn’t break something else. The bar lengths are illustrative.

Saturation hides progress. Frontier models now score above 80% on SWE-bench Verified and are nearing saturation. Near the top only the hardest tasks remain, so big capability gains show up as small score bumps. The code review startup Qodo was at first unimpressed by Opus 4.5. Its one-shot coding evals missed the gains on longer, more complex tasks. Qodo built a new agentic eval framework, which gave a much clearer picture.

Zero to one

Nine steps from no evals to evals you can trust

collect tasksharness and gradersmaintain Start earlystep 0 Manual QAstep 1 Clear tasksstep 2 Both sidesstep 3 Clean envstep 4 Gradersstep 5 Transcriptsstep 6 Saturationstep 7 Ownershipstep 8
Fig. 7The article calls this eval-driven agent development. Define success early, measure it clearly, and keep iterating.
  1. Start early. Teams wait because they think they need hundreds of tasks. 20 to 50 simple tasks drawn from real failures is a great start. Early changes have large effects, so small samples are enough.
  2. Start with what you test by hand. Use the checks you run before each release. In production, mine the bug tracker and support queue, and prioritize by user impact.
  3. Write unambiguous tasks with reference solutions. Two domain experts should reach the same pass or fail verdict on their own. A reference solution proves the task is solvable and the graders work.
  4. Build balanced sets. Test when a behavior should happen and when it shouldn’t. Claude.ai’s web search evals had queries that need a search, like the weather, and queries it should answer from memory, like “who founded Apple?”
  5. Isolate every trial. Start each one from a clean environment. Leftover files, cached data or resource exhaustion cause correlated failures that have nothing to do with the agent.
  6. Design graders with care. Grade the output, add partial credit, give judges an “Unknown” option, calibrate them against human experts, and make them hard to cheat.
  7. Read the transcripts. A failure should look fair. The transcript shows whether the agent made a real mistake or the grader rejected a valid solution.
  8. Watch for saturation. A suite at 100% gives no signal for improvement. Don’t take a score at face value until someone has read the details and some transcripts.
  9. Give the suite owners. At Anthropic, an evals team owns the infrastructure, and domain experts and product teams write most tasks. Product managers or salespeople can add a task as a PR with Claude Code.

When the eval is wrong

A low score can mean a broken eval, not a weak agent

With frontier models, a 0% pass rate across many trials (0% pass@100) is most often a broken task. The article has four cases where the eval, not the agent, moved the score.

Hidden spec

Terminal-Bench

“write a script”no filepath given tests assume a path ✗ not the agent’s fault

Everything the grader checks should be clear from the task.

Rigid grading

CORE-Bench

agent: 96.12a correct answer wants 96.124991… ✗ FAIL

Plus ambiguous specs and stochastic tasks that could not be reproduced exactly.

Wrong threshold

METR time horizon

“reach this score”the stated goal grader: must exceed it ✗ obeying is punished

Models that ignored the stated goal scored better than Claude.

Shared state

Anthropic internal

trial 2 startsenv not reset reads old git history ▲ score inflated

Shared state can make an agent look better than it is.

Fig. 8Three evals that scored a good agent too low, and one that scored it too high.
as graded42% after fixes95%
Fig. 9Opus 4.5 on CORE-Bench. An Anthropic researcher fixed the grading bugs and used a less constrained scaffold. Same model, more than double the score.

Beyond evals

Automated evals are one slice of Swiss cheese

Evals run thousands of tasks without touching real users. They still miss things. The article pairs them with five other ways to watch an agent, each with its own blind spots.

failures → ✕ ✕ ✕ Automatedevalsevery change Productionmonitoringafter launch A/B testswith traffic Userfeedbacktriage always Transcriptreviewweekly sample Humanstudiesto calibrate
Fig. 10No single layer catches every issue. A failure that slips through one layer gets caught by another.
Automated evals

Fast, reproducible, no user impact. Cost up front and drift if not maintained. Can give false confidence if they don’t match real usage.

Production monitoring

Shows real behavior at scale. Reactive, so problems reach users first, and it lacks ground truth for grading.

A/B tests

Measure real outcomes like retention. Slow, need traffic, and only test changes you ship.

User feedback

Surfaces what you didn’t expect, with real examples. Sparse, skewed to severe issues, and users rarely say why.

Transcript review

Builds intuition and catches subtle issues. Slow, doesn’t scale, and mostly qualitative.

Human studies

Gold-standard ratings for subjective work. Expensive, slow, and raters disagree. Use them to calibrate LLM graders.

What to remember

Start small, grade outcomes, and read the transcripts

  1. Write 20 to 50 tasks from real failures now. Don’t wait for hundreds.
  2. Make each task clear enough that two experts agree, and give it a reference solution that passes.
  3. Grade the outcome in the environment, not the agent’s claim or its exact tool path. Add partial credit.
  4. Use code graders first, LLM judges where needed, and humans to calibrate the judges.
  5. Run several trials. Report pass^k when users need it to work every time.
  6. Keep a regression suite near 100% and a capability suite with room to climb. Read transcripts before you trust either.

Check yourself

Three quick questions

The agent ends with “Refund processed.” What should the grader check?

The refund record, not the message

That sentence is in the transcript. The outcome is the state of the system, such as a processed refund and a resolved ticket.

A support agent succeeds 75% of the time. A customer contacts it three times. Which metric?

pass^k, about 42%

The customer needs all three to go well, so 0.75³ is the honest number.

A new capability task scores 0% across 100 trials. What do you check first?

The task and its grader

With frontier models, 0% pass@100 most often means a broken task. Write a reference solution and see if it passes.

Related lesson

Step 5 says calibrate your LLM judges. Lesson 01 shows how.

  1. Lesson 01Evals are fuzzy unit tests, and the judge can lieFour biases of LLM judges, and how to check a judge against about 30 human-labeled cases.

Sources

hardeep.cv

© 2026 Hardeep Singh