Did that new prompt, skill or model actually make things better? Most teams vibe-check. Evals turn that into a measurement. Start by reading traces and counting failure modes, then build small, binary checks from real failures, validate your judges, and read the transcripts before trusting any score.
Key takeaways
- An eval is a test for an AI system: an input plus grading logic for the output.
- Start with error analysis: read traces, note the first failure in each, cluster, and count.
- 20 to 50 tasks drawn from real failures is a great start.
- Prefer code-based graders; use binary LLM judges for subjective checks, and validate them against human labels.
- Read the transcripts. Broken tasks and graders are common, and they hide real capability.
Try it with your agent
Pick a skill, prompt or memory file you rely on:
Help me build a tiny eval for [skill/prompt]. Collect 10 real past tasks from our history. For each, define a binary pass/fail check, scripted where possible. Run the current setup, report the pass rate, and show me the transcripts of the failures. Save it to evals/[name]/.
Transcript
Intro
You changed a prompt, added a skill, or switched models. Did things actually get better? Most teams answer that with a vibe check. This lesson is about answering it with a measurement.
What an eval is
Anthropic's definition is refreshingly simple. An eval is a test for an AI system. Give it an input, then apply grading logic to its output. Hamel Husain adds that at its core, evals are just data analytics on your AI application.
Why bother
Why bother? Without evals, Anthropic warns, you get stuck in reactive loops, catching problems in production, where fixing one failure creates others. With them, Andrew Ng's team can test every new model the day it's released and switch if it scores better. And Lauren Tan uses evals as unit tests for her skills.
Error analysis first
Here's the surprise: don't start by writing tests. Hamel Husain and Shreya Shankar start with error analysis. Read real traces, the full logs of what the agent did. For each one, write down only the first thing that went wrong. Then cluster those notes into failure modes, and count them.
One caution from Shreya: don't hand that first reading to an AI. It lacks your product context, so it tends to say the trace looks fine. You read; the AI can help you cluster. And you don't need thousands of traces. Often around thirty is enough before new ones stop revealing new problems.
Small and real
Then turn the failures that keep coming back into eval tasks. Anthropic's advice: twenty to fifty simple tasks drawn from real failures is a great start. Early changes have big effects, so small samples are enough, and evals get harder to build the longer you wait.
Graders
How should you grade? Prefer code: checking that tests pass, the JSON is valid, the output has the right shape. It's cheap and objective. For subjective qualities, use an LLM judge, but make it binary, pass or fail. Hamel calls one to five scales a weasel way of not making a decision. And give each failure mode its own narrow judge.
Validate your judges
Always check your judges against human labels, and beware of a trap. If real failures happen ten percent of the time, a judge that always says pass agrees with humans ninety percent of the time, while catching nothing. Look at what it gets wrong, not just the agreement rate.
Consistency
Also decide what success means. If one success out of several tries is enough, measure pass at k. If every run must succeed, like a customer-facing agent, measure pass to the power k. An agent that succeeds seventy-five percent of the time passes three runs in a row only about forty-two percent of the time.
Read the transcripts
Finally, read the transcripts. Anthropic found Claude Opus 4.5 scoring forty-two percent on one benchmark, until they fixed bugs in the grading and the tasks. Then it scored ninety-five percent. Broken graders hide real ability. And grade the outcome, not the exact path the agent took.
They know
One last twist from Lauren Tan. When she evaluates skills, she runs the agents in neutrally named folders, because agents can tell when they're being evaluated, and when they do, they change their behavior.
Quick check
Quick check. Your LLM judge agrees with human labels ninety percent of the time, and real failures happen in about ten percent of cases. What should you check? Tap your answer.
Try it
Build a tiny eval for something you use every day: a skill, a prompt or a memory file. Ten real tasks and a pass or fail check each is enough to start.
Recap
Recap. Measure, don't vibe. Start with error analysis and counting. Build twenty to fifty tasks from real failures. Prefer code graders and binary judges, and validate them. And read the transcripts. Next: the security risk every agent user needs to understand.