{"id":"20-evals","number":20,"slug":"evals-for-agents","title":"Evals: measure, don't vibe","short":"Evals","module":"Quality, safety, teams","summary":"Did that new prompt, skill or model actually make things better? Most teams vibe-check. Evals turn that into a measurement. Start by reading traces and counting failure modes, then build small, binary checks from real failures, validate your judges, and read the transcripts before trusting any score.","voice":"Eric","scenes":[{"template":"title","chapter":"Intro","narration":"You changed a prompt, added a skill, or switched models. Did things actually get better? Most teams answer that with a vibe check. This lesson is about answering it with a measurement.","params":{"kicker":"Lesson 20","module":"Module 5 · Quality, safety, teams","title":"Evals: *measure*, don't vibe","subtitle":"How to know whether a change made your agents better.","bubble":"Did I get better? Prove it.","mood":"think"},"start":0.5,"dur":12.668,"words":[{"text":"You","s":0.8,"e":0.96},{"text":"changed","s":1.12,"e":1.28},{"text":"a","s":1.32,"e":1.36},{"text":"prompt,","s":1.44,"e":2.16},{"text":"added","s":2.28,"e":2.4},{"text":"a","s":2.48,"e":2.56},{"text":"skill,","s":2.8,"e":3.2},{"text":"or","s":3.32,"e":3.44},{"text":"switched","s":3.6,"e":3.92},{"text":"models.","s":4,"e":4.64},{"text":"Did","s":4.88,"e":5.04},{"text":"things","s":5.2,"e":5.36},{"text":"actually","s":5.44,"e":5.84},{"text":"get","s":5.92,"e":6.08},{"text":"better?","s":6.24,"e":6.88},{"text":"Most","s":6.96,"e":7.2},{"text":"teams","s":7.36,"e":7.52},{"text":"answer","s":7.68,"e":7.84},{"text":"that","s":7.96,"e":8.08},{"text":"with","s":8.12,"e":8.16},{"text":"a","s":8.24,"e":8.32},{"text":"vibe","s":8.4,"e":8.64},{"text":"check.","s":8.72,"e":9.52},{"text":"This","s":9.64,"e":9.76},{"text":"lesson","s":10,"e":10.24},{"text":"is","s":10.28,"e":10.32},{"text":"about","s":10.4,"e":10.64},{"text":"answering","s":10.76,"e":11.04},{"text":"it","s":11.12,"e":11.2},{"text":"with","s":11.28,"e":11.36},{"text":"a","s":11.44,"e":11.52},{"text":"measurement.","s":11.8,"e":12.32}]},{"template":"formula","chapter":"What an eval is","narration":"Anthropic's definition is refreshingly simple. An eval is a test for an AI system. Give it an input, then apply grading logic to its output. Hamel Husain adds that at its core, evals are just data analytics on your AI application.","params":{"parts":[{"text":"Eval","tone":"agent","at":"eval is"},{"op":"="},{"text":"Input","icon":"doc","sub":"a realistic task","at":"input"},{"op":"+"},{"text":"Grader","icon":"check","sub":"logic that scores the output","at":"grading logic"}],"caption":"“At its core, data analytics on your LLM application.” — Hamel Husain","caption_at":"analytics"},"start":13.168,"dur":16.989,"words":[{"text":"Anthropic's","s":13.468,"e":14.108},{"text":"definition","s":14.428,"e":14.748},{"text":"is","s":14.828,"e":14.988},{"text":"refreshingly","s":15.228,"e":15.708},{"text":"simple.","s":15.788,"e":16.468},{"text":"An","s":16.748,"e":16.908},{"text":"eval","s":16.988,"e":17.468},{"text":"is","s":17.548,"e":17.628},{"text":"a","s":17.708,"e":17.788},{"text":"test","s":17.948,"e":18.108},{"text":"for","s":18.188,"e":18.268},{"text":"an","s":18.348,"e":18.428},{"text":"AI","s":18.508,"e":18.748},{"text":"system.","s":18.828,"e":19.468},{"text":"Give","s":19.708,"e":19.868},{"text":"it","s":19.908,"e":19.948},{"text":"an","s":19.988,"e":20.028},{"text":"input,","s":20.108,"e":20.748},{"text":"then","s":20.868,"e":20.988},{"text":"apply","s":21.068,"e":21.388},{"text":"grading","s":21.468,"e":21.788},{"text":"logic","s":21.988,"e":22.188},{"text":"to","s":22.228,"e":22.268},{"text":"its","s":22.348,"e":22.428},{"text":"output.","s":22.508,"e":22.908},{"text":"Hamel","s":23.388,"e":23.628},{"text":"Husain","s":23.708,"e":24.188},{"text":"adds","s":24.268,"e":24.508},{"text":"that","s":24.588,"e":24.748},{"text":"at","s":24.828,"e":24.908},{"text":"its","s":25.028,"e":25.148},{"text":"core,","s":25.228,"e":25.948},{"text":"evals","s":26.028,"e":26.428},{"text":"are","s":26.508,"e":26.668},{"text":"just","s":26.788,"e":26.908},{"text":"data","s":26.988,"e":27.228},{"text":"analytics","s":27.308,"e":27.868},{"text":"on","s":27.948,"e":28.028},{"text":"your","s":28.108,"e":28.188},{"text":"AI","s":28.268,"e":28.588},{"text":"application.","s":28.908,"e":29.548}]},{"template":"list","chapter":"Why bother","narration":"Why bother? Without evals, Anthropic warns, you get stuck in reactive loops, catching problems in production, where fixing one failure creates others. With them, Andrew Ng's team can test every new model the day it's released and switch if it scores better. And Lauren Tan uses evals as unit tests for her skills.","params":{"title":"What evals buy you","items":[{"text":"Escape reactive loops","sub":"see regressions before users do","at":"reactive loops"},{"text":"Decide on model upgrades","sub":"Andrew Ng: test the new model the day it ships","at":"new model"},{"text":"Unit tests for skills and prompts","sub":"Lauren Tan","at":"unit tests"}]},"start":30.157,"dur":21.064,"words":[{"text":"Why","s":30.457,"e":30.657},{"text":"bother?","s":30.697,"e":31.337},{"text":"Without","s":31.817,"e":32.137},{"text":"evals,","s":32.217,"e":32.937},{"text":"Anthropic","s":33.057,"e":33.497},{"text":"warns,","s":33.617,"e":34.217},{"text":"you","s":34.297,"e":34.377},{"text":"get","s":34.457,"e":34.537},{"text":"stuck","s":34.617,"e":34.857},{"text":"in","s":34.937,"e":35.017},{"text":"reactive","s":35.217,"e":35.577},{"text":"loops,","s":35.657,"e":36.297},{"text":"catching","s":36.377,"e":36.617},{"text":"problems","s":36.697,"e":37.097},{"text":"in","s":37.137,"e":37.177},{"text":"production,","s":37.257,"e":38.057},{"text":"where","s":38.137,"e":38.217},{"text":"fixing","s":38.297,"e":38.617},{"text":"one","s":38.697,"e":38.857},{"text":"failure","s":38.937,"e":39.337},{"text":"creates","s":39.417,"e":39.737},{"text":"others.","s":39.817,"e":40.457},{"text":"With","s":40.697,"e":40.937},{"text":"them,","s":41.057,"e":41.417},{"text":"Andrew","s":41.497,"e":41.737},{"text":"Ng's","s":41.817,"e":42.057},{"text":"team","s":42.137,"e":42.377},{"text":"can","s":42.457,"e":42.537},{"text":"test","s":42.617,"e":42.937},{"text":"every","s":43.017,"e":43.257},{"text":"new","s":43.337,"e":43.417},{"text":"model","s":43.577,"e":43.817},{"text":"the","s":43.897,"e":44.057},{"text":"day","s":44.137,"e":44.217},{"text":"it's","s":44.297,"e":44.457},{"text":"released","s":44.737,"e":45.017},{"text":"and","s":45.097,"e":45.177},{"text":"switch","s":45.257,"e":45.577},{"text":"if","s":45.657,"e":45.737},{"text":"it","s":45.777,"e":45.817},{"text":"scores","s":45.897,"e":46.217},{"text":"better.","s":46.337,"e":46.937},{"text":"And","s":47.017,"e":47.097},{"text":"Lauren","s":47.137,"e":47.417},{"text":"Tan","s":47.577,"e":47.737},{"text":"uses","s":47.817,"e":48.057},{"text":"evals","s":48.137,"e":48.617},{"text":"as","s":48.697,"e":48.857},{"text":"unit","s":48.937,"e":49.177},{"text":"tests","s":49.257,"e":49.577},{"text":"for","s":49.657,"e":49.737},{"text":"her","s":49.817,"e":49.897},{"text":"skills.","s":50.137,"e":50.617}]},{"template":"flow","chapter":"Error analysis first","narration":"Here's the surprise: don't start by writing tests. Hamel Husain and Shreya Shankar start with error analysis. Read real traces, the full logs of what the agent did. For each one, write down only the first thing that went wrong. Then cluster those notes into failure modes, and count them.","params":{"nodes":[{"label":"Read traces","sub":"real runs, start to end","kind":"human","at":"Read real traces"},{"label":"Note the first error","sub":"the most upstream one","kind":"doc","at":"first thing"},{"label":"Cluster","sub":"into failure modes","kind":"agent","at":"cluster"},{"label":"Count","sub":"fix or eval the top ones","kind":"check","at":"count"}],"caption":"“Basic counting is the most powerful analytical technique.” — Hamel Husain","caption_at":"count them"},"start":51.221,"dur":20.907,"words":[{"text":"Here's","s":51.521,"e":51.841},{"text":"the","s":51.881,"e":51.921},{"text":"surprise:","s":52.001,"e":53.121},{"text":"don't","s":53.281,"e":53.521},{"text":"start","s":53.681,"e":54.001},{"text":"by","s":54.081,"e":54.161},{"text":"writing","s":54.241,"e":54.561},{"text":"tests.","s":54.641,"e":55.201},{"text":"Hamel","s":55.681,"e":56.001},{"text":"Husain","s":56.041,"e":56.481},{"text":"and","s":56.561,"e":56.641},{"text":"Shreya","s":56.721,"e":57.041},{"text":"Shankar","s":57.121,"e":57.521},{"text":"start","s":57.601,"e":57.841},{"text":"with","s":57.921,"e":58.081},{"text":"error","s":58.241,"e":58.481},{"text":"analysis.","s":58.561,"e":59.481},{"text":"Read","s":59.761,"e":60.081},{"text":"real","s":60.241,"e":60.481},{"text":"traces,","s":60.561,"e":61.361},{"text":"the","s":61.401,"e":61.441},{"text":"full","s":61.601,"e":61.761},{"text":"logs","s":61.841,"e":62.241},{"text":"of","s":62.281,"e":62.321},{"text":"what","s":62.401,"e":62.481},{"text":"the","s":62.521,"e":62.561},{"text":"agent","s":62.641,"e":62.881},{"text":"did.","s":62.961,"e":63.281},{"text":"For","s":63.761,"e":63.921},{"text":"each","s":64.041,"e":64.161},{"text":"one,","s":64.241,"e":64.801},{"text":"write","s":64.921,"e":65.041},{"text":"down","s":65.161,"e":65.281},{"text":"only","s":65.441,"e":65.761},{"text":"the","s":65.801,"e":65.841},{"text":"first","s":66.081,"e":66.321},{"text":"thing","s":66.401,"e":66.561},{"text":"that","s":66.641,"e":66.721},{"text":"went","s":66.801,"e":66.881},{"text":"wrong.","s":66.961,"e":67.521},{"text":"Then","s":67.761,"e":68.001},{"text":"cluster","s":68.241,"e":68.641},{"text":"those","s":68.761,"e":68.881},{"text":"notes","s":68.961,"e":69.201},{"text":"into","s":69.321,"e":69.441},{"text":"failure","s":69.601,"e":69.921},{"text":"modes,","s":70.001,"e":70.561},{"text":"and","s":70.601,"e":70.641},{"text":"count","s":70.721,"e":70.961},{"text":"them.","s":71.041,"e":71.521}]},{"template":"statement","chapter":"Error analysis first","narration":"One caution from Shreya: don't hand that first reading to an AI. It lacks your product context, so it tends to say the trace looks fine. You read; the AI can help you cluster. And you don't need thousands of traces. Often around thirty is enough before new ones stop revealing new problems.","params":{"text":"You read the traces. The AI helps you *cluster*.","sub":"~30 traces often reveal most failure modes. — Shreya Shankar","art":"eye","tone":"human","sync":false},"start":72.128,"dur":19.706,"words":[{"text":"One","s":72.428,"e":72.668},{"text":"caution","s":72.748,"e":73.148},{"text":"from","s":73.228,"e":73.308},{"text":"Shreya:","s":73.388,"e":74.348},{"text":"don't","s":74.508,"e":74.748},{"text":"hand","s":74.828,"e":74.988},{"text":"that","s":75.068,"e":75.228},{"text":"first","s":75.308,"e":75.548},{"text":"reading","s":75.708,"e":75.868},{"text":"to","s":75.908,"e":75.948},{"text":"an","s":76.028,"e":76.108},{"text":"AI.","s":76.188,"e":76.908},{"text":"It","s":77.148,"e":77.308},{"text":"lacks","s":77.348,"e":77.548},{"text":"your","s":77.628,"e":77.708},{"text":"product","s":77.788,"e":78.108},{"text":"context,","s":78.188,"e":78.988},{"text":"so","s":79.108,"e":79.228},{"text":"it","s":79.308,"e":79.388},{"text":"tends","s":79.428,"e":79.628},{"text":"to","s":79.708,"e":79.788},{"text":"say","s":79.868,"e":80.028},{"text":"the","s":80.068,"e":80.108},{"text":"trace","s":80.188,"e":80.508},{"text":"looks","s":80.628,"e":80.748},{"text":"fine.","s":80.828,"e":81.468},{"text":"You","s":81.788,"e":81.948},{"text":"read;","s":82.028,"e":82.668},{"text":"the","s":82.708,"e":82.748},{"text":"AI","s":82.828,"e":83.228},{"text":"can","s":83.308,"e":83.388},{"text":"help","s":83.468,"e":83.628},{"text":"you","s":83.668,"e":83.708},{"text":"cluster.","s":83.788,"e":84.508},{"text":"And","s":84.748,"e":84.828},{"text":"you","s":84.908,"e":84.988},{"text":"don't","s":85.028,"e":85.228},{"text":"need","s":85.348,"e":85.468},{"text":"thousands","s":85.548,"e":85.948},{"text":"of","s":86.028,"e":86.068},{"text":"traces.","s":86.108,"e":86.908},{"text":"Often","s":87.148,"e":87.388},{"text":"around","s":87.548,"e":87.708},{"text":"thirty","s":87.868,"e":88.108},{"text":"is","s":88.188,"e":88.268},{"text":"enough","s":88.348,"e":88.668},{"text":"before","s":88.748,"e":88.988},{"text":"new","s":89.068,"e":89.228},{"text":"ones","s":89.308,"e":89.468},{"text":"stop","s":89.548,"e":89.788},{"text":"revealing","s":89.908,"e":90.188},{"text":"new","s":90.268,"e":90.348},{"text":"problems.","s":90.428,"e":91.228}]},{"template":"statement","chapter":"Small and real","narration":"Then turn the failures that keep coming back into eval tasks. Anthropic's advice: twenty to fifty simple tasks drawn from real failures is a great start. Early changes have big effects, so small samples are enough, and evals get harder to build the longer you wait.","params":{"text":"Start with *20–50 tasks* drawn from real failures.","sub":"Anthropic, “Demystifying evals for AI agents”","art":"list","tone":"agent","sync":false},"start":91.834,"dur":18.739,"words":[{"text":"Then","s":92.134,"e":92.454},{"text":"turn","s":92.694,"e":92.934},{"text":"the","s":92.974,"e":93.014},{"text":"failures","s":93.094,"e":93.494},{"text":"that","s":93.574,"e":93.654},{"text":"keep","s":93.734,"e":93.894},{"text":"coming","s":93.974,"e":94.214},{"text":"back","s":94.294,"e":94.614},{"text":"into","s":94.694,"e":94.934},{"text":"eval","s":95.014,"e":95.414},{"text":"tasks.","s":95.694,"e":96.414},{"text":"Anthropic's","s":96.854,"e":97.254},{"text":"advice:","s":97.334,"e":98.294},{"text":"twenty","s":98.534,"e":98.614},{"text":"to","s":98.774,"e":98.814},{"text":"fifty","s":98.854,"e":99.254},{"text":"simple","s":99.574,"e":99.654},{"text":"tasks","s":100.134,"e":100.214},{"text":"drawn","s":100.374,"e":100.534},{"text":"from","s":100.694,"e":100.774},{"text":"real","s":101.014,"e":101.094},{"text":"failures","s":101.334,"e":101.654},{"text":"is","s":101.734,"e":101.814},{"text":"a","s":101.894,"e":102.054},{"text":"great","s":102.214,"e":102.294},{"text":"start.","s":102.614,"e":103.174},{"text":"Early","s":103.414,"e":103.494},{"text":"changes","s":103.894,"e":104.014},{"text":"have","s":104.134,"e":104.214},{"text":"big","s":104.374,"e":104.614},{"text":"effects,","s":104.854,"e":105.174},{"text":"so","s":105.334,"e":105.414},{"text":"small","s":105.654,"e":105.734},{"text":"samples","s":106.054,"e":106.094},{"text":"are","s":106.134,"e":106.214},{"text":"enough,","s":106.374,"e":107.134},{"text":"and","s":107.254,"e":107.334},{"text":"evals","s":107.654,"e":107.814},{"text":"get","s":107.894,"e":107.974},{"text":"harder","s":108.214,"e":108.374},{"text":"to","s":108.454,"e":108.614},{"text":"build","s":108.774,"e":108.854},{"text":"the","s":108.934,"e":109.054},{"text":"longer","s":109.174,"e":109.334},{"text":"you","s":109.414,"e":109.574},{"text":"wait.","s":109.734,"e":109.974}]},{"template":"compare","chapter":"Graders","narration":"How should you grade? Prefer code: checking that tests pass, the JSON is valid, the output has the right shape. It's cheap and objective. For subjective qualities, use an LLM judge, but make it binary, pass or fail. Hamel calls one to five scales a weasel way of not making a decision. And give each failure mode its own narrow judge.","params":{"left":{"title":"Weak grading","items":["One judge for “is this good?”","1–5 scores nobody can interpret","Never checked against humans"],"at":"weasel"},"right":{"title":"Strong grading","items":["Code checks wherever possible","Binary pass/fail judges","One judge per failure mode"],"at":"Prefer code"}},"start":110.573,"dur":24.434,"words":[{"text":"How","s":110.873,"e":111.113},{"text":"should","s":111.193,"e":111.353},{"text":"you","s":111.393,"e":111.433},{"text":"grade?","s":111.513,"e":112.193},{"text":"Prefer","s":112.473,"e":112.793},{"text":"code:","s":112.873,"e":113.753},{"text":"checking","s":113.913,"e":114.073},{"text":"that","s":114.153,"e":114.233},{"text":"tests","s":114.313,"e":114.633},{"text":"pass,","s":114.713,"e":115.353},{"text":"the","s":115.393,"e":115.433},{"text":"JSON","s":115.513,"e":115.913},{"text":"is","s":115.993,"e":116.073},{"text":"valid,","s":116.273,"e":116.633},{"text":"the","s":116.673,"e":116.713},{"text":"output","s":116.873,"e":117.193},{"text":"has","s":117.313,"e":117.433},{"text":"the","s":117.473,"e":117.513},{"text":"right","s":117.633,"e":117.753},{"text":"shape.","s":117.833,"e":118.393},{"text":"It's","s":118.633,"e":118.793},{"text":"cheap","s":118.953,"e":119.113},{"text":"and","s":119.273,"e":119.353},{"text":"objective.","s":119.433,"e":120.273},{"text":"For","s":120.553,"e":120.713},{"text":"subjective","s":120.953,"e":121.273},{"text":"qualities,","s":121.353,"e":122.073},{"text":"use","s":122.153,"e":122.313},{"text":"an","s":122.393,"e":122.473},{"text":"LLM","s":122.553,"e":122.953},{"text":"judge,","s":123.113,"e":123.753},{"text":"but","s":123.833,"e":123.913},{"text":"make","s":123.993,"e":124.073},{"text":"it","s":124.153,"e":124.233},{"text":"binary,","s":124.313,"e":124.953},{"text":"pass","s":125.033,"e":125.353},{"text":"or","s":125.433,"e":125.513},{"text":"fail.","s":125.673,"e":125.993},{"text":"Hamel","s":126.473,"e":126.793},{"text":"calls","s":126.953,"e":127.113},{"text":"one","s":127.193,"e":127.273},{"text":"to","s":127.353,"e":127.513},{"text":"five","s":127.593,"e":127.753},{"text":"scales","s":127.833,"e":128.233},{"text":"a","s":128.313,"e":128.393},{"text":"weasel","s":128.553,"e":128.953},{"text":"way","s":129.033,"e":129.193},{"text":"of","s":129.233,"e":129.273},{"text":"not","s":129.433,"e":129.513},{"text":"making","s":129.593,"e":129.833},{"text":"a","s":129.913,"e":129.953},{"text":"decision.","s":129.993,"e":130.753},{"text":"And","s":131.033,"e":131.273},{"text":"give","s":131.513,"e":131.673},{"text":"each","s":131.753,"e":131.993},{"text":"failure","s":132.073,"e":132.393},{"text":"mode","s":132.473,"e":132.633},{"text":"its","s":132.713,"e":132.953},{"text":"own","s":133.033,"e":133.193},{"text":"narrow","s":133.273,"e":133.593},{"text":"judge.","s":133.673,"e":134.393}]},{"template":"stat","chapter":"Validate your judges","narration":"Always check your judges against human labels, and beware of a trap. If real failures happen ten percent of the time, a judge that always says pass agrees with humans ninety percent of the time, while catching nothing. Look at what it gets wrong, not just the agreement rate.","params":{"items":[{"value":90,"suffix":"%","label":"agreement from a judge that always says “pass”","sub":"when only 10% of outputs actually fail","tone":"bad","at":"ninety percent"}],"src":"Hamel Husain on validating LLM judges"},"start":135.007,"dur":18.348,"words":[{"text":"Always","s":135.307,"e":135.787},{"text":"check","s":135.907,"e":136.027},{"text":"your","s":136.107,"e":136.187},{"text":"judges","s":136.267,"e":136.587},{"text":"against","s":136.787,"e":136.987},{"text":"human","s":137.147,"e":137.307},{"text":"labels,","s":137.387,"e":138.107},{"text":"and","s":138.187,"e":138.427},{"text":"beware","s":138.587,"e":138.987},{"text":"of","s":139.027,"e":139.067},{"text":"a","s":139.147,"e":139.187},{"text":"trap.","s":139.227,"e":140.267},{"text":"If","s":140.347,"e":140.427},{"text":"real","s":140.507,"e":140.747},{"text":"failures","s":140.827,"e":141.227},{"text":"happen","s":141.307,"e":141.627},{"text":"ten","s":141.707,"e":141.787},{"text":"percent","s":141.867,"e":142.187},{"text":"of","s":142.227,"e":142.267},{"text":"the","s":142.347,"e":142.427},{"text":"time,","s":142.587,"e":143.147},{"text":"a","s":143.227,"e":143.267},{"text":"judge","s":143.307,"e":143.547},{"text":"that","s":143.627,"e":143.867},{"text":"always","s":144.067,"e":144.267},{"text":"says","s":144.387,"e":144.507},{"text":"pass","s":144.587,"e":145.067},{"text":"agrees","s":145.147,"e":145.467},{"text":"with","s":145.547,"e":145.707},{"text":"humans","s":145.787,"e":146.187},{"text":"ninety","s":146.267,"e":146.427},{"text":"percent","s":146.587,"e":146.827},{"text":"of","s":146.907,"e":146.947},{"text":"the","s":146.987,"e":147.067},{"text":"time,","s":147.227,"e":147.627},{"text":"while","s":147.747,"e":147.867},{"text":"catching","s":147.987,"e":148.347},{"text":"nothing.","s":148.547,"e":149.387},{"text":"Look","s":149.467,"e":149.627},{"text":"at","s":149.707,"e":149.787},{"text":"what","s":149.827,"e":149.867},{"text":"it","s":149.947,"e":150.027},{"text":"gets","s":150.147,"e":150.267},{"text":"wrong,","s":150.347,"e":151.067},{"text":"not","s":151.147,"e":151.387},{"text":"just","s":151.467,"e":151.547},{"text":"the","s":151.627,"e":151.707},{"text":"agreement","s":151.787,"e":152.187},{"text":"rate.","s":152.347,"e":152.747}]},{"template":"stat","chapter":"Consistency","narration":"Also decide what success means. If one success out of several tries is enough, measure pass at k. If every run must succeed, like a customer-facing agent, measure pass to the power k. An agent that succeeds seventy-five percent of the time passes three runs in a row only about forty-two percent of the time.","params":{"items":[{"value":75,"suffix":"%","label":"success on a single run","tone":"good","at":"seventy-five"},{"value":42,"suffix":"%","label":"chance all 3 runs succeed (0.75³)","tone":"bad","at":"forty-two"}],"src":"Anthropic: pass@k (any success) vs pass^k (every run succeeds)"},"start":153.355,"dur":21.378,"words":[{"text":"Also","s":153.655,"e":153.975},{"text":"decide","s":154.055,"e":154.535},{"text":"what","s":154.615,"e":154.775},{"text":"success","s":154.855,"e":155.415},{"text":"means.","s":155.495,"e":156.175},{"text":"If","s":156.455,"e":156.615},{"text":"one","s":156.695,"e":156.855},{"text":"success","s":157.095,"e":157.335},{"text":"out","s":157.415,"e":157.495},{"text":"of","s":157.575,"e":157.655},{"text":"several","s":157.735,"e":158.055},{"text":"tries","s":158.135,"e":158.455},{"text":"is","s":158.495,"e":158.535},{"text":"enough,","s":158.615,"e":159.255},{"text":"measure","s":159.335,"e":159.575},{"text":"pass","s":159.735,"e":159.975},{"text":"at","s":160.055,"e":160.215},{"text":"k.","s":160.295,"e":160.935},{"text":"If","s":161.255,"e":161.415},{"text":"every","s":161.495,"e":161.735},{"text":"run","s":161.815,"e":162.055},{"text":"must","s":162.135,"e":162.375},{"text":"succeed,","s":162.455,"e":163.095},{"text":"like","s":163.175,"e":163.255},{"text":"a","s":163.295,"e":163.335},{"text":"customer-facing","s":163.415,"e":164.135},{"text":"agent,","s":164.215,"e":164.935},{"text":"measure","s":165.015,"e":165.255},{"text":"pass","s":165.335,"e":165.655},{"text":"to","s":165.695,"e":165.735},{"text":"the","s":165.815,"e":165.895},{"text":"power","s":165.975,"e":166.295},{"text":"k.","s":166.375,"e":166.975},{"text":"An","s":167.335,"e":167.415},{"text":"agent","s":167.575,"e":167.815},{"text":"that","s":167.895,"e":167.975},{"text":"succeeds","s":168.055,"e":168.535},{"text":"seventy-five","s":168.615,"e":169.175},{"text":"percent","s":169.255,"e":169.495},{"text":"of","s":169.575,"e":169.615},{"text":"the","s":169.655,"e":169.735},{"text":"time","s":169.815,"e":170.055},{"text":"passes","s":170.215,"e":170.615},{"text":"three","s":170.695,"e":170.935},{"text":"runs","s":171.015,"e":171.175},{"text":"in","s":171.255,"e":171.295},{"text":"a","s":171.335,"e":171.375},{"text":"row","s":171.415,"e":171.655},{"text":"only","s":171.815,"e":172.055},{"text":"about","s":172.175,"e":172.295},{"text":"forty-two","s":172.455,"e":172.935},{"text":"percent","s":173.015,"e":173.335},{"text":"of","s":173.375,"e":173.415},{"text":"the","s":173.455,"e":173.495},{"text":"time.","s":173.575,"e":174.135}]},{"template":"quote","chapter":"Read the transcripts","narration":"Finally, read the transcripts. Anthropic found Claude Opus 4.5 scoring forty-two percent on one benchmark, until they fixed bugs in the grading and the tasks. Then it scored ninety-five percent. Broken graders hide real ability. And grade the outcome, not the exact path the agent took.","params":{"text":"You won't know if your graders are working well unless you *read the transcripts* and grades from many trials.","who":"Anthropic","role":"Demystifying evals for AI agents","src":"anthropic.com/engineering"},"start":174.733,"dur":21.064,"words":[{"text":"Finally,","s":175.033,"e":175.833},{"text":"read","s":175.993,"e":176.153},{"text":"the","s":176.233,"e":176.273},{"text":"transcripts.","s":176.313,"e":177.313},{"text":"Anthropic","s":177.593,"e":178.153},{"text":"found","s":178.273,"e":178.393},{"text":"Claude","s":178.473,"e":178.793},{"text":"Opus","s":178.873,"e":179.193},{"text":"4.5","s":179.273,"e":179.993},{"text":"scoring","s":180.153,"e":180.633},{"text":"forty-two","s":180.753,"e":181.113},{"text":"percent","s":181.193,"e":181.673},{"text":"on","s":181.753,"e":181.913},{"text":"one","s":181.993,"e":182.073},{"text":"benchmark,","s":182.153,"e":183.033},{"text":"until","s":183.113,"e":183.353},{"text":"they","s":183.433,"e":183.513},{"text":"fixed","s":183.633,"e":183.753},{"text":"bugs","s":183.833,"e":184.073},{"text":"in","s":184.153,"e":184.193},{"text":"the","s":184.233,"e":184.313},{"text":"grading","s":184.513,"e":184.713},{"text":"and","s":184.753,"e":184.793},{"text":"the","s":184.833,"e":184.873},{"text":"tasks.","s":184.953,"e":185.913},{"text":"Then","s":186.113,"e":186.313},{"text":"it","s":186.393,"e":186.473},{"text":"scored","s":186.553,"e":186.873},{"text":"ninety-five","s":186.953,"e":187.513},{"text":"percent.","s":187.673,"e":188.633},{"text":"Broken","s":188.793,"e":188.953},{"text":"graders","s":189.033,"e":189.513},{"text":"hide","s":189.593,"e":189.833},{"text":"real","s":189.993,"e":190.233},{"text":"ability.","s":190.473,"e":191.353},{"text":"And","s":191.473,"e":191.593},{"text":"grade","s":191.753,"e":191.993},{"text":"the","s":192.033,"e":192.073},{"text":"outcome,","s":192.233,"e":192.953},{"text":"not","s":193.033,"e":193.193},{"text":"the","s":193.233,"e":193.273},{"text":"exact","s":193.353,"e":193.753},{"text":"path","s":193.833,"e":194.153},{"text":"the","s":194.193,"e":194.233},{"text":"agent","s":194.313,"e":194.633},{"text":"took.","s":194.793,"e":195.193}]},{"template":"statement","chapter":"They know","narration":"One last twist from Lauren Tan. When she evaluates skills, she runs the agents in neutrally named folders, because agents can tell when they're being evaluated, and when they do, they change their behavior.","params":{"text":"Agents can tell when they're being tested, and they *behave differently*.","sub":"Lauren Tan runs evals in neutrally named folders.","art":"eye","tone":"agent","sync":false},"start":195.797,"dur":13.463,"words":[{"text":"One","s":196.097,"e":196.337},{"text":"last","s":196.417,"e":196.657},{"text":"twist","s":196.737,"e":197.057},{"text":"from","s":197.137,"e":197.297},{"text":"Lauren","s":197.337,"e":197.617},{"text":"Tan.","s":197.817,"e":198.297},{"text":"When","s":198.577,"e":198.657},{"text":"she","s":198.737,"e":198.897},{"text":"evaluates","s":199.057,"e":199.457},{"text":"skills,","s":199.537,"e":200.257},{"text":"she","s":200.337,"e":200.497},{"text":"runs","s":200.617,"e":200.737},{"text":"the","s":200.817,"e":200.897},{"text":"agents","s":201.097,"e":201.297},{"text":"in","s":201.377,"e":201.537},{"text":"neutrally","s":201.577,"e":202.017},{"text":"named","s":202.097,"e":202.337},{"text":"folders,","s":202.417,"e":203.217},{"text":"because","s":203.377,"e":203.537},{"text":"agents","s":203.617,"e":203.937},{"text":"can","s":204.017,"e":204.177},{"text":"tell","s":204.257,"e":204.497},{"text":"when","s":204.577,"e":204.657},{"text":"they're","s":204.697,"e":204.817},{"text":"being","s":204.897,"e":204.977},{"text":"evaluated,","s":205.057,"e":206.097},{"text":"and","s":206.177,"e":206.257},{"text":"when","s":206.337,"e":206.417},{"text":"they","s":206.497,"e":206.577},{"text":"do,","s":206.657,"e":207.217},{"text":"they","s":207.297,"e":207.377},{"text":"change","s":207.497,"e":207.617},{"text":"their","s":207.697,"e":207.777},{"text":"behavior.","s":208.057,"e":208.657}]},{"template":"quiz","chapter":"Quick check","narration":"Quick check. Your LLM judge agrees with human labels ninety percent of the time, and real failures happen in about ten percent of cases. What should you check? Tap your answer.","params":{"q":"Your judge agrees with humans 90% of the time; real failures are ~10%. What do you check?","options":["Nothing, 90% agreement is great","What it gets wrong: an always-pass judge also scores 90%","Switch the judge to a 1–5 scale","Add three more judges and average them"],"answer":1,"explain":"With rare failures, agreement is misleading. Check whether it actually catches the failures."},"start":209.26,"dur":16.733,"words":[{"text":"Quick","s":209.56,"e":209.8},{"text":"check.","s":209.88,"e":210.2},{"text":"Your","s":210.68,"e":210.84},{"text":"LLM","s":211,"e":211.4},{"text":"judge","s":211.48,"e":211.8},{"text":"agrees","s":211.88,"e":212.28},{"text":"with","s":212.36,"e":212.44},{"text":"human","s":212.6,"e":212.76},{"text":"labels","s":212.84,"e":213.16},{"text":"ninety","s":213.32,"e":213.56},{"text":"percent","s":213.64,"e":213.96},{"text":"of","s":214,"e":214.04},{"text":"the","s":214.08,"e":214.12},{"text":"time,","s":214.2,"e":214.84},{"text":"and","s":214.92,"e":215.08},{"text":"real","s":215.16,"e":215.4},{"text":"failures","s":215.48,"e":215.96},{"text":"happen","s":216,"e":216.28},{"text":"in","s":216.32,"e":216.36},{"text":"about","s":216.48,"e":216.6},{"text":"ten","s":216.76,"e":216.84},{"text":"percent","s":216.92,"e":217.24},{"text":"of","s":217.28,"e":217.32},{"text":"cases.","s":217.4,"e":218.16},{"text":"What","s":218.44,"e":218.6},{"text":"should","s":218.68,"e":218.76},{"text":"you","s":218.84,"e":218.88},{"text":"check?","s":218.92,"e":219.32},{"text":"Tap","s":219.8,"e":219.96},{"text":"your","s":220.04,"e":220.12},{"text":"answer.","s":220.2,"e":220.84}]},{"template":"exercise","chapter":"Try it","narration":"Build a tiny eval for something you use every day: a skill, a prompt or a memory file. Ten real tasks and a pass or fail check each is enough to start.","params":{"title":"Try it with your agent","intro":"Pick a skill, prompt or memory file you rely on:","prompt":"Help me build a tiny eval for [skill/prompt]. Collect 10 real past\ntasks from our history. For each, define a binary pass/fail check,\nscripted where possible. Run the current setup, report the pass rate,\nand show me the transcripts of the failures. Save it to evals/[name]/."},"start":225.993,"dur":11.477,"words":[{"text":"Build","s":226.293,"e":226.533},{"text":"a","s":226.613,"e":226.693},{"text":"tiny","s":226.773,"e":227.093},{"text":"eval","s":227.173,"e":227.573},{"text":"for","s":227.653,"e":227.733},{"text":"something","s":227.813,"e":228.133},{"text":"you","s":228.213,"e":228.293},{"text":"use","s":228.373,"e":228.613},{"text":"every","s":228.693,"e":228.933},{"text":"day:","s":229.013,"e":229.653},{"text":"a","s":229.733,"e":229.773},{"text":"skill,","s":229.813,"e":230.533},{"text":"a","s":230.573,"e":230.613},{"text":"prompt","s":230.693,"e":231.173},{"text":"or","s":231.333,"e":231.493},{"text":"a","s":231.533,"e":231.573},{"text":"memory","s":231.733,"e":231.893},{"text":"file.","s":231.973,"e":232.293},{"text":"Ten","s":232.853,"e":233.093},{"text":"real","s":233.173,"e":233.413},{"text":"tasks","s":233.493,"e":233.973},{"text":"and","s":234.053,"e":234.133},{"text":"a","s":234.173,"e":234.213},{"text":"pass","s":234.293,"e":234.533},{"text":"or","s":234.613,"e":234.693},{"text":"fail","s":234.813,"e":234.933},{"text":"check","s":235.013,"e":235.253},{"text":"each","s":235.333,"e":235.653},{"text":"is","s":235.733,"e":235.893},{"text":"enough","s":235.933,"e":236.133},{"text":"to","s":236.173,"e":236.213},{"text":"start.","s":236.293,"e":236.853}]},{"template":"recap","chapter":"Recap","narration":"Recap. Measure, don't vibe. Start with error analysis and counting. Build twenty to fifty tasks from real failures. Prefer code graders and binary judges, and validate them. And read the transcripts. Next: the security risk every agent user needs to understand.","params":{"items":["Measure, don't *vibe*","Error analysis → count failure modes","Code graders; binary, validated judges","Read the transcripts"],"next":"Security: prompt injection and the lethal trifecta","next_at":"Next"},"start":237.47,"dur":21.701,"words":[{"text":"Recap.","s":237.77,"e":238.41},{"text":"Measure,","s":238.89,"e":239.53},{"text":"don't","s":239.69,"e":239.93},{"text":"vibe.","s":240.01,"e":240.49},{"text":"Start","s":240.97,"e":241.21},{"text":"with","s":241.29,"e":241.45},{"text":"error","s":241.53,"e":241.77},{"text":"analysis","s":241.85,"e":242.41},{"text":"and","s":242.49,"e":242.57},{"text":"counting.","s":242.65,"e":243.21},{"text":"Build","s":243.61,"e":243.85},{"text":"twenty","s":243.93,"e":244.17},{"text":"to","s":244.25,"e":244.41},{"text":"fifty","s":244.45,"e":244.65},{"text":"tasks","s":244.73,"e":245.21},{"text":"from","s":245.29,"e":245.45},{"text":"real","s":245.61,"e":245.85},{"text":"failures.","s":245.93,"e":246.57},{"text":"Prefer","s":246.97,"e":247.37},{"text":"code","s":247.45,"e":247.69},{"text":"graders","s":247.73,"e":248.09},{"text":"and","s":248.17,"e":248.25},{"text":"binary","s":248.33,"e":248.73},{"text":"judges,","s":248.81,"e":249.45},{"text":"and","s":249.53,"e":249.61},{"text":"validate","s":249.69,"e":250.17},{"text":"them.","s":250.25,"e":250.41},{"text":"And","s":250.81,"e":251.13},{"text":"read","s":251.45,"e":251.69},{"text":"the","s":251.73,"e":251.77},{"text":"transcripts.","s":251.85,"e":253.13},{"text":"Next:","s":253.69,"e":254.13},{"text":"the","s":254.17,"e":254.41},{"text":"security","s":254.65,"e":254.73},{"text":"risk","s":254.97,"e":255.13},{"text":"every","s":255.45,"e":255.53},{"text":"agent","s":255.85,"e":255.93},{"text":"user","s":256.17,"e":256.25},{"text":"needs","s":256.49,"e":256.53},{"text":"to","s":256.57,"e":256.65},{"text":"understand.","s":256.89,"e":257.61}]}],"total":259.571,"audio":"/audio/20-evals.mp3"}