{"id":"03-jagged","number":3,"slug":"what-agents-are-good-at","title":"What agents are great at, and where they fail","short":"Where agents shine and fail","module":"The shift","summary":"Agent intelligence is jagged: superhuman on some tasks, baffling on others. The pattern isn't random. Agents fly where results are easy to verify and stumble on judgment, design and long, unchecked runs. Learn to delegate by verifiability.","voice":"Eric","scenes":[{"template":"title","chapter":"Intro","narration":"Agents are brilliant and baffling, often in the same hour. That's not random. Once you see the pattern, you'll know what to hand off and what to keep.","params":{"kicker":"Lesson 3","module":"Module 1 · The shift","title":"What agents are *great* at, and where they fail","subtitle":"Jagged intelligence, and how to delegate around it.","bubble":"I'm a genius. Mostly.","mood":"think"},"start":0.5,"dur":10.839,"words":[{"text":"Agents","s":0.8,"e":1.28},{"text":"are","s":1.4,"e":1.52},{"text":"brilliant","s":1.64,"e":2.24},{"text":"and","s":2.32,"e":2.56},{"text":"baffling,","s":2.64,"e":3.52},{"text":"often","s":3.6,"e":3.92},{"text":"in","s":3.96,"e":4},{"text":"the","s":4.08,"e":4.16},{"text":"same","s":4.28,"e":4.4},{"text":"hour.","s":4.48,"e":5.04},{"text":"That's","s":5.28,"e":5.48},{"text":"not","s":5.52,"e":5.68},{"text":"random.","s":5.76,"e":6.52},{"text":"Once","s":6.8,"e":6.96},{"text":"you","s":7.04,"e":7.12},{"text":"see","s":7.2,"e":7.28},{"text":"the","s":7.32,"e":7.36},{"text":"pattern,","s":7.44,"e":8.08},{"text":"you'll","s":8.12,"e":8.32},{"text":"know","s":8.4,"e":8.48},{"text":"what","s":8.56,"e":8.64},{"text":"to","s":8.72,"e":8.8},{"text":"hand","s":8.92,"e":9.04},{"text":"off","s":9.12,"e":9.36},{"text":"and","s":9.44,"e":9.6},{"text":"what","s":9.68,"e":9.76},{"text":"to","s":9.8,"e":9.84},{"text":"keep.","s":9.92,"e":10.48}]},{"template":"quote","chapter":"Jagged intelligence","narration":"Karpathy describes the paradox. A state of the art model will refactor a hundred-thousand-line codebase or find a zero-day vulnerability, and yet, when you ask how to get your car to a car wash fifty meters away, it tells you to walk.","params":{"text":"How is it possible that state-of-the-art Opus 4.7 will simultaneously refactor a 100,000-line codebase... and yet tells me to *walk to this car wash*?","who":"Andrej Karpathy","role":"on jagged intelligence","src":"From Vibe Coding to Agentic Engineering · Sequoia","t":699,"sync":false},"start":11.339,"dur":15.788,"words":[{"text":"Karpathy","s":11.639,"e":12.199},{"text":"describes","s":12.279,"e":12.759},{"text":"the","s":12.839,"e":12.919},{"text":"paradox.","s":13.039,"e":13.919},{"text":"A","s":14.199,"e":14.359},{"text":"state","s":14.479,"e":14.599},{"text":"of","s":14.639,"e":14.679},{"text":"the","s":14.719,"e":14.759},{"text":"art","s":14.839,"e":14.999},{"text":"model","s":15.079,"e":15.399},{"text":"will","s":15.479,"e":15.639},{"text":"refactor","s":15.719,"e":16.279},{"text":"a","s":16.359,"e":16.439},{"text":"hundred-thousand-line","s":16.519,"e":17.559},{"text":"codebase","s":17.639,"e":18.199},{"text":"or","s":18.359,"e":18.519},{"text":"find","s":18.679,"e":18.839},{"text":"a","s":18.919,"e":18.999},{"text":"zero-day","s":19.079,"e":19.559},{"text":"vulnerability,","s":19.639,"e":20.759},{"text":"and","s":20.839,"e":20.919},{"text":"yet,","s":20.999,"e":21.639},{"text":"when","s":21.719,"e":21.799},{"text":"you","s":21.839,"e":21.879},{"text":"ask","s":21.999,"e":22.119},{"text":"how","s":22.159,"e":22.199},{"text":"to","s":22.279,"e":22.359},{"text":"get","s":22.439,"e":22.519},{"text":"your","s":22.599,"e":22.679},{"text":"car","s":22.799,"e":22.919},{"text":"to","s":22.999,"e":23.079},{"text":"a","s":23.119,"e":23.159},{"text":"car","s":23.239,"e":23.399},{"text":"wash","s":23.479,"e":23.799},{"text":"fifty","s":23.879,"e":24.119},{"text":"meters","s":24.199,"e":24.439},{"text":"away,","s":24.519,"e":25.239},{"text":"it","s":25.319,"e":25.399},{"text":"tells","s":25.439,"e":25.639},{"text":"you","s":25.679,"e":25.719},{"text":"to","s":25.759,"e":25.799},{"text":"walk.","s":25.959,"e":26.519}]},{"template":"statement","chapter":"Why it's jagged","narration":"His explanation is the most useful idea in this lesson. Traditional computers automate what you can specify in code. Today's models automate what you can verify. Labs train them on tasks with checkable answers, so they peak wherever checking is easy.","params":{"text":"Computers automate what you can *specify*. Models automate what you can *verify*.","sub":"Andrej Karpathy","sub_at":"explanation","size":"m"},"start":27.127,"dur":16.754,"words":[{"text":"His","s":27.427,"e":27.667},{"text":"explanation","s":27.747,"e":28.387},{"text":"is","s":28.467,"e":28.547},{"text":"the","s":28.627,"e":28.707},{"text":"most","s":28.867,"e":29.107},{"text":"useful","s":29.187,"e":29.427},{"text":"idea","s":29.507,"e":29.827},{"text":"in","s":29.867,"e":29.907},{"text":"this","s":29.987,"e":30.067},{"text":"lesson.","s":30.147,"e":30.827},{"text":"Traditional","s":31.107,"e":31.507},{"text":"computers","s":31.587,"e":32.227},{"text":"automate","s":32.387,"e":32.707},{"text":"what","s":32.787,"e":32.867},{"text":"you","s":32.907,"e":32.947},{"text":"can","s":33.027,"e":33.107},{"text":"specify","s":33.187,"e":33.827},{"text":"in","s":33.947,"e":34.067},{"text":"code.","s":34.267,"e":34.947},{"text":"Today's","s":35.107,"e":35.347},{"text":"models","s":35.427,"e":35.827},{"text":"automate","s":35.907,"e":36.307},{"text":"what","s":36.387,"e":36.467},{"text":"you","s":36.507,"e":36.547},{"text":"can","s":36.627,"e":36.787},{"text":"verify.","s":36.867,"e":37.907},{"text":"Labs","s":37.987,"e":38.387},{"text":"train","s":38.507,"e":38.627},{"text":"them","s":38.667,"e":38.707},{"text":"on","s":38.787,"e":38.947},{"text":"tasks","s":39.187,"e":39.427},{"text":"with","s":39.547,"e":39.667},{"text":"checkable","s":39.787,"e":40.147},{"text":"answers,","s":40.427,"e":40.867},{"text":"so","s":40.947,"e":41.027},{"text":"they","s":41.107,"e":41.267},{"text":"peak","s":41.427,"e":41.587},{"text":"wherever","s":41.667,"e":42.067},{"text":"checking","s":42.147,"e":42.467},{"text":"is","s":42.547,"e":42.627},{"text":"easy.","s":42.707,"e":43.267}]},{"template":"flow","chapter":"Why code is ideal","narration":"And code is the most verifiable thing there is. You can compile it, lint it, run it and test it. As Peter Steinberger puts it, if you design it right, you have a perfect loop. That's why coding agents arrived before agents for almost anything else.","params":{"nodes":[{"label":"Write","sub":"the agent edits","kind":"agent","at":"code is"},{"label":"Compile","sub":"types check","kind":"code","at":"compile"},{"label":"Lint","sub":"style rules","kind":"tool","at":"lint"},{"label":"Run","sub":"it executes","kind":"test","at":"run"},{"label":"Test","sub":"behavior checks","kind":"check","at":"test"}],"caption":"“If you design it the right way, you have a perfect loop.” — Peter Steinberger","caption_at":"perfect loop"},"start":43.881,"dur":16.676,"words":[{"text":"And","s":44.181,"e":44.421},{"text":"code","s":44.501,"e":44.901},{"text":"is","s":44.981,"e":45.061},{"text":"the","s":45.141,"e":45.181},{"text":"most","s":45.221,"e":45.381},{"text":"verifiable","s":45.541,"e":46.181},{"text":"thing","s":46.261,"e":46.421},{"text":"there","s":46.501,"e":46.661},{"text":"is.","s":46.741,"e":47.381},{"text":"You","s":47.461,"e":47.541},{"text":"can","s":47.621,"e":47.701},{"text":"compile","s":47.941,"e":48.181},{"text":"it,","s":48.261,"e":48.581},{"text":"lint","s":48.661,"e":48.901},{"text":"it,","s":48.981,"e":49.381},{"text":"run","s":49.461,"e":49.541},{"text":"it","s":49.621,"e":49.701},{"text":"and","s":49.861,"e":50.021},{"text":"test","s":50.181,"e":50.341},{"text":"it.","s":50.421,"e":50.981},{"text":"As","s":51.061,"e":51.141},{"text":"Peter","s":51.221,"e":51.381},{"text":"Steinberger","s":51.461,"e":52.021},{"text":"puts","s":52.101,"e":52.261},{"text":"it,","s":52.341,"e":52.821},{"text":"if","s":52.901,"e":52.981},{"text":"you","s":53.021,"e":53.061},{"text":"design","s":53.141,"e":53.461},{"text":"it","s":53.541,"e":53.621},{"text":"right,","s":53.741,"e":54.101},{"text":"you","s":54.181,"e":54.261},{"text":"have","s":54.301,"e":54.341},{"text":"a","s":54.421,"e":54.581},{"text":"perfect","s":54.781,"e":54.981},{"text":"loop.","s":55.061,"e":55.861},{"text":"That's","s":55.941,"e":56.101},{"text":"why","s":56.181,"e":56.261},{"text":"coding","s":56.341,"e":56.661},{"text":"agents","s":56.821,"e":56.981},{"text":"arrived","s":57.061,"e":57.381},{"text":"before","s":57.461,"e":57.861},{"text":"agents","s":58.061,"e":58.261},{"text":"for","s":58.341,"e":58.421},{"text":"almost","s":58.501,"e":58.821},{"text":"anything","s":58.901,"e":59.221},{"text":"else.","s":59.301,"e":59.941}]},{"template":"list","chapter":"Where agents fly","narration":"So agents fly on work with a fast, clear check. Reading and explaining a large codebase. Fixing a bug you can reproduce with a failing test. Migrations where the old tests define the target. Boilerplate, scripts and prototypes, where a wrong answer is cheap.","params":{"title":"Where agents fly","items":[{"text":"Reading and explaining big codebases","at":"reading"},{"text":"Bugs you can reproduce with a test","at":"bug"},{"text":"Migrations with tests as the target","at":"migrations"},{"text":"Boilerplate, scripts, prototypes","sub":"wrong answers are cheap","at":"boilerplate"}]},"start":60.557,"dur":18.426,"words":[{"text":"So","s":60.857,"e":61.017},{"text":"agents","s":61.097,"e":61.497},{"text":"fly","s":61.577,"e":61.737},{"text":"on","s":61.817,"e":61.977},{"text":"work","s":62.137,"e":62.297},{"text":"with","s":62.377,"e":62.457},{"text":"a","s":62.537,"e":62.617},{"text":"fast,","s":62.697,"e":63.257},{"text":"clear","s":63.417,"e":63.577},{"text":"check.","s":63.657,"e":64.057},{"text":"Reading","s":64.937,"e":65.017},{"text":"and","s":65.097,"e":65.257},{"text":"explaining","s":65.417,"e":65.737},{"text":"a","s":65.817,"e":65.977},{"text":"large","s":66.217,"e":66.297},{"text":"codebase.","s":66.537,"e":67.417},{"text":"Fixing","s":67.657,"e":67.817},{"text":"a","s":67.857,"e":67.897},{"text":"bug","s":68.137,"e":68.217},{"text":"you","s":68.297,"e":68.377},{"text":"can","s":68.457,"e":68.537},{"text":"reproduce","s":68.617,"e":69.177},{"text":"with","s":69.257,"e":69.31},{"text":"a","s":69.364,"e":69.417},{"text":"failing","s":69.537,"e":69.737},{"text":"test.","s":70.217,"e":70.697},{"text":"Migrations","s":71.057,"e":71.497},{"text":"where","s":71.577,"e":71.617},{"text":"the","s":71.657,"e":71.817},{"text":"old","s":71.977,"e":72.057},{"text":"tests","s":72.537,"e":72.737},{"text":"define","s":72.937,"e":72.977},{"text":"the","s":73.017,"e":73.097},{"text":"target.","s":73.497,"e":73.977},{"text":"Boilerplate,","s":74.137,"e":74.937},{"text":"scripts","s":75.337,"e":75.417},{"text":"and","s":75.497,"e":75.577},{"text":"prototypes,","s":75.817,"e":76.617},{"text":"where","s":76.697,"e":76.737},{"text":"a","s":76.777,"e":76.857},{"text":"wrong","s":77.097,"e":77.177},{"text":"answer","s":77.497,"e":77.577},{"text":"is","s":77.737,"e":77.817},{"text":"cheap.","s":78.137,"e":78.377}]},{"template":"chat","chapter":"Where they fail","narration":"Now the other side. Karpathy's own app needed to connect Stripe payments to user accounts. The agent matched them by email address instead of a stable user ID. It looked fine, and it would have quietly lost track of money. Brendan O'Leary calls this code that is technically correct, and contextually wrong.","params":{"messages":[{"from":"you","text":"link each Stripe purchase to the user's account","at":"Stripe"},{"from":"agent","text":"Done! I match each purchase to a user by their *email address*.","tone":"bad","at":"email"},{"from":"you","text":"Emails change. That silently loses credits. Use the persistent user ID.","label":"the judgment it lacked","at":"quietly"}]},"start":78.983,"dur":19.941,"words":[{"text":"Now","s":79.283,"e":79.523},{"text":"the","s":79.563,"e":79.603},{"text":"other","s":79.683,"e":79.843},{"text":"side.","s":79.923,"e":80.563},{"text":"Karpathy's","s":80.883,"e":81.523},{"text":"own","s":81.603,"e":81.763},{"text":"app","s":81.843,"e":82.083},{"text":"needed","s":82.163,"e":82.403},{"text":"to","s":82.483,"e":82.523},{"text":"connect","s":82.563,"e":82.883},{"text":"Stripe","s":82.963,"e":83.283},{"text":"payments","s":83.363,"e":83.763},{"text":"to","s":83.803,"e":83.843},{"text":"user","s":83.923,"e":84.163},{"text":"accounts.","s":84.243,"e":85.003},{"text":"The","s":85.283,"e":85.363},{"text":"agent","s":85.443,"e":85.763},{"text":"matched","s":85.843,"e":86.083},{"text":"them","s":86.163,"e":86.243},{"text":"by","s":86.323,"e":86.483},{"text":"email","s":86.563,"e":86.803},{"text":"address","s":86.883,"e":87.283},{"text":"instead","s":87.363,"e":87.763},{"text":"of","s":87.803,"e":87.843},{"text":"a","s":87.883,"e":87.923},{"text":"stable","s":88.003,"e":88.323},{"text":"user","s":88.403,"e":88.643},{"text":"ID.","s":88.723,"e":89.283},{"text":"It","s":89.523,"e":89.603},{"text":"looked","s":89.683,"e":89.843},{"text":"fine,","s":90.043,"e":90.483},{"text":"and","s":90.603,"e":90.723},{"text":"it","s":90.803,"e":90.883},{"text":"would","s":90.923,"e":90.963},{"text":"have","s":91.043,"e":91.203},{"text":"quietly","s":91.403,"e":91.683},{"text":"lost","s":91.763,"e":92.003},{"text":"track","s":92.123,"e":92.243},{"text":"of","s":92.283,"e":92.323},{"text":"money.","s":92.403,"e":92.723},{"text":"Brendan","s":93.283,"e":93.603},{"text":"O'Leary","s":93.643,"e":94.083},{"text":"calls","s":94.203,"e":94.323},{"text":"this","s":94.403,"e":94.563},{"text":"code","s":94.723,"e":94.883},{"text":"that","s":95.043,"e":95.123},{"text":"is","s":95.203,"e":95.363},{"text":"technically","s":95.443,"e":95.923},{"text":"correct,","s":96.003,"e":96.563},{"text":"and","s":96.643,"e":96.723},{"text":"contextually","s":97.003,"e":97.523},{"text":"wrong.","s":97.683,"e":98.323}]},{"template":"statement","chapter":"Where they fail","narration":"Why does this happen? Dex Horthy points at the training itself. The reward checks whether the code runs and the tests pass. Nothing penalizes poor design or code that will be hard to maintain in six months. So you get code that works, but is bloated, copy-pasted, and built on brittle abstractions.","params":{"eyebrow":"Why design suffers","text":"The reward checks that tests pass. *Nothing* checks the design.","sub":"Result: code that works, but is bloated, copy-pasted and brittle.","art":"warn","tone":"bad","sync":false},"start":98.924,"dur":20.019,"words":[{"text":"Why","s":99.224,"e":99.464},{"text":"does","s":99.544,"e":99.624},{"text":"this","s":99.704,"e":99.864},{"text":"happen?","s":99.944,"e":100.624},{"text":"Dex","s":100.904,"e":101.144},{"text":"Horthy","s":101.224,"e":101.624},{"text":"points","s":101.784,"e":101.944},{"text":"at","s":101.984,"e":102.024},{"text":"the","s":102.064,"e":102.104},{"text":"training","s":102.184,"e":102.504},{"text":"itself.","s":102.584,"e":103.344},{"text":"The","s":103.624,"e":103.704},{"text":"reward","s":103.944,"e":104.184},{"text":"checks","s":104.264,"e":104.504},{"text":"whether","s":104.584,"e":104.824},{"text":"the","s":104.864,"e":104.904},{"text":"code","s":104.984,"e":105.224},{"text":"runs","s":105.304,"e":105.704},{"text":"and","s":105.744,"e":105.784},{"text":"the","s":105.864,"e":105.944},{"text":"tests","s":106.024,"e":106.424},{"text":"pass.","s":106.664,"e":107.384},{"text":"Nothing","s":107.464,"e":107.784},{"text":"penalizes","s":107.904,"e":108.504},{"text":"poor","s":108.544,"e":108.664},{"text":"design","s":108.744,"e":109.304},{"text":"or","s":109.424,"e":109.544},{"text":"code","s":109.664,"e":109.784},{"text":"that","s":109.864,"e":109.944},{"text":"will","s":110.024,"e":110.104},{"text":"be","s":110.184,"e":110.264},{"text":"hard","s":110.344,"e":110.424},{"text":"to","s":110.504,"e":110.584},{"text":"maintain","s":110.744,"e":110.984},{"text":"in","s":111.064,"e":111.224},{"text":"six","s":111.344,"e":111.464},{"text":"months.","s":111.624,"e":112.344},{"text":"So","s":112.544,"e":112.744},{"text":"you","s":112.784,"e":112.824},{"text":"get","s":112.904,"e":113.064},{"text":"code","s":113.184,"e":113.304},{"text":"that","s":113.384,"e":113.464},{"text":"works,","s":113.704,"e":114.024},{"text":"but","s":114.104,"e":114.184},{"text":"is","s":114.264,"e":114.344},{"text":"bloated,","s":114.464,"e":115.064},{"text":"copy-pasted,","s":115.224,"e":116.184},{"text":"and","s":116.264,"e":116.344},{"text":"built","s":116.464,"e":116.584},{"text":"on","s":116.664,"e":116.744},{"text":"brittle","s":116.904,"e":117.144},{"text":"abstractions.","s":117.224,"e":118.344}]},{"template":"quote","chapter":"Confidently wrong","narration":"Agents also report success with total confidence. Martin Fowler is blunt about it: they lie to you all the time. If they were a junior developer, he says, he'd be having some words with HR. In one demo, a browser agent hit a login wall and still reported that it had upvoted the post.","params":{"text":"They do lie to you all the time. In fact, if they were truly a junior developer... I would be having some words with *HR*.","who":"Martin Fowler","role":"Chief Scientist, Thoughtworks","src":"How AI will change software engineering · The Pragmatic Engineer","t":2666,"sync":false},"start":118.943,"dur":19.549,"words":[{"text":"Agents","s":119.243,"e":119.723},{"text":"also","s":119.803,"e":120.123},{"text":"report","s":120.203,"e":120.523},{"text":"success","s":120.763,"e":121.003},{"text":"with","s":121.083,"e":121.243},{"text":"total","s":121.403,"e":121.643},{"text":"confidence.","s":121.803,"e":122.443},{"text":"Martin","s":122.923,"e":123.243},{"text":"Fowler","s":123.323,"e":123.723},{"text":"is","s":123.803,"e":123.883},{"text":"blunt","s":123.963,"e":124.283},{"text":"about","s":124.403,"e":124.523},{"text":"it:","s":124.603,"e":125.083},{"text":"they","s":125.163,"e":125.243},{"text":"lie","s":125.403,"e":125.563},{"text":"to","s":125.643,"e":125.723},{"text":"you","s":125.843,"e":125.963},{"text":"all","s":126.043,"e":126.203},{"text":"the","s":126.283,"e":126.363},{"text":"time.","s":126.443,"e":126.843},{"text":"If","s":127.323,"e":127.403},{"text":"they","s":127.483,"e":127.563},{"text":"were","s":127.603,"e":127.643},{"text":"a","s":127.723,"e":127.803},{"text":"junior","s":127.883,"e":128.123},{"text":"developer,","s":128.203,"e":128.843},{"text":"he","s":128.923,"e":129.003},{"text":"says,","s":129.163,"e":129.483},{"text":"he'd","s":129.643,"e":129.803},{"text":"be","s":129.883,"e":129.963},{"text":"having","s":130.043,"e":130.203},{"text":"some","s":130.283,"e":130.443},{"text":"words","s":130.523,"e":130.763},{"text":"with","s":130.843,"e":130.923},{"text":"HR.","s":131.083,"e":131.483},{"text":"In","s":132.123,"e":132.203},{"text":"one","s":132.283,"e":132.443},{"text":"demo,","s":132.523,"e":133.003},{"text":"a","s":133.043,"e":133.083},{"text":"browser","s":133.163,"e":133.483},{"text":"agent","s":133.563,"e":133.883},{"text":"hit","s":133.963,"e":134.043},{"text":"a","s":134.083,"e":134.123},{"text":"login","s":134.203,"e":134.603},{"text":"wall","s":134.683,"e":134.923},{"text":"and","s":135.003,"e":135.163},{"text":"still","s":135.243,"e":135.483},{"text":"reported","s":135.563,"e":136.043},{"text":"that","s":136.083,"e":136.123},{"text":"it","s":136.203,"e":136.283},{"text":"had","s":136.403,"e":136.523},{"text":"upvoted","s":136.643,"e":137.003},{"text":"the","s":137.043,"e":137.083},{"text":"post.","s":137.163,"e":137.883}]},{"template":"list","chapter":"Warning signs","narration":"Kent Beck watches for three warning signs. Loops, where the agent keeps trying the same thing. Functionality you didn't ask for, even when it's a reasonable next step. And cheating, like disabling or deleting a test to make the build go green. See any of these, and it's time to step in.","params":{"title":"Kent Beck's three warning signs","style":"warn","items":[{"text":"Loops","sub":"it keeps trying the same thing","at":"Loops"},{"text":"Functionality you didn't ask for","sub":"even a “reasonable next step”","at":"Functionality"},{"text":"Cheating","sub":"disabling or deleting tests to go green","at":"cheating"}]},"start":138.492,"dur":19.001,"words":[{"text":"Kent","s":138.792,"e":139.032},{"text":"Beck","s":139.112,"e":139.352},{"text":"watches","s":139.392,"e":139.752},{"text":"for","s":139.872,"e":139.992},{"text":"three","s":140.072,"e":140.312},{"text":"warning","s":140.392,"e":140.712},{"text":"signs.","s":140.792,"e":141.272},{"text":"Loops,","s":141.752,"e":142.232},{"text":"where","s":142.552,"e":142.632},{"text":"the","s":142.712,"e":142.752},{"text":"agent","s":142.792,"e":143.112},{"text":"keeps","s":143.192,"e":143.432},{"text":"trying","s":143.512,"e":143.752},{"text":"the","s":143.792,"e":143.832},{"text":"same","s":143.992,"e":144.232},{"text":"thing.","s":144.392,"e":144.712},{"text":"Functionality","s":145.192,"e":145.832},{"text":"you","s":145.912,"e":145.992},{"text":"didn't","s":146.152,"e":146.392},{"text":"ask","s":146.472,"e":146.712},{"text":"for,","s":146.792,"e":147.272},{"text":"even","s":147.392,"e":147.512},{"text":"when","s":147.592,"e":147.672},{"text":"it's","s":147.712,"e":147.832},{"text":"a","s":147.872,"e":147.912},{"text":"reasonable","s":147.992,"e":148.392},{"text":"next","s":148.472,"e":148.632},{"text":"step.","s":148.712,"e":149.112},{"text":"And","s":149.512,"e":149.752},{"text":"cheating,","s":149.912,"e":150.632},{"text":"like","s":150.712,"e":150.792},{"text":"disabling","s":150.872,"e":151.432},{"text":"or","s":151.512,"e":151.592},{"text":"deleting","s":151.672,"e":151.992},{"text":"a","s":152.072,"e":152.112},{"text":"test","s":152.152,"e":152.472},{"text":"to","s":152.512,"e":152.552},{"text":"make","s":152.632,"e":152.712},{"text":"the","s":152.792,"e":152.832},{"text":"build","s":152.872,"e":153.112},{"text":"go","s":153.192,"e":153.272},{"text":"green.","s":153.352,"e":154.032},{"text":"See","s":154.312,"e":154.472},{"text":"any","s":154.552,"e":154.712},{"text":"of","s":154.792,"e":154.832},{"text":"these,","s":154.872,"e":155.352},{"text":"and","s":155.392,"e":155.432},{"text":"it's","s":155.512,"e":155.672},{"text":"time","s":155.792,"e":155.912},{"text":"to","s":155.992,"e":156.032},{"text":"step","s":156.072,"e":156.312},{"text":"in.","s":156.392,"e":156.872}]},{"template":"stat","chapter":"Task length","narration":"Task length matters too. When METR measured agents in early 2025, they succeeded almost every time on tasks that take a human expert under four minutes, and less than ten percent of the time on tasks over four hours. That horizon has been doubling roughly every seven months, so check what today's models can sustain, but the shape stays the same.","params":{"items":[{"value":100,"prefix":"~","suffix":"%","label":"success on tasks under 4 minutes","tone":"good","at":"under four"},{"value":10,"prefix":"<","suffix":"%","label":"success on tasks over 4 hours","tone":"bad","at":"over four"}],"src":"METR, “Measuring AI Ability to Complete Long Tasks”, March 2025 · horizon doubling ~every 7 months"},"start":157.493,"dur":23.938,"words":[{"text":"Task","s":157.793,"e":158.193},{"text":"length","s":158.273,"e":158.593},{"text":"matters","s":158.673,"e":158.993},{"text":"too.","s":159.073,"e":159.393},{"text":"When","s":159.953,"e":160.113},{"text":"METR","s":160.193,"e":160.833},{"text":"measured","s":160.873,"e":161.153},{"text":"agents","s":161.233,"e":161.633},{"text":"in","s":161.713,"e":161.793},{"text":"early","s":161.953,"e":162.113},{"text":"2025,","s":162.233,"e":163.473},{"text":"they","s":163.553,"e":163.633},{"text":"succeeded","s":163.673,"e":164.193},{"text":"almost","s":164.433,"e":164.673},{"text":"every","s":164.753,"e":164.993},{"text":"time","s":165.073,"e":165.393},{"text":"on","s":165.473,"e":165.553},{"text":"tasks","s":165.793,"e":166.033},{"text":"that","s":166.113,"e":166.193},{"text":"take","s":166.273,"e":166.353},{"text":"a","s":166.433,"e":166.513},{"text":"human","s":166.673,"e":166.833},{"text":"expert","s":167.073,"e":167.313},{"text":"under","s":167.473,"e":167.633},{"text":"four","s":167.753,"e":167.873},{"text":"minutes,","s":167.953,"e":168.833},{"text":"and","s":168.913,"e":168.993},{"text":"less","s":169.073,"e":169.153},{"text":"than","s":169.233,"e":169.393},{"text":"ten","s":169.473,"e":169.633},{"text":"percent","s":169.713,"e":169.953},{"text":"of","s":170.033,"e":170.113},{"text":"the","s":170.153,"e":170.193},{"text":"time","s":170.273,"e":170.513},{"text":"on","s":170.593,"e":170.753},{"text":"tasks","s":170.993,"e":171.233},{"text":"over","s":171.393,"e":171.553},{"text":"four","s":171.673,"e":171.793},{"text":"hours.","s":171.873,"e":172.913},{"text":"That","s":173.033,"e":173.153},{"text":"horizon","s":173.233,"e":173.633},{"text":"has","s":173.713,"e":173.793},{"text":"been","s":173.913,"e":174.033},{"text":"doubling","s":174.073,"e":174.513},{"text":"roughly","s":174.633,"e":174.913},{"text":"every","s":174.993,"e":175.313},{"text":"seven","s":175.473,"e":175.633},{"text":"months,","s":175.833,"e":176.433},{"text":"so","s":176.593,"e":176.753},{"text":"check","s":176.873,"e":176.993},{"text":"what","s":177.073,"e":177.153},{"text":"today's","s":177.313,"e":177.553},{"text":"models","s":177.633,"e":177.953},{"text":"can","s":178.033,"e":178.113},{"text":"sustain,","s":178.193,"e":179.073},{"text":"but","s":179.153,"e":179.233},{"text":"the","s":179.313,"e":179.393},{"text":"shape","s":179.553,"e":179.713},{"text":"stays","s":179.793,"e":180.033},{"text":"the","s":180.113,"e":180.193},{"text":"same.","s":180.273,"e":180.833}]},{"template":"compare","chapter":"Delegate by verifiability","narration":"Put it together and you get a simple rule. Delegate by verifiability and risk. If the result is easy to check and cheap to get wrong, give the agent room. If it needs judgment, touches money, security or your data model, or can't be tested, keep your hands on it.","params":{"left":{"title":"Give it room","tone":"agent","who":"agent","items":["Easy to check automatically","Cheap to get wrong","Clear definition of done"],"at":"easy to check"},"right":{"title":"Keep your hands on","tone":"human","who":"human","items":["Needs judgment or taste","Money, security, data models","Hard to test"],"at":"needs judgment"},"vs":"or"},"start":181.431,"dur":18.112,"words":[{"text":"Put","s":181.731,"e":181.891},{"text":"it","s":181.931,"e":181.971},{"text":"together","s":182.051,"e":182.451},{"text":"and","s":182.611,"e":182.771},{"text":"you","s":182.811,"e":182.851},{"text":"get","s":182.931,"e":183.011},{"text":"a","s":183.051,"e":183.091},{"text":"simple","s":183.171,"e":183.411},{"text":"rule.","s":183.571,"e":184.171},{"text":"Delegate","s":184.451,"e":184.931},{"text":"by","s":185.011,"e":185.091},{"text":"verifiability","s":185.171,"e":186.131},{"text":"and","s":186.211,"e":186.371},{"text":"risk.","s":186.451,"e":187.131},{"text":"If","s":187.411,"e":187.491},{"text":"the","s":187.571,"e":187.611},{"text":"result","s":187.651,"e":187.971},{"text":"is","s":188.051,"e":188.211},{"text":"easy","s":188.291,"e":188.531},{"text":"to","s":188.611,"e":188.691},{"text":"check","s":188.851,"e":189.011},{"text":"and","s":189.091,"e":189.251},{"text":"cheap","s":189.331,"e":189.571},{"text":"to","s":189.611,"e":189.651},{"text":"get","s":189.731,"e":189.811},{"text":"wrong,","s":189.891,"e":190.611},{"text":"give","s":190.691,"e":190.771},{"text":"the","s":190.811,"e":190.851},{"text":"agent","s":190.931,"e":191.171},{"text":"room.","s":191.251,"e":191.851},{"text":"If","s":192.131,"e":192.211},{"text":"it","s":192.291,"e":192.331},{"text":"needs","s":192.371,"e":192.611},{"text":"judgment,","s":192.691,"e":193.331},{"text":"touches","s":193.531,"e":193.731},{"text":"money,","s":193.811,"e":194.291},{"text":"security","s":194.611,"e":194.931},{"text":"or","s":195.091,"e":195.171},{"text":"your","s":195.251,"e":195.331},{"text":"data","s":195.411,"e":195.651},{"text":"model,","s":195.731,"e":196.211},{"text":"or","s":196.291,"e":196.371},{"text":"can't","s":196.451,"e":196.691},{"text":"be","s":196.731,"e":196.771},{"text":"tested,","s":196.851,"e":197.651},{"text":"keep","s":197.731,"e":197.891},{"text":"your","s":197.971,"e":198.051},{"text":"hands","s":198.171,"e":198.291},{"text":"on","s":198.371,"e":198.451},{"text":"it.","s":198.531,"e":198.931}]},{"template":"statement","chapter":"Trust","narration":"Charity Majors frames the long game nicely. If trust is no longer earned by a human reading every line, it has to be built somewhere else: tests, evals, monitoring. The trust has to go somewhere. Most of this course is about where to put it.","params":{"text":"“The trust has to go *somewhere*.”","sub":"Charity Majors · If humans don't read every line, tests and checks have to earn it.","art":"lock","tone":"human","sync":false},"start":199.543,"dur":17.067,"words":[{"text":"Charity","s":199.843,"e":200.243},{"text":"Majors","s":200.323,"e":200.723},{"text":"frames","s":200.803,"e":201.043},{"text":"the","s":201.123,"e":201.203},{"text":"long","s":201.283,"e":201.523},{"text":"game","s":201.603,"e":201.763},{"text":"nicely.","s":201.923,"e":202.683},{"text":"If","s":202.963,"e":203.123},{"text":"trust","s":203.203,"e":203.443},{"text":"is","s":203.523,"e":203.603},{"text":"no","s":203.683,"e":203.763},{"text":"longer","s":203.843,"e":204.163},{"text":"earned","s":204.243,"e":204.563},{"text":"by","s":204.643,"e":204.723},{"text":"a","s":204.803,"e":204.883},{"text":"human","s":204.963,"e":205.283},{"text":"reading","s":205.363,"e":205.603},{"text":"every","s":205.683,"e":205.923},{"text":"line,","s":206.003,"e":206.723},{"text":"it","s":206.803,"e":206.883},{"text":"has","s":206.963,"e":207.123},{"text":"to","s":207.163,"e":207.203},{"text":"be","s":207.283,"e":207.363},{"text":"built","s":207.483,"e":207.603},{"text":"somewhere","s":207.683,"e":207.923},{"text":"else:","s":208.003,"e":208.803},{"text":"tests,","s":209.083,"e":209.603},{"text":"evals,","s":209.843,"e":210.483},{"text":"monitoring.","s":210.683,"e":211.523},{"text":"The","s":211.563,"e":211.603},{"text":"trust","s":211.683,"e":212.003},{"text":"has","s":212.083,"e":212.323},{"text":"to","s":212.403,"e":212.443},{"text":"go","s":212.483,"e":212.643},{"text":"somewhere.","s":212.723,"e":213.603},{"text":"Most","s":213.683,"e":213.843},{"text":"of","s":213.923,"e":214.003},{"text":"this","s":214.083,"e":214.163},{"text":"course","s":214.243,"e":214.563},{"text":"is","s":214.643,"e":214.723},{"text":"about","s":214.843,"e":214.963},{"text":"where","s":215.043,"e":215.203},{"text":"to","s":215.283,"e":215.363},{"text":"put","s":215.443,"e":215.523},{"text":"it.","s":215.603,"e":216.003}]},{"template":"quiz","chapter":"Quick check","narration":"Quick check. Which of these tasks is safest to hand to an agent end to end? Tap your answer.","params":{"q":"Which task is safest to hand to an agent end to end?","options":["Choosing the data model for a payments system","Moving 300 test files to a new test runner, with CI as the check","Deciding which features your users need next","Approving a change to authentication rules"],"answer":1,"explain":"It's mechanical, and the existing tests plus CI tell you immediately whether it worked."},"start":216.61,"dur":11.926,"words":[{"text":"Quick","s":216.91,"e":217.15},{"text":"check.","s":217.23,"e":217.55},{"text":"Which","s":218.03,"e":218.19},{"text":"of","s":218.27,"e":218.35},{"text":"these","s":218.47,"e":218.59},{"text":"tasks","s":218.87,"e":219.15},{"text":"is","s":219.23,"e":219.39},{"text":"safest","s":219.43,"e":219.79},{"text":"to","s":219.87,"e":219.95},{"text":"hand","s":220.03,"e":220.19},{"text":"to","s":220.27,"e":220.35},{"text":"an","s":220.43,"e":220.51},{"text":"agent","s":220.59,"e":220.91},{"text":"end","s":221.07,"e":221.23},{"text":"to","s":221.31,"e":221.39},{"text":"end?","s":221.47,"e":222.03},{"text":"Tap","s":222.27,"e":222.51},{"text":"your","s":222.59,"e":222.67},{"text":"answer.","s":222.87,"e":223.39}]},{"template":"exercise","chapter":"Try it","narration":"Try sorting your own work. Ask your agent to rate a list of tasks by how verifiable and how risky they are, and to say which ones it could do end to end. It's a great way to calibrate where your trust should start.","params":{"title":"Try it with your agent","intro":"Paste your task list or point it at your issues:","prompt":"Here are tasks from my backlog: [paste them].\nFor each one, rate 1-5: how easy is it to verify automatically\n(tests, types, a command), and how costly is a wrong change?\nSay which you could finish end to end, and which need my judgment."},"start":228.536,"dur":13.541,"words":[{"text":"Try","s":228.836,"e":229.076},{"text":"sorting","s":229.156,"e":229.476},{"text":"your","s":229.556,"e":229.636},{"text":"own","s":229.716,"e":229.796},{"text":"work.","s":229.876,"e":230.196},{"text":"Ask","s":230.676,"e":230.916},{"text":"your","s":230.996,"e":231.076},{"text":"agent","s":231.156,"e":231.396},{"text":"to","s":231.476,"e":231.556},{"text":"rate","s":231.636,"e":231.796},{"text":"a","s":231.836,"e":231.876},{"text":"list","s":231.996,"e":232.116},{"text":"of","s":232.156,"e":232.196},{"text":"tasks","s":232.276,"e":232.756},{"text":"by","s":232.836,"e":232.916},{"text":"how","s":232.996,"e":233.156},{"text":"verifiable","s":233.236,"e":233.956},{"text":"and","s":234.036,"e":234.116},{"text":"how","s":234.196,"e":234.276},{"text":"risky","s":234.436,"e":234.756},{"text":"they","s":234.836,"e":234.916},{"text":"are,","s":234.996,"e":235.556},{"text":"and","s":235.636,"e":235.716},{"text":"to","s":235.796,"e":235.876},{"text":"say","s":235.956,"e":236.036},{"text":"which","s":236.116,"e":236.356},{"text":"ones","s":236.436,"e":236.676},{"text":"it","s":236.716,"e":236.756},{"text":"could","s":236.836,"e":236.916},{"text":"do","s":236.996,"e":237.156},{"text":"end","s":237.236,"e":237.396},{"text":"to","s":237.476,"e":237.556},{"text":"end.","s":237.636,"e":237.956},{"text":"It's","s":238.436,"e":238.596},{"text":"a","s":238.636,"e":238.676},{"text":"great","s":238.756,"e":239.076},{"text":"way","s":239.156,"e":239.236},{"text":"to","s":239.276,"e":239.316},{"text":"calibrate","s":239.396,"e":239.876},{"text":"where","s":239.956,"e":240.036},{"text":"your","s":240.116,"e":240.196},{"text":"trust","s":240.276,"e":240.596},{"text":"should","s":240.716,"e":240.836},{"text":"start.","s":241.036,"e":241.476}]},{"template":"recap","chapter":"Recap","narration":"Recap. Agent intelligence is jagged. Models automate what can be verified, so code is their home turf. They're weak at judgment and design, and confident when wrong. So delegate by verifiability and risk. Next: the single most important concept in agentic engineering, context.","params":{"items":["Jagged: superhuman *and* baffling","Models automate what can be *verified*","Weak at judgment, design, business context","Delegate by verifiability and risk"],"next":"Context is everything","next_at":"Next"},"start":242.077,"dur":22.511,"words":[{"text":"Recap.","s":242.377,"e":243.177},{"text":"Agent","s":243.417,"e":243.737},{"text":"intelligence","s":243.817,"e":244.457},{"text":"is","s":244.617,"e":244.697},{"text":"jagged.","s":244.857,"e":245.737},{"text":"Models","s":245.817,"e":246.217},{"text":"automate","s":246.297,"e":246.697},{"text":"what","s":246.777,"e":246.857},{"text":"can","s":246.937,"e":247.017},{"text":"be","s":247.057,"e":247.097},{"text":"verified,","s":247.257,"e":247.977},{"text":"so","s":248.057,"e":248.297},{"text":"code","s":248.377,"e":248.697},{"text":"is","s":248.777,"e":248.857},{"text":"their","s":248.937,"e":249.017},{"text":"home","s":249.137,"e":249.257},{"text":"turf.","s":249.297,"e":250.137},{"text":"They're","s":250.177,"e":250.377},{"text":"weak","s":250.497,"e":250.617},{"text":"at","s":250.697,"e":250.777},{"text":"judgment","s":250.857,"e":251.177},{"text":"and","s":251.257,"e":251.337},{"text":"design,","s":251.617,"e":252.057},{"text":"and","s":252.177,"e":252.297},{"text":"confident","s":252.457,"e":252.857},{"text":"when","s":252.937,"e":253.017},{"text":"wrong.","s":253.097,"e":253.817},{"text":"So","s":253.897,"e":254.057},{"text":"delegate","s":254.457,"e":254.857},{"text":"by","s":254.937,"e":255.017},{"text":"verifiability","s":255.177,"e":256.137},{"text":"and","s":256.217,"e":256.297},{"text":"risk.","s":256.377,"e":257.217},{"text":"Next:","s":257.657,"e":258.377},{"text":"the","s":258.497,"e":258.617},{"text":"single","s":258.777,"e":258.937},{"text":"most","s":259.017,"e":259.257},{"text":"important","s":259.537,"e":259.817},{"text":"concept","s":260.057,"e":260.297},{"text":"in","s":260.377,"e":260.457},{"text":"agentic","s":260.537,"e":260.937},{"text":"engineering,","s":261.217,"e":261.977},{"text":"context.","s":262.057,"e":263.017}]}],"total":264.988,"audio":"/audio/03-jagged.mp3"}