{"id":"08-verification","number":8,"slug":"verification-closing-the-loop","title":"Close the loop: let agents check their work","short":"Close the loop","module":"Working with one agent","summary":"Ask people who ship with agents for their single best tip and you hear the same thing: give the agent a way to check its own work. Tests, types, a running app, a browser. Without a check, “looks done” is the only signal, and you become the bottleneck.","voice":"Eric","scenes":[{"template":"title","chapter":"Intro","narration":"Ask people who ship real software with agents for their single best tip, and a surprising number give you the same answer. This lesson is that answer.","params":{"kicker":"Lesson 8","module":"Module 2 · Working with one agent","title":"Close the loop: let agents *check their work*","subtitle":"The habit that makes agents trustworthy.","bubble":"Let me run that again.","mood":"work"},"start":0.5,"dur":10.447,"words":[{"text":"Ask","s":0.8,"e":1.12},{"text":"people","s":1.2,"e":1.44},{"text":"who","s":1.52,"e":1.6},{"text":"ship","s":1.68,"e":1.92},{"text":"real","s":2,"e":2.24},{"text":"software","s":2.32,"e":2.8},{"text":"with","s":2.88,"e":2.96},{"text":"agents","s":3.12,"e":3.44},{"text":"for","s":3.52,"e":3.6},{"text":"their","s":3.72,"e":3.84},{"text":"single","s":3.92,"e":4.16},{"text":"best","s":4.32,"e":4.56},{"text":"tip,","s":4.72,"e":5.12},{"text":"and","s":5.2,"e":5.28},{"text":"a","s":5.32,"e":5.36},{"text":"surprising","s":5.48,"e":6},{"text":"number","s":6.08,"e":6.32},{"text":"give","s":6.4,"e":6.56},{"text":"you","s":6.6,"e":6.64},{"text":"the","s":6.72,"e":6.88},{"text":"same","s":7,"e":7.12},{"text":"answer.","s":7.2,"e":8.16},{"text":"This","s":8.24,"e":8.4},{"text":"lesson","s":8.48,"e":8.96},{"text":"is","s":9.04,"e":9.2},{"text":"that","s":9.28,"e":9.36},{"text":"answer.","s":9.44,"e":10.08}]},{"template":"quote","chapter":"The big secret","narration":"Peter Steinberger, creator of OpenClaw, calls it the big secret. To be effective with a coding agent, you have to close the loop. It needs to be able to debug and test itself. Boris Cherny says the same: give it a tool to check its work, and it will iterate by itself.","params":{"text":"You have to *close the loop*. It needs to be able to debug and test itself. That's the big secret.","who":"Peter Steinberger","role":"creator of OpenClaw","src":"The creator of OpenClaw · The Pragmatic Engineer","t":3447},"start":10.947,"dur":18.661,"words":[{"text":"Peter","s":11.247,"e":11.487},{"text":"Steinberger,","s":11.567,"e":12.447},{"text":"creator","s":12.527,"e":12.847},{"text":"of","s":12.927,"e":13.007},{"text":"OpenClaw,","s":13.087,"e":13.887},{"text":"calls","s":13.967,"e":14.207},{"text":"it","s":14.287,"e":14.367},{"text":"the","s":14.527,"e":14.607},{"text":"big","s":14.767,"e":15.007},{"text":"secret.","s":15.167,"e":15.647},{"text":"To","s":16.207,"e":16.247},{"text":"be","s":16.287,"e":16.367},{"text":"effective","s":16.607,"e":16.847},{"text":"with","s":16.927,"e":17.007},{"text":"a","s":17.047,"e":17.087},{"text":"coding","s":17.167,"e":17.407},{"text":"agent,","s":17.487,"e":18.047},{"text":"you","s":18.127,"e":18.207},{"text":"have","s":18.287,"e":18.447},{"text":"to","s":18.527,"e":18.607},{"text":"close","s":18.687,"e":19.087},{"text":"the","s":19.167,"e":19.247},{"text":"loop.","s":19.327,"e":19.647},{"text":"It","s":20.127,"e":20.207},{"text":"needs","s":20.287,"e":20.447},{"text":"to","s":20.527,"e":20.567},{"text":"be","s":20.607,"e":20.687},{"text":"able","s":20.767,"e":20.927},{"text":"to","s":21.007,"e":21.087},{"text":"debug","s":21.167,"e":21.647},{"text":"and","s":21.687,"e":21.727},{"text":"test","s":21.887,"e":22.127},{"text":"itself.","s":22.207,"e":22.847},{"text":"Boris","s":23.487,"e":23.807},{"text":"Cherny","s":23.887,"e":24.287},{"text":"says","s":24.367,"e":24.447},{"text":"the","s":24.527,"e":24.607},{"text":"same:","s":24.687,"e":25.487},{"text":"give","s":25.567,"e":25.647},{"text":"it","s":25.687,"e":25.727},{"text":"a","s":25.807,"e":25.887},{"text":"tool","s":26.007,"e":26.127},{"text":"to","s":26.207,"e":26.287},{"text":"check","s":26.367,"e":26.447},{"text":"its","s":26.527,"e":26.687},{"text":"work,","s":26.767,"e":27.167},{"text":"and","s":27.207,"e":27.247},{"text":"it","s":27.327,"e":27.367},{"text":"will","s":27.407,"e":27.567},{"text":"iterate","s":27.647,"e":27.967},{"text":"by","s":28.047,"e":28.127},{"text":"itself.","s":28.247,"e":29.007}]},{"template":"loop","chapter":"The big secret","narration":"Here's what a closed loop looks like. The agent makes a change. It runs a check: tests, types, the app itself. It reads the result. And it fixes what's wrong, then runs the check again, until it passes. No human needed in the middle.","params":{"steps":[{"label":"Change","sub":"edit the code","icon":"code","at":"change"},{"label":"Check","sub":"tests, types, the app","icon":"test","at":"runs a check"},{"label":"Read","sub":"what actually happened","icon":"eye","at":"reads"},{"label":"Fix","sub":"and check again","icon":"tool","at":"fixes"}],"center":{"label":"until green","agent":true,"at":"until it passes"},"period":2.6},"start":29.608,"dur":16.911,"words":[{"text":"Here's","s":29.908,"e":30.148},{"text":"what","s":30.228,"e":30.308},{"text":"a","s":30.348,"e":30.388},{"text":"closed","s":30.468,"e":30.788},{"text":"loop","s":30.868,"e":31.028},{"text":"looks","s":31.108,"e":31.348},{"text":"like.","s":31.508,"e":31.908},{"text":"The","s":32.148,"e":32.228},{"text":"agent","s":32.308,"e":32.628},{"text":"makes","s":32.708,"e":32.868},{"text":"a","s":32.948,"e":32.988},{"text":"change.","s":33.028,"e":33.708},{"text":"It","s":33.908,"e":34.068},{"text":"runs","s":34.188,"e":34.308},{"text":"a","s":34.348,"e":34.388},{"text":"check:","s":34.468,"e":35.188},{"text":"tests,","s":35.468,"e":35.988},{"text":"types,","s":36.228,"e":36.708},{"text":"the","s":36.748,"e":36.788},{"text":"app","s":36.868,"e":37.028},{"text":"itself.","s":37.108,"e":37.828},{"text":"It","s":38.068,"e":38.148},{"text":"reads","s":38.228,"e":38.468},{"text":"the","s":38.548,"e":38.588},{"text":"result.","s":38.628,"e":39.308},{"text":"And","s":39.508,"e":39.748},{"text":"it","s":39.908,"e":39.988},{"text":"fixes","s":40.148,"e":40.468},{"text":"what's","s":40.548,"e":40.708},{"text":"wrong,","s":40.868,"e":41.268},{"text":"then","s":41.388,"e":41.508},{"text":"runs","s":41.628,"e":41.748},{"text":"the","s":41.828,"e":41.868},{"text":"check","s":41.908,"e":42.068},{"text":"again,","s":42.148,"e":42.708},{"text":"until","s":42.868,"e":43.028},{"text":"it","s":43.068,"e":43.108},{"text":"passes.","s":43.188,"e":43.748},{"text":"No","s":44.228,"e":44.388},{"text":"human","s":44.468,"e":44.788},{"text":"needed","s":44.868,"e":45.108},{"text":"in","s":45.188,"e":45.228},{"text":"the","s":45.268,"e":45.308},{"text":"middle.","s":45.348,"e":45.908}]},{"template":"statement","chapter":"You are the loop","narration":"Without a check, Anthropic's guide warns, looks done is the only signal available, and you become the verification loop. Every mistake waits for you to notice it. Lauren Tan puts it more bluntly: without a way to verify, you are the bottleneck, and you can't run agents in parallel.","params":{"text":"No check? Then *you* are the verification loop.","sub":"Every mistake waits for you to notice it, and you become the bottleneck.","art":"human","tone":"human","sync":false},"start":46.519,"dur":19.236,"words":[{"text":"Without","s":46.819,"e":47.139},{"text":"a","s":47.179,"e":47.219},{"text":"check,","s":47.299,"e":47.939},{"text":"Anthropic's","s":48.059,"e":48.499},{"text":"guide","s":48.579,"e":48.819},{"text":"warns,","s":48.899,"e":49.299},{"text":"looks","s":49.779,"e":50.019},{"text":"done","s":50.099,"e":50.339},{"text":"is","s":50.419,"e":50.499},{"text":"the","s":50.579,"e":50.659},{"text":"only","s":50.819,"e":51.139},{"text":"signal","s":51.219,"e":51.539},{"text":"available,","s":51.819,"e":52.419},{"text":"and","s":52.499,"e":52.659},{"text":"you","s":52.739,"e":52.979},{"text":"become","s":53.059,"e":53.299},{"text":"the","s":53.379,"e":53.419},{"text":"verification","s":53.459,"e":54.179},{"text":"loop.","s":54.259,"e":54.779},{"text":"Every","s":55.059,"e":55.299},{"text":"mistake","s":55.459,"e":55.859},{"text":"waits","s":56.019,"e":56.259},{"text":"for","s":56.339,"e":56.419},{"text":"you","s":56.459,"e":56.499},{"text":"to","s":56.579,"e":56.619},{"text":"notice","s":56.659,"e":56.979},{"text":"it.","s":57.059,"e":57.219},{"text":"Lauren","s":57.939,"e":58.339},{"text":"Tan","s":58.499,"e":58.579},{"text":"puts","s":58.739,"e":58.779},{"text":"it","s":58.819,"e":58.899},{"text":"more","s":58.979,"e":59.059},{"text":"bluntly:","s":59.139,"e":60.179},{"text":"without","s":60.339,"e":60.379},{"text":"a","s":60.419,"e":60.459},{"text":"way","s":60.499,"e":60.579},{"text":"to","s":60.659,"e":60.939},{"text":"verify,","s":61.219,"e":61.619},{"text":"you","s":61.859,"e":61.939},{"text":"are","s":62.019,"e":62.059},{"text":"the","s":62.099,"e":62.179},{"text":"bottleneck,","s":62.259,"e":63.019},{"text":"and","s":63.059,"e":63.139},{"text":"you","s":63.219,"e":63.299},{"text":"can't","s":63.459,"e":63.619},{"text":"run","s":63.779,"e":63.859},{"text":"agents","s":64.179,"e":64.259},{"text":"in","s":64.299,"e":64.339},{"text":"parallel.","s":64.819,"e":65.139}]},{"template":"list","chapter":"Kinds of checks","narration":"Checks come in many shapes. Unit and integration tests. Type checkers and linters. Running the app and reading its output or logs. A real browser the agent can click through and screenshot. And comparisons against ground truth, like expected output or a known-good page.","params":{"title":"Ways an agent can check its work","items":[{"text":"Tests","sub":"unit, integration, end to end","at":"tests"},{"text":"Types and linters","sub":"fast, cheap, always on","at":"type"},{"text":"Run it","sub":"the app, a CLI, the logs","at":"running"},{"text":"A real browser","sub":"click, screenshot, compare","at":"browser"},{"text":"Ground truth","sub":"expected output, a known-good page","at":"ground truth"}]},"start":65.755,"dur":19.079,"words":[{"text":"Checks","s":66.055,"e":66.375},{"text":"come","s":66.455,"e":66.615},{"text":"in","s":66.695,"e":66.855},{"text":"many","s":66.935,"e":67.175},{"text":"shapes.","s":67.255,"e":67.735},{"text":"Unit","s":68.375,"e":68.695},{"text":"and","s":68.775,"e":68.935},{"text":"integration","s":69.015,"e":69.575},{"text":"tests.","s":69.655,"e":70.335},{"text":"Type","s":70.535,"e":70.775},{"text":"checkers","s":70.855,"e":71.255},{"text":"and","s":71.295,"e":71.335},{"text":"linters.","s":71.415,"e":72.135},{"text":"Running","s":72.375,"e":72.615},{"text":"the","s":72.655,"e":72.695},{"text":"app","s":72.855,"e":73.015},{"text":"and","s":73.095,"e":73.255},{"text":"reading","s":73.335,"e":73.575},{"text":"its","s":73.655,"e":73.735},{"text":"output","s":73.815,"e":74.215},{"text":"or","s":74.295,"e":74.375},{"text":"logs.","s":74.455,"e":75.175},{"text":"A","s":75.415,"e":75.495},{"text":"real","s":75.655,"e":75.895},{"text":"browser","s":75.975,"e":76.375},{"text":"the","s":76.455,"e":76.535},{"text":"agent","s":76.615,"e":76.855},{"text":"can","s":76.935,"e":77.095},{"text":"click","s":77.215,"e":77.335},{"text":"through","s":77.415,"e":77.655},{"text":"and","s":77.695,"e":77.735},{"text":"screenshot.","s":77.815,"e":78.855},{"text":"And","s":78.975,"e":79.095},{"text":"comparisons","s":79.255,"e":79.895},{"text":"against","s":80.055,"e":80.215},{"text":"ground","s":80.295,"e":80.615},{"text":"truth,","s":80.695,"e":81.175},{"text":"like","s":81.295,"e":81.415},{"text":"expected","s":81.495,"e":81.975},{"text":"output","s":82.055,"e":82.455},{"text":"or","s":82.535,"e":82.695},{"text":"a","s":82.775,"e":82.855},{"text":"known-good","s":82.935,"e":83.415},{"text":"page.","s":83.495,"e":84.135}]},{"template":"terminal","chapter":"Red, then green","narration":"The strongest pattern is test first. Simon Willison's shorthand is: use red green TDD. Write a test, watch it fail, that's red. Then write just enough code to make it pass, that's green. Every good model understands those few words.","params":{"title":"~/shop · red/green TDD","lines":[{"k":"you","text":"add a 10% discount for orders over $100. use red/green TDD","at":"red green"},{"k":"tool","text":"Edit(src/discount.test.ts)  + 'applies 10% over $100'","at":"Write a test"},{"k":"tool","text":"Bash(npm test discount)"},{"k":"err","text":"FAIL  applies 10% over $100  (expected 108, got 120)","at":"watch it fail"},{"k":"tool","text":"Edit(src/discount.ts)","at":"just enough"},{"k":"tool","text":"Bash(npm test)"},{"k":"ok","text":"PASS  48 tests  (0.4s)","at":"green"}]},"start":84.834,"dur":18.426,"words":[{"text":"The","s":85.134,"e":85.294},{"text":"strongest","s":85.494,"e":85.774},{"text":"pattern","s":85.854,"e":86.254},{"text":"is","s":86.334,"e":86.494},{"text":"test","s":86.574,"e":86.894},{"text":"first.","s":86.974,"e":87.694},{"text":"Simon","s":88.014,"e":88.334},{"text":"Willison's","s":88.414,"e":88.894},{"text":"shorthand","s":88.934,"e":89.374},{"text":"is:","s":89.534,"e":89.774},{"text":"use","s":90.174,"e":90.414},{"text":"red","s":90.494,"e":90.734},{"text":"green","s":90.894,"e":91.214},{"text":"TDD.","s":91.374,"e":92.014},{"text":"Write","s":92.574,"e":92.734},{"text":"a","s":92.774,"e":92.814},{"text":"test,","s":92.894,"e":93.614},{"text":"watch","s":93.734,"e":93.854},{"text":"it","s":93.934,"e":94.014},{"text":"fail,","s":94.094,"e":94.734},{"text":"that's","s":94.814,"e":94.974},{"text":"red.","s":95.054,"e":95.374},{"text":"Then","s":95.934,"e":96.174},{"text":"write","s":96.254,"e":96.494},{"text":"just","s":96.574,"e":96.814},{"text":"enough","s":96.854,"e":97.054},{"text":"code","s":97.134,"e":97.294},{"text":"to","s":97.374,"e":97.454},{"text":"make","s":97.534,"e":97.614},{"text":"it","s":97.654,"e":97.694},{"text":"pass,","s":97.774,"e":98.494},{"text":"that's","s":98.574,"e":98.734},{"text":"green.","s":98.814,"e":99.454},{"text":"Every","s":99.774,"e":100.014},{"text":"good","s":100.094,"e":100.254},{"text":"model","s":100.334,"e":100.654},{"text":"understands","s":100.734,"e":101.294},{"text":"those","s":101.374,"e":101.614},{"text":"few","s":101.694,"e":101.854},{"text":"words.","s":101.934,"e":102.654}]},{"template":"chat","chapter":"Why test first","narration":"Why does order matter? If the agent writes the code first, it tends to write tests that simply match whatever it wrote. And when it's stuck, it may quietly change the test instead of the code. Kent Beck once found an agent had replaced his logic with a lookup table of expected answers.","params":{"messages":[{"from":"agent","text":"The test expects 42 but we return 41. I'll update the test to expect 41.","tone":"bad","at":"change the test"},{"from":"you","text":"No. The expected value is the spec. Fix the code.","label":"hold the line","at":"instead of the code"},{"from":"agent","text":"Understood. Found it: rounding happens too early.","tone":"good","at":"lookup table"}]},"start":103.26,"dur":19.314,"words":[{"text":"Why","s":103.56,"e":103.88},{"text":"does","s":104,"e":104.12},{"text":"order","s":104.28,"e":104.52},{"text":"matter?","s":104.68,"e":105.08},{"text":"If","s":105.72,"e":105.88},{"text":"the","s":105.92,"e":105.96},{"text":"agent","s":106.04,"e":106.36},{"text":"writes","s":106.52,"e":106.68},{"text":"the","s":106.72,"e":106.76},{"text":"code","s":106.84,"e":107.08},{"text":"first,","s":107.24,"e":107.88},{"text":"it","s":107.96,"e":108.04},{"text":"tends","s":108.08,"e":108.36},{"text":"to","s":108.4,"e":108.44},{"text":"write","s":108.52,"e":108.68},{"text":"tests","s":108.76,"e":109.24},{"text":"that","s":109.32,"e":109.4},{"text":"simply","s":109.48,"e":109.88},{"text":"match","s":110.04,"e":110.2},{"text":"whatever","s":110.36,"e":110.52},{"text":"it","s":110.6,"e":110.68},{"text":"wrote.","s":110.72,"e":111.56},{"text":"And","s":111.6,"e":111.64},{"text":"when","s":111.72,"e":111.8},{"text":"it's","s":111.84,"e":111.96},{"text":"stuck,","s":112.04,"e":112.68},{"text":"it","s":112.76,"e":112.84},{"text":"may","s":112.92,"e":113.08},{"text":"quietly","s":113.16,"e":113.64},{"text":"change","s":113.72,"e":113.96},{"text":"the","s":114.04,"e":114.2},{"text":"test","s":114.44,"e":114.68},{"text":"instead","s":114.88,"e":115.08},{"text":"of","s":115.12,"e":115.16},{"text":"the","s":115.24,"e":115.32},{"text":"code.","s":115.52,"e":116.36},{"text":"Kent","s":116.48,"e":116.6},{"text":"Beck","s":116.68,"e":116.92},{"text":"once","s":117.04,"e":117.16},{"text":"found","s":117.28,"e":117.4},{"text":"an","s":117.48,"e":117.56},{"text":"agent","s":117.64,"e":117.88},{"text":"had","s":117.96,"e":118.04},{"text":"replaced","s":118.12,"e":118.6},{"text":"his","s":118.68,"e":118.76},{"text":"logic","s":118.84,"e":119.32},{"text":"with","s":119.4,"e":119.48},{"text":"a","s":119.56,"e":119.64},{"text":"lookup","s":119.72,"e":120.04},{"text":"table","s":120.2,"e":120.36},{"text":"of","s":120.44,"e":120.52},{"text":"expected","s":120.8,"e":121.08},{"text":"answers.","s":121.16,"e":121.96}]},{"template":"quote","chapter":"Tests aren't enough","narration":"But be careful: passing tests aren't proof. Simon Willison again: just because code passes tests doesn't mean it works as intended. The app can still crash on startup, or miss a button. So ask the agent to run the real thing, and show you.","params":{"text":"Just because code passes tests doesn't mean it *works as intended*.","who":"Simon Willison","role":"creator of Datasette","src":"Agentic manual testing · Agentic Engineering Patterns"},"start":122.574,"dur":16.911,"words":[{"text":"But","s":122.874,"e":123.034},{"text":"be","s":123.114,"e":123.194},{"text":"careful:","s":123.354,"e":124.154},{"text":"passing","s":124.274,"e":124.554},{"text":"tests","s":124.794,"e":125.034},{"text":"aren't","s":125.114,"e":125.354},{"text":"proof.","s":125.594,"e":126.154},{"text":"Simon","s":126.474,"e":126.794},{"text":"Willison","s":126.874,"e":127.194},{"text":"again:","s":127.274,"e":128.074},{"text":"just","s":128.154,"e":128.314},{"text":"because","s":128.394,"e":128.634},{"text":"code","s":128.714,"e":128.954},{"text":"passes","s":129.034,"e":129.434},{"text":"tests","s":129.514,"e":130.074},{"text":"doesn't","s":130.154,"e":130.474},{"text":"mean","s":130.554,"e":130.634},{"text":"it","s":130.714,"e":130.794},{"text":"works","s":130.914,"e":131.034},{"text":"as","s":131.114,"e":131.194},{"text":"intended.","s":131.274,"e":132.234},{"text":"The","s":132.314,"e":132.394},{"text":"app","s":132.474,"e":132.554},{"text":"can","s":132.634,"e":132.794},{"text":"still","s":132.874,"e":133.114},{"text":"crash","s":133.194,"e":133.434},{"text":"on","s":133.514,"e":133.594},{"text":"startup,","s":133.674,"e":134.314},{"text":"or","s":134.394,"e":134.474},{"text":"miss","s":134.634,"e":134.714},{"text":"a","s":134.794,"e":134.874},{"text":"button.","s":135.034,"e":135.754},{"text":"So","s":135.914,"e":136.074},{"text":"ask","s":136.154,"e":136.314},{"text":"the","s":136.394,"e":136.474},{"text":"agent","s":136.634,"e":136.794},{"text":"to","s":136.874,"e":136.954},{"text":"run","s":137.034,"e":137.114},{"text":"the","s":137.194,"e":137.274},{"text":"real","s":137.434,"e":137.594},{"text":"thing,","s":137.794,"e":138.047},{"text":"and","s":138.101,"e":138.154},{"text":"show","s":138.234,"e":138.394},{"text":"you.","s":138.474,"e":138.874}]},{"template":"flow","chapter":"Tests aren't enough","narration":"Anthropic saw this in their long-running agent experiments. The agent kept marking features complete after unit tests passed, without noticing they didn't work end to end. Once it was told to use browser automation and test like a human user, results improved dramatically.","params":{"nodes":[{"label":"Unit tests pass","sub":"“feature complete!”","kind":"warn","at":"unit tests"},{"label":"Open a real browser","sub":"click like a user","kind":"web","at":"browser automation"},{"label":"Find the real bug","sub":"invisible from the code","kind":"agent","at":"human user"},{"label":"Actually done","sub":"seen working","kind":"check","at":"dramatically"}]},"start":139.485,"dur":17.799,"words":[{"text":"Anthropic","s":139.785,"e":140.281},{"text":"saw","s":140.345,"e":140.585},{"text":"this","s":140.665,"e":140.865},{"text":"in","s":140.905,"e":141.025},{"text":"their","s":141.065,"e":141.225},{"text":"long-running","s":141.305,"e":141.785},{"text":"agent","s":141.865,"e":142.105},{"text":"experiments.","s":142.185,"e":143.385},{"text":"The","s":143.465,"e":143.545},{"text":"agent","s":143.705,"e":143.865},{"text":"kept","s":143.945,"e":144.105},{"text":"marking","s":144.265,"e":144.585},{"text":"features","s":144.665,"e":144.985},{"text":"complete","s":145.065,"e":145.625},{"text":"after","s":145.785,"e":145.945},{"text":"unit","s":146.025,"e":146.265},{"text":"tests","s":146.465,"e":146.665},{"text":"passed,","s":146.745,"e":147.465},{"text":"without","s":147.625,"e":147.785},{"text":"noticing","s":147.865,"e":148.265},{"text":"they","s":148.385,"e":148.505},{"text":"didn't","s":148.585,"e":148.825},{"text":"work","s":148.945,"e":149.065},{"text":"end","s":149.145,"e":149.305},{"text":"to","s":149.385,"e":149.465},{"text":"end.","s":149.545,"e":150.425},{"text":"Once","s":150.505,"e":150.585},{"text":"it","s":150.665,"e":150.705},{"text":"was","s":150.745,"e":150.905},{"text":"told","s":150.985,"e":151.145},{"text":"to","s":151.185,"e":151.225},{"text":"use","s":151.305,"e":151.385},{"text":"browser","s":151.545,"e":151.865},{"text":"automation","s":151.945,"e":152.585},{"text":"and","s":152.705,"e":152.825},{"text":"test","s":152.945,"e":153.065},{"text":"like","s":153.145,"e":153.305},{"text":"a","s":153.358,"e":153.412},{"text":"human","s":153.465,"e":153.705},{"text":"user,","s":153.785,"e":154.505},{"text":"results","s":154.745,"e":154.985},{"text":"improved","s":155.065,"e":155.545},{"text":"dramatically.","s":155.625,"e":156.665}]},{"template":"list","chapter":"Make it checkable","narration":"So make your system easy for agents to check. Keep tests fast; Kent Beck's suite runs in about three hundred milliseconds, so it can run constantly. Make behavior visible: Armin Ronacher logs sign-in emails to the console in debug mode, so the agent can finish a login flow on its own. And when an app is slow to test, Peter Steinberger has the agent build a small CLI that runs the same code path.","params":{"title":"Make your system agent-checkable","items":[{"text":"Fast tests","sub":"Kent Beck's suite: ~300 ms, run constantly","at":"fast"},{"text":"Visible behavior","sub":"e.g. log sign-in emails to stdout in debug mode","at":"visible"},{"text":"A CLI into the hard parts","sub":"same code path, no clicking","at":"CLI"}]},"start":157.284,"dur":26.681,"words":[{"text":"So","s":157.584,"e":157.744},{"text":"make","s":157.824,"e":157.984},{"text":"your","s":158.064,"e":158.144},{"text":"system","s":158.224,"e":158.544},{"text":"easy","s":158.704,"e":159.104},{"text":"for","s":159.184,"e":159.264},{"text":"agents","s":159.464,"e":159.664},{"text":"to","s":159.744,"e":159.824},{"text":"check.","s":159.984,"e":160.224},{"text":"Keep","s":160.704,"e":160.864},{"text":"tests","s":161.024,"e":161.424},{"text":"fast;","s":161.504,"e":162.464},{"text":"Kent","s":162.544,"e":162.704},{"text":"Beck's","s":162.784,"e":163.024},{"text":"suite","s":163.184,"e":163.344},{"text":"runs","s":163.424,"e":163.584},{"text":"in","s":163.664,"e":163.704},{"text":"about","s":163.744,"e":164.064},{"text":"three","s":164.184,"e":164.304},{"text":"hundred","s":164.384,"e":164.544},{"text":"milliseconds,","s":164.704,"e":165.584},{"text":"so","s":165.664,"e":165.744},{"text":"it","s":165.784,"e":165.824},{"text":"can","s":165.904,"e":165.984},{"text":"run","s":166.104,"e":166.224},{"text":"constantly.","s":166.424,"e":166.944},{"text":"Make","s":167.584,"e":167.824},{"text":"behavior","s":167.904,"e":168.464},{"text":"visible:","s":168.624,"e":169.504},{"text":"Armin","s":169.584,"e":169.824},{"text":"Ronacher","s":169.904,"e":170.384},{"text":"logs","s":170.544,"e":170.704},{"text":"sign-in","s":170.784,"e":171.104},{"text":"emails","s":171.184,"e":171.584},{"text":"to","s":171.624,"e":171.664},{"text":"the","s":171.744,"e":171.824},{"text":"console","s":172.064,"e":172.304},{"text":"in","s":172.384,"e":172.464},{"text":"debug","s":172.544,"e":172.944},{"text":"mode,","s":173.024,"e":173.504},{"text":"so","s":173.584,"e":173.664},{"text":"the","s":173.744,"e":173.824},{"text":"agent","s":173.984,"e":174.144},{"text":"can","s":174.224,"e":174.384},{"text":"finish","s":174.504,"e":174.624},{"text":"a","s":174.664,"e":174.704},{"text":"login","s":174.784,"e":175.104},{"text":"flow","s":175.184,"e":175.424},{"text":"on","s":175.464,"e":175.504},{"text":"its","s":175.584,"e":175.664},{"text":"own.","s":175.744,"e":176.464},{"text":"And","s":176.544,"e":176.624},{"text":"when","s":176.664,"e":176.704},{"text":"an","s":176.784,"e":176.864},{"text":"app","s":176.944,"e":177.104},{"text":"is","s":177.184,"e":177.264},{"text":"slow","s":177.344,"e":177.504},{"text":"to","s":177.584,"e":177.664},{"text":"test,","s":177.744,"e":178.464},{"text":"Peter","s":178.584,"e":178.704},{"text":"Steinberger","s":178.784,"e":179.424},{"text":"has","s":179.544,"e":179.664},{"text":"the","s":179.704,"e":179.744},{"text":"agent","s":179.824,"e":180.064},{"text":"build","s":180.144,"e":180.304},{"text":"a","s":180.384,"e":180.464},{"text":"small","s":180.624,"e":180.784},{"text":"CLI","s":180.944,"e":181.504},{"text":"that","s":181.584,"e":181.664},{"text":"runs","s":181.744,"e":181.904},{"text":"the","s":181.984,"e":182.144},{"text":"same","s":182.264,"e":182.384},{"text":"code","s":182.464,"e":182.704},{"text":"path.","s":182.784,"e":183.344}]},{"template":"statement","chapter":"The rule","narration":"Which brings us to the rule, straight from Anthropic's guide. If you can't verify it, don't ship it. Mitchell Hashimoto adds the payoff: give an agent a way to verify its work, and more often than not, it fixes its own mistakes.","params":{"text":"If you can't verify it, *don't ship it*.","sub":"Give it a way to verify, and it usually fixes its own mistakes. — Mitchell Hashimoto","sub_at":"payoff","art":"check","tone":"good"},"start":183.965,"dur":15.631,"words":[{"text":"Which","s":184.265,"e":184.505},{"text":"brings","s":184.545,"e":184.745},{"text":"us","s":184.825,"e":184.905},{"text":"to","s":184.945,"e":184.985},{"text":"the","s":185.065,"e":185.145},{"text":"rule,","s":185.225,"e":185.785},{"text":"straight","s":185.865,"e":186.185},{"text":"from","s":186.265,"e":186.345},{"text":"Anthropic's","s":186.465,"e":186.985},{"text":"guide.","s":187.065,"e":187.385},{"text":"If","s":187.865,"e":188.025},{"text":"you","s":188.105,"e":188.185},{"text":"can't","s":188.265,"e":188.425},{"text":"verify","s":188.505,"e":188.985},{"text":"it,","s":189.065,"e":189.465},{"text":"don't","s":189.545,"e":189.785},{"text":"ship","s":189.865,"e":190.025},{"text":"it.","s":190.105,"e":190.265},{"text":"Mitchell","s":190.825,"e":191.145},{"text":"Hashimoto","s":191.265,"e":191.865},{"text":"adds","s":191.985,"e":192.105},{"text":"the","s":192.185,"e":192.225},{"text":"payoff:","s":192.265,"e":193.305},{"text":"give","s":193.385,"e":193.545},{"text":"an","s":193.585,"e":193.625},{"text":"agent","s":193.705,"e":194.025},{"text":"a","s":194.065,"e":194.105},{"text":"way","s":194.185,"e":194.265},{"text":"to","s":194.345,"e":194.425},{"text":"verify","s":194.505,"e":194.905},{"text":"its","s":194.985,"e":195.065},{"text":"work,","s":195.145,"e":195.705},{"text":"and","s":195.825,"e":195.945},{"text":"more","s":196.065,"e":196.185},{"text":"often","s":196.305,"e":196.425},{"text":"than","s":196.505,"e":196.585},{"text":"not,","s":196.665,"e":197.145},{"text":"it","s":197.225,"e":197.305},{"text":"fixes","s":197.385,"e":197.705},{"text":"its","s":197.785,"e":197.865},{"text":"own","s":197.945,"e":198.025},{"text":"mistakes.","s":198.105,"e":198.985}]},{"template":"quiz","chapter":"Quick check","narration":"Quick check. Your agent reports: feature done, I added tests, all twelve pass. What's the best next step? Tap your answer.","params":{"q":"“Feature done. I added tests and all 12 pass.” Best next step?","options":["Merge it. Tests pass.","Ask it to show the real test output and demo the feature running","Ask for a longer written summary","Ask it to raise coverage to 100%"],"answer":1,"explain":"Summaries aren't evidence. Real output plus the feature actually running is."},"start":199.596,"dur":14.251,"words":[{"text":"Quick","s":199.896,"e":200.136},{"text":"check.","s":200.296,"e":200.536},{"text":"Your","s":200.936,"e":201.096},{"text":"agent","s":201.256,"e":201.416},{"text":"reports:","s":201.696,"e":202.376},{"text":"feature","s":202.456,"e":202.696},{"text":"done,","s":202.776,"e":203.336},{"text":"I","s":203.416,"e":203.496},{"text":"added","s":203.656,"e":203.816},{"text":"tests,","s":204.056,"e":204.536},{"text":"all","s":204.656,"e":204.776},{"text":"twelve","s":204.856,"e":205.096},{"text":"pass.","s":205.176,"e":205.976},{"text":"What's","s":206.056,"e":206.216},{"text":"the","s":206.256,"e":206.296},{"text":"best","s":206.376,"e":206.616},{"text":"next","s":206.736,"e":206.856},{"text":"step?","s":207.016,"e":207.656},{"text":"Tap","s":207.776,"e":207.896},{"text":"your","s":207.936,"e":207.976},{"text":"answer.","s":208.056,"e":208.696}]},{"template":"exercise","chapter":"Try it","narration":"Try this on your next task. It makes the agent plan its verification first, prove the test fails, and show you real output at the end instead of a summary.","params":{"title":"Try it with your agent","intro":"Add this to your next task:","prompt":"Before changing anything, tell me how you'll verify this end to end.\nUse red/green TDD: write the failing test and show it failing.\nThen implement until it passes, run the full suite, and run the\nreal app to confirm. Show me actual output, not a summary."},"start":213.847,"dur":10.589,"words":[{"text":"Try","s":214.147,"e":214.387},{"text":"this","s":214.467,"e":214.627},{"text":"on","s":214.707,"e":214.787},{"text":"your","s":214.827,"e":214.867},{"text":"next","s":214.947,"e":215.107},{"text":"task.","s":215.187,"e":215.827},{"text":"It","s":216.067,"e":216.147},{"text":"makes","s":216.227,"e":216.387},{"text":"the","s":216.467,"e":216.547},{"text":"agent","s":216.627,"e":216.867},{"text":"plan","s":216.947,"e":217.187},{"text":"its","s":217.267,"e":217.347},{"text":"verification","s":217.427,"e":218.067},{"text":"first,","s":218.147,"e":218.947},{"text":"prove","s":219.067,"e":219.187},{"text":"the","s":219.267,"e":219.307},{"text":"test","s":219.347,"e":219.587},{"text":"fails,","s":219.747,"e":220.467},{"text":"and","s":220.547,"e":220.627},{"text":"show","s":220.707,"e":220.867},{"text":"you","s":220.947,"e":221.027},{"text":"real","s":221.187,"e":221.427},{"text":"output","s":221.507,"e":221.907},{"text":"at","s":221.947,"e":221.987},{"text":"the","s":222.067,"e":222.147},{"text":"end","s":222.227,"e":222.467},{"text":"instead","s":222.547,"e":222.867},{"text":"of","s":222.947,"e":222.987},{"text":"a","s":223.027,"e":223.067},{"text":"summary.","s":223.107,"e":223.827}]},{"template":"recap","chapter":"Recap","narration":"Recap. Close the loop: give the agent a check it can run. Without one, you're the bottleneck. Test first, so it can't grade its own homework. Then run the real thing, because passing tests aren't proof. Next: when the agent is done, how do you review its code?","params":{"items":["Close the loop: a check the agent can *run*","No check? You're the bottleneck","Red/green TDD: test first","Passing tests aren't proof: run it"],"next":"Reviewing agent code","next_at":"Next"},"start":224.436,"dur":20.029,"words":[{"text":"Recap.","s":224.736,"e":225.376},{"text":"Close","s":225.856,"e":226.096},{"text":"the","s":226.176,"e":226.256},{"text":"loop:","s":226.336,"e":227.056},{"text":"give","s":227.136,"e":227.296},{"text":"the","s":227.336,"e":227.376},{"text":"agent","s":227.536,"e":227.776},{"text":"a","s":227.856,"e":227.936},{"text":"check","s":228.016,"e":228.256},{"text":"it","s":228.336,"e":228.416},{"text":"can","s":228.496,"e":228.576},{"text":"run.","s":228.656,"e":228.976},{"text":"Without","s":229.376,"e":229.696},{"text":"one,","s":229.816,"e":230.176},{"text":"you're","s":230.336,"e":230.496},{"text":"the","s":230.536,"e":230.576},{"text":"bottleneck.","s":230.656,"e":231.536},{"text":"Test","s":231.856,"e":232.096},{"text":"first,","s":232.176,"e":232.736},{"text":"so","s":232.816,"e":232.896},{"text":"it","s":232.976,"e":233.056},{"text":"can't","s":233.136,"e":233.376},{"text":"grade","s":233.456,"e":233.616},{"text":"its","s":233.696,"e":233.776},{"text":"own","s":233.856,"e":233.936},{"text":"homework.","s":234.016,"e":234.736},{"text":"Then","s":234.976,"e":235.216},{"text":"run","s":235.296,"e":235.376},{"text":"the","s":235.456,"e":235.536},{"text":"real","s":235.616,"e":235.776},{"text":"thing,","s":235.936,"e":236.336},{"text":"because","s":236.456,"e":236.576},{"text":"passing","s":236.656,"e":236.976},{"text":"tests","s":237.136,"e":237.536},{"text":"aren't","s":237.696,"e":237.936},{"text":"proof.","s":238.136,"e":238.416},{"text":"Next:","s":239.056,"e":239.696},{"text":"when","s":239.936,"e":240.016},{"text":"the","s":240.056,"e":240.096},{"text":"agent","s":240.176,"e":240.416},{"text":"is","s":240.496,"e":240.576},{"text":"done,","s":240.656,"e":241.296},{"text":"how","s":241.376,"e":241.456},{"text":"do","s":241.496,"e":241.536},{"text":"you","s":241.616,"e":241.696},{"text":"review","s":241.856,"e":242.016},{"text":"its","s":242.096,"e":242.176},{"text":"code?","s":242.256,"e":242.816}]}],"total":244.865,"audio":"/audio/08-verification.mp3"}