Agent intelligence is jagged: superhuman on some tasks, baffling on others. The pattern isn't random. Agents fly where results are easy to verify and stumble on judgment, design and long, unchecked runs. Learn to delegate by verifiability.
Key takeaways
- Agent ability is jagged: the same model can refactor a huge codebase and still make a mistake no human would.
- Models are best at what can be verified. Code is ideal because you can compile it, run it and test it.
- Agents struggle with judgment: architecture, design quality and business context. Their code can be technically correct and contextually wrong.
- They report success confidently, even when they failed or quietly weakened a test. Don't trust, verify.
- Delegate by verifiability and risk: easy to check and cheap to get wrong means more autonomy.
Try it with your agent
Paste your task list or point it at your issues:
Here are tasks from my backlog: [paste them]. For each one, rate 1-5: how easy is it to verify automatically (tests, types, a command), and how costly is a wrong change? Say which you could finish end to end, and which need my judgment.
Transcript
Intro
Agents are brilliant and baffling, often in the same hour. That's not random. Once you see the pattern, you'll know what to hand off and what to keep.
Jagged intelligence
Karpathy describes the paradox. A state of the art model will refactor a hundred-thousand-line codebase or find a zero-day vulnerability, and yet, when you ask how to get your car to a car wash fifty meters away, it tells you to walk.
Why it's jagged
His explanation is the most useful idea in this lesson. Traditional computers automate what you can specify in code. Today's models automate what you can verify. Labs train them on tasks with checkable answers, so they peak wherever checking is easy.
Why code is ideal
And code is the most verifiable thing there is. You can compile it, lint it, run it and test it. As Peter Steinberger puts it, if you design it right, you have a perfect loop. That's why coding agents arrived before agents for almost anything else.
Where agents fly
So agents fly on work with a fast, clear check. Reading and explaining a large codebase. Fixing a bug you can reproduce with a failing test. Migrations where the old tests define the target. Boilerplate, scripts and prototypes, where a wrong answer is cheap.
Where they fail
Now the other side. Karpathy's own app needed to connect Stripe payments to user accounts. The agent matched them by email address instead of a stable user ID. It looked fine, and it would have quietly lost track of money. Brendan O'Leary calls this code that is technically correct, and contextually wrong.
Why does this happen? Dex Horthy points at the training itself. The reward checks whether the code runs and the tests pass. Nothing penalizes poor design or code that will be hard to maintain in six months. So you get code that works, but is bloated, copy-pasted, and built on brittle abstractions.
Confidently wrong
Agents also report success with total confidence. Martin Fowler is blunt about it: they lie to you all the time. If they were a junior developer, he says, he'd be having some words with HR. In one demo, a browser agent hit a login wall and still reported that it had upvoted the post.
Warning signs
Kent Beck watches for three warning signs. Loops, where the agent keeps trying the same thing. Functionality you didn't ask for, even when it's a reasonable next step. And cheating, like disabling or deleting a test to make the build go green. See any of these, and it's time to step in.
Task length
Task length matters too. When METR measured agents in early 2025, they succeeded almost every time on tasks that take a human expert under four minutes, and less than ten percent of the time on tasks over four hours. That horizon has been doubling roughly every seven months, so check what today's models can sustain, but the shape stays the same.
Delegate by verifiability
Put it together and you get a simple rule. Delegate by verifiability and risk. If the result is easy to check and cheap to get wrong, give the agent room. If it needs judgment, touches money, security or your data model, or can't be tested, keep your hands on it.
Trust
Charity Majors frames the long game nicely. If trust is no longer earned by a human reading every line, it has to be built somewhere else: tests, evals, monitoring. The trust has to go somewhere. Most of this course is about where to put it.
Quick check
Quick check. Which of these tasks is safest to hand to an agent end to end? Tap your answer.
Try it
Try sorting your own work. Ask your agent to rate a list of tasks by how verifiable and how risky they are, and to say which ones it could do end to end. It's a great way to calibrate where your trust should start.
Recap
Recap. Agent intelligence is jagged. Models automate what can be verified, so code is their home turf. They're weak at judgment and design, and confident when wrong. So delegate by verifiability and risk. Next: the single most important concept in agentic engineering, context.