--- title: Ask what the green check proved description: When a check goes green, ask what it proved, then what it didn't. Addy Osmani's checklist for critical thinking when agents do the work, held up against my own setup, where agents merge and deploy about 80 pull requests a day. date: 2026-11-04 tags: AI coding agents, code review, verification, critical thinking --- When a check goes green, ask what it proved. Then ask what it didn't. That's the advice I keep from Addy Osmani's new article on critical thinking when agents do the work. It matters to me because my coding agents now merge and deploy their own pull requests. The article came out five days after I gave them that last step. In that first week, 557 pull requests merged on my GitHub account. So I held his checklist up against my setup. It held up where I had defined "verified" before the work, and failed where I trusted a green check for more than it checked. Addy works on Claude Code at Anthropic, and he likes agents. He says so first: this isn't a call to use fewer of them. [Read the article on X](https://x.com/addyosmani/status/2117512471930167377), posted on November 3. It's my third note on code review. The first was [review the plan, not the pull request](/notes/review-the-plan-not-the-pull-request.html). The second was [how the Claude Code team reviews code that Claude writes](/notes/claude-code-team-code-review.html), a post Addy co-wrote. ## The slow slide into "looks fine" Here's the trap. Agent answers look right most of the time now. They're well written, mostly correct, and they don't ask for a second look. You can run ten agents at once, but you still have only one pair of eyes. Addy draws three levels of trust in an agent: 1. Offloading. You hand over the task, but you still own the result. 2. Deferral. The answer becomes your next input. You no longer treat it as a guess to test. 3. Surrender. The answer quietly becomes yours, and you stop checking it. "Cognitive surrender" is a term from Steven Shaw and Gideon Nave, two researchers at Wharton. Nobody gets there in one step. You read one answer carefully. It was good, so you skim the next. Of the third, you only glance at the outline. Then someone points at one line and asks why it's there. You can't say, not even with your agent's help. That's his test: if you can't explain the answer, even with your agent, it isn't yours anymore. Sometimes that's fine. Addy let an AI pick between three similar hotels for a course recording, and he'd do it again. He wouldn't do it for money, for his children's careers, or to decide if code is safe to ship.

  1. 1Offloading

    You hand over the task, but you still own the result

  2. 2Deferral

    The answer becomes your next input, no longer a guess to test

  3. 3Surrender

    The answer quietly becomes yours, and you stop checking it

How you slide down

  1. 01Read it

    The last answer was good

  2. 02Skim it

    So you skim the next one

  3. 03Glance at it

    Only the outline of the one after

  4. 04Lose it

    Asked about a line, you can't explain it

The testIf you can't explain the answer, even with your agent's help, you've given it up.

Addy's three levels, then the slide down them, one reasonable step at a time. "Cognitive surrender" is Shaw and Nave's term.
## Eighty pull requests a day Until October, every change ended with me. My [How I work with AI](/ai.html) page still says so: "a pull request I read and merge". That sentence is now out of date. It changed in two steps. On October 8, eight pull requests had been waiting for days in the repo that holds my agents' setup. Two were stacked, one built on top of the other. So I told each agent: merge and clean up the pull requests you open. On October 29, I removed the last exceptions, my finance app, my health apps and this site among them. And I added one more step: deploy. The rule lives in `CLAUDE.md`, the file my coding agents read when a session starts. Once a change is verified and GitHub says it can merge, the agent merges it and deploys it. Then it checks the live app and cleans up after itself. It stays that way until I say otherwise. The same paragraph pins down "verified": tests, CI (the checks GitHub runs on every change) and a real browser for anything with a screen. That's Addy's advice: decide what "verified" means before you start, because afterwards you'll find a reason to accept any result. My definition was there before the first agent merged anything. So no agent decides afterwards what counts as done. And the volume is exactly his point. From October 29 to November 4, 557 pull requests merged across 65 repos on my GitHub account. About 80 a day. Nobody reads 80 pull requests a day, and the rule doesn't ask me to.

September, on my AI page

I read and merge

  1. MeAsks for a change
  2. AgentOpens the issue, builds on its own branch
  3. AgentProves it works
  4. MeReads the pull request and merges it

then

Since October 29

The agent merges

  1. MeAsks for a change
  2. AgentOpens the issue, builds on its own branch
  3. AgentProves it: tests, CI, a real browser
  4. AgentMerges, deploys, checks the live app

Verified meanstestsCIa real browser for any screenGitHub says it can merge

557pull requests merged, October 29 to November 4

65repos on my GitHub account

80pull requests a day, on average

The last step moved from me to the agent. The counts come from a GitHub search of merged pull requests on my account.
## A green check proves one thing Addy's warning: you can give up on evidence without noticing. "Is there evidence that supports my conclusion?" quietly turns into "did the check turn green?" Agents make tests cheap, so there's green everywhere. But a test written with the same mistake as the code still passes. And a check covers one thing, even when it feels like it covers everything. His finish line is proof you can point at: a passing test, a screenshot, a log, a trace. When an agent says "done", he first reads the command it ran and what came back. The agent's own summary comes last. Three times in my own setup, something passed and proved less than it seemed: - The stylesheet. In September, a note on this site went live with its diagrams unstyled. The agent had checked the CSS with a cache-buster, an extra bit in the URL that skips the cache. So the agent got the new file, but readers' browsers still had the old one. The check proved the server had the new CSS. It didn't prove anyone would see it. - The blank page. A script wrote a page into my notes app, a OneNote replacement I host myself. Then it read the page back through the API, and the two matched. But in the app, the page was blank: one empty text node in a table cell made the editor refuse the whole page. The check proved the server stored the page, not that the page opens. - The daily backup. The same app saves a copy of its database every day and keeps the last 14. The job runs without an error. But the clean-up also counts 14 older one-off backups whose file names start the same way. So it deletes each new copy right after writing it. Every step does its job, and there is no daily backup. The bug is still open as I write this.

The stylesheetpassed

ProvedThe server had the new CSS, fetched past the cache

MissedReaders' browsers kept the old file. The diagrams went live unstyled.

The blank pagepassed

ProvedThe API stored the page a script wrote

MissedOne empty text node, and the editor refused the page. It opened blank.

The daily backupno error

ProvedA copy is written every day

MissedThe clean-up deletes it right after. Still open.

AddyWhen a check goes green, ask what it proved. Then ask what it didn't.

Two checks that passed and one job that ran without an error, from this site and my notes app. Each was true about one thing only.
## Every mistake becomes a rule This is where my setup does what Addy asks. When something isn't good enough, he says, write down why "in shared config, a hook or a test so that it doesn't happen again. That's how taste compounds." A hook is a tiny script that runs before an agent's command and can say no. Most of my agent rules were born from a mistake: - Docker, stopped twice. On macOS, `pkill` reads every option written after the first pattern as one more pattern. An agent wrote its options last. They matched Docker Desktop and stopped every container, on October 13 and again on October 22. Since October 25, a hook refuses that word order and prints the fixed command. I replayed 830 old commands through it: 12 refused, and all 12 were real mistakes. - Port clashes. Agents kept taking ports that another app already used. Since October 15, a hook checks docker and Tailscale commands against my list of ports before they run. It has 43 tests. - Unstyled diagrams. That one became a rule: open the page in a real browser, at phone and desktop widths, on my machine and again on the live site. This site has a checker that does exactly that, in Chromium and WebKit, the engines behind Chrome and Safari. - Tests on live data. On October 25, an agent ran my films app's tests inside the live container. The container's database path won over the one the tests set, so the tests deleted the live tables. The film list was rebuilt from the copy on the drive that holds the films. The resume points came back from that morning's backup, minus everything watched after it. This rule is a line in my agents' memory and a fix in that app's tests. It isn't a hook yet, so it works only as long as an agent reads it.

  1. The mistakepkill read the options after the pattern as patterns and stopped every container, twice

    The ruleOptions before the pattern

    Hook12 of 830 old commands refused, all real

  2. The mistakeAgents took ports another app already used

    The ruleThe port list decides

    Hook43 tests

  3. The mistakeA note's diagrams went live unstyled

    The ruleA real browser, phone and desktop, local and live

    Check scriptChromium and WebKit

  4. The mistakeTests in a live container wiped my films app's tables

    The ruleTests never run in a live container

    Memorynot a hook yet

AddyWrite down why in shared config, a hook or a test, so it doesn't happen again.

Each rule is a mistake I already paid for. The dashed one still depends on an agent reading it.
## I scored my setup on his questions Addy keeps a short list of questions he asks himself. I tried five of them on my setup, and most found a gap. "Did I form a hypothesis before asking?" Every change starts as a GitHub issue, and a hook refuses a pull request without one. But the issue says what I want, not what I expect. "What would have to be true for this to be wrong? Then go find out." My real-browser check asks exactly that, with a script, on every visual change. This one holds. "Have I heard the counter argument from a different angle?" Only partly. In the pipeline that runs while I'm away, a model from another company reviews the change before the pull request opens. Everywhere else, the agent that built the change also grades it. "What did it decide for me?" Addy asks the agent to list its choices: the defaults, what it left out, which requirement won when two clashed. My pull requests say what changed and how it was tested. Nothing in them asks for the choices. "What am I no longer practicing?" He writes that list every month. Nothing in my setup tracks it. So one holds, one holds partly, and three are gaps. My setup checks the result well, but barely checks my own thinking, before the work or after.

  1. Did I form a hypothesis before asking?

    The issue says what I want, not what I expect

    Gap

  2. What would have to be true for this to be wrong?

    A real browser at 320 px and on desktop, local and live

    Holds

  3. Have I heard it from another angle?

    Another company's model, only in the pipeline that runs while I'm away

    Partly

  4. What did it decide for me?

    Pull requests say what changed, not what it chose

    Gap

  5. What am I no longer practicing?

    Nothing in my setup tracks it

    Gap

1holds

1partly

3gaps

His questions, shortened, against what my setup does today. The checks on the result hold. The ones on my thinking mostly don't.
## Where I'm still just hoping Addy has a name for the gap between what your agents produce and what you'd sign: the orchestration tax. When finished work piles up faster than you can check it, one more agent session won't help. Either make the checks stronger, so less of the work needs you, or slow down. Then his sharpest line: "When you're producing more than you can stand behind, you're no longer reviewing. You're hoping." On October 8, I didn't slow down. I stepped out of the merge and kept the checks. Two of the hooks above arrived in the next three weeks. That's his first option, partly done. Three places in my setup still run on hope: - Notes that publish themselves. Every Monday, a script drafts up to three short notes for this site from what I shipped. Since September 26, it also publishes them, with no human step. The safety net I designed for it is [still an open issue](https://github.com/brbousnguar/brbousnguar.github.io/issues/135): a second model to check the facts, a privacy and style check, and a one-tap unpublish. - A phone fix. On October 31, a fix for my notes app merged and deployed. Its own pull request said: "Not provable here: the AutoFill bar and the real keyboard geometry." The agent said so plainly. The fix still shipped, before anyone tried it on an iPhone. - The backup bug from above. An agent found it and filed it. Nobody has fixed it yet. Addy also asks what happens when the models are better at this than we are. His answer: someone still answers for what ships. He quotes Anthropic's research on its own engineers, which calls it the paradox of supervision: "supervising Claude requires the very coding skills that may atrophy from AI overuse." Then: "You may offload the work and much of the auditing, but not the accountability." My rule moved the merge button. It didn't move the accountability.

  1. Notes that publish themselves

    Every Monday since September 26. Fact check, style check and unpublish are designed, not built.

    Safety net not built

  2. A phone fix in my notes app

    "Not provable here: the AutoFill bar and the real keyboard geometry."

    Not tried on a phone

  3. The daily backup bug

    Found and filed by an agent, October 31

    Bug open

AddyWhen you're producing more than you can stand behind, you're no longer reviewing. You're hoping.

What my setup ships with nobody standing behind it yet, as of November 4.
## Try it Addy ends with an exercise that costs two lines. Before you ask, write down what you think will happen. When the agent is done, write down one thing it didn't check. His other habits, in short: - Have a guess, even a bad one. - Read what the agent ran and what it returned before you read its summary. - Ask it what it decided for you. - Write the request in your own words: who it's for, what's out of scope, what done means. - Once a week, do one real task without an agent. - If you wouldn't put your name on it, it doesn't go out. In my setup, the two cheapest gaps to close are his first and fourth questions: one line in each issue for what I expect, and a list of the agent's choices in each pull request.

Before you ask

Write what you expect

Even a bad guess. Once you've read a well-written answer, it's hard to imagine another.

then

After it's done

Write what it didn't check

One thing. A green check means one thing, so name another.

Once a weekone real task without an agent

Once a montha list of what you've stopped doing

His questionHow lost would you be if AI didn't exist?

The two lines from the end of the article, then the habits he suggests around them.
If you want more: - [Critical thinking when agents do the work](https://x.com/addyosmani/status/2117512471930167377), Addy's article, about 15 minutes - [Cognitive surrender](https://addyosmani.com/blog/cognitive-surrender/), his earlier post on the term - [The orchestration tax](https://addyosmani.com/blog/orchestration-tax/), on the gap between what agents produce and what you'd sign - [How AI is transforming work at Anthropic](https://www.anthropic.com/research/how-ai-is-transforming-work-at-anthropic), the research behind the paradox of supervision - [How the Claude Code team reviews code that Claude writes](/notes/claude-code-team-code-review.html), my notes on the post Addy co-wrote Try the two lines on your next agent task. Then tell me what your agent didn't check.