--- title: Review the plan, not the pull request description: Overmind stopped reading its own pull requests and moved peer review to a page-and-a-half plan written before an agent writes any code. Notes on Dylan Ratcliffe's talk, the until loop behind it, and lead time down from 3.2 to 1.3 days. date: 2026-10-15 tags: AI coding agents, code review, planning, MCP --- Overmind stopped reading its own pull requests. Almost all the code it ships to production is now read by no human, the author included. Review didn't go away, though. It moved to a plan of about a page and a half that people argue over before an agent writes a line, and in the three weeks after the switch the team closed three times as many issues while lead time fell from 3.2 days to 1.3. The talk is [Pull Requests Are Dead, Long Live Peer Review](https://www.youtube.com/watch?v=5lq5zp6wYbQ) by Dylan Ratcliffe, founder and CEO of [Overmind](https://overmind.tech/), from AGNTCon + MCPCon North America 2026. Overmind uses AI to simulate changes to large back-end infrastructure. In my [software factory note](/notes/software-factory-2022-to-lights-off.html), the slow part of my own setup is my review queue. This talk is one team's answer to that. ## Review was doing four jobs Like most teams, Overmind leaned on code review for much more than bugs. It caught bugs, it covered SOC 2 and ISO 27001 compliance, it trained junior engineers, and it built a shared picture of how the system actually works. What made it work is that a human cared about every line. Comment on a person's pull request and something good happens either way. If you're right, they learn and the product gets better. If they push back, there's a constraint you didn't know about, and now you do. Comment on a pull request from a bot that addresses its own review comments, and you get the same eager agreement whether you're absolutely right or absolutely wrong. That's his first lesson: AI is not your peer, so you can't peer review its work.
  1. 1Catching bugs
  2. 2SOC 2 and ISO 27001 compliance
  3. 3Training junior engineers
  4. 4A shared picture of how the system works

The author is a person

Your review comment

If you were rightThey learn, and the product gets better

If they were rightYou learn a constraint you didn't know

vs

The author is a bot

Your review comment

If you were rightAgrees and addresses it

If you were wrongAgrees and addresses it

Lesson 1AI is not your peer, so you can't peer review its work.

The four jobs, then one comment sent to two authors. Only the person gives you an answer that depends on whether you were right.
## Two people out-shipped the company At the start of the year Overmind was about 80% engineers. Product was Dylan and one product manager. They came up with the idea, did the research and worked out the shape, meaning the high-level architecture. Then came a handover meeting, a round of questions, and engineers planned and built it with AI. The catch: once Dylan had done everything up to the shape, handing it straight to an agent was faster than handing it to a person. He lives in San Francisco, a bunch of his engineers are in Europe, and the agent never sleeps. So one week he and the product manager produced more than every other person in the company combined, on every metric they had: lines of code, pull requests, tickets closed and story points. That's only six people, but it's two people doing the work of six full-time engineers. "Neither of us are engineers," he said. He's open that he seriously considered firing half the engineers and replacing them with product managers. He didn't. He flew everyone to France for a week to sort out the process instead.

Dylan and the product managerhanding work straight to agents

The other sixeveryone else in the company

Ahead on every metriclines of codepull requeststickets closedstory points

ConsideredFire half the engineers, hire product managers

DidFlew everyone to France for a week

Dylan"Neither of us are engineers."

One week's output as Dylan told it. He gave no amounts, only "more than everyone else combined", so the tilt shows which side won, not by how much.
## The work moved left This is the slide I grabbed from the talk, redrawn. Before, product owned the idea, the research and the shape, and engineering owned the plan, the build and the ship. Now product keeps the idea, engineering does the research, the shape and the plan, and an agent builds and ships.

one person on the workthe work moved left

Lesson 2If your job is turning detailed specs into high-quality code, you're out of a job. True since January and Opus 4.5, he says.

Redrawn from the slide. Its caption: "Once the plan existed, handing it to an agent was faster than handing it to a person."
His second lesson came out of that week: if you think your job is turning detailed specs into high-quality code, you're out of a job. He says that's been true since January and Opus 4.5. ## One page, 24 hours, no questions Moving engineers left came with a rule. Product hands over a one-page doc. The engineer who gets it shows a demo within 24 hours, and can't ask a single question in that time. A one-page doc is full of gaps, and that's the point. Does this need to work on mobile? Go and research whether people use this kind of app on mobile. Forced to settle the ambiguity themselves, the engineers were mostly right. Nobody can absorb all that new ambiguity and still care about every line of code, though. So the line-by-line detail went to agents.

  1. 1One page

    From product, full of gaps on purpose

  2. 224 hours

    Zero questions allowed

  3. 3A demo

    Every gap filled by the engineer's own call

ResultLeft to settle the ambiguity themselves, the engineers were mostly right.

The rule from the France week. The dashed lines in the doc are the gaps the engineer has to fill alone.
## Not every agent loop is equal When they studied how people build agentic "software factories" and "dark factories", most ran what he calls a do-while loop. The agent picks up a ticket, decides the approach, makes the trade-offs, writes code and tests, maybe records a demo video, and opens a pull request at the very end. Then a human has to reverse-engineer every decision from the code. He's a little worried about AI taking his job. He's more worried about AI taking the interesting part and leaving him paid to sign off on decisions he wasn't in the room for. So they built an until loop. Agents research the codebase, for example to check whether there's already a practice for this kind of change. Humans decide the approach, make the trade-offs and write an implementation plan, and a peer reviews it. Then an agent writes the code and a second agent compares it with the plan. Any difference sends it back, and it loops until code and plan match.

Most agent factories

do-while

  1. AgentPicks up a ticket
  2. AgentDecides the approach and the trade-offs
  3. AgentWrites code and tests, maybe a demo video
  4. AgentOpens a pull request
  5. HumanReviews: reverse-engineers every decision from the code

vs

What Overmind runs

until

  1. AgentResearches the codebase
  2. HumanDecides the approach and the trade-offs
  3. HumanWrites the plan
  4. HumanReviews: a peer reads the plan
  5. Repeats until the code matches the plan
    1. AgentWrites the code
    2. AgentCompares the code with the plan
  6. ProdShips, with no human reading the code
Both loops, top to bottom. The marker row is where a human reviews: at the end of one, before the code exists in the other.
Once the loop looks like that, the place for peer review is obvious: before the loop starts. ## Review the plan, then stop reading Plans get real review because people care about them again. When someone comments on Dylan's pull request today, his honest reaction is "I didn't even read that code. If anything, you're the first one to have read it." When someone comments on his plan, he argues back, because he wrote every line and knows why each one is there. He treats an approved plan as a pre-approved change. Most compliance setups already know that idea: a well-defined change can be approved in advance, as long as you do exactly that. The loop can't exit until the code matches an approved plan, so the code goes to production without a second human review. Two checks keep it honest: - Humans sample a small percentage of the plan-versus-code checks, to make sure the review agent isn't getting them wrong. - If the coding agent finds a better idea, or a real reason it can't follow the plan, it records a difference, and a human acknowledges it. Overmind was being audited when he gave the talk. He strongly believes the process is compliant, and he'll know when the audit ends. The tooling has no user interface at all. Plans get written in Claude Code or whatever harness someone uses, and sending a plan, pulling it down and leaving a review are all MCP calls.

Humans

Approved plan

  • About a page and a half
  • The trade-offs and why
  • A pre-approved change

Agents

The until loop

  • One writes the code
  • One compares it with the plan
  • Can't exit until they match

No second review

Production

  • Code no human has read
  • Matches an approved plan

SamplingHumans spot-check a small percentage of the plan-versus-code checks

A differenceThe agent records why it left the plan, and a human acknowledges it

No UIPlans are written in Claude Code or any harness. Sending, pulling and reviewing them are MCP calls.

The pre-approved change, end to end, with the two checks underneath. The audit was still open at the time of the talk.
## Three weeks later They changed the process in one week, then measured the next three weeks against the three before. Issues completed tripled, and so did commits. Lead time, from a ticket being picked up to running in production, fell from 3.2 days to 1.3. These are one small team's numbers, three weeks against three weeks, measured by the company itself. And Overmind now sells a product built on this exact idea, so take the talk as a strong argument from someone with a stake in it.

3×issues completed

3×commits

1.3 dayslead time, down from 3.2

Lead time, ticket picked up to running in production

Three weeks before3.2 days

Three weeks after1.3 days

Overmind's own numbers for its own team, as Dylan gave them on stage. The two bars are to scale.
## Attention was the bottleneck They were never spending 90% of their day on code review, so the 3x didn't come from skipping review time. His answer is attention. Pull requests kept too many balls in the air and forced constant context switching. The old timeline for one change: about 3 hours writing code, 15 minutes opening the pull request, a day and a half waiting for a busy reviewer, then an hour of back and forth. You can't forget any of it, because you're the only one who understands the code. By the time a comment came back, he'd written 15,000 more lines and couldn't remember what he was trying to do, so abandoning the change could be easier than switching back into it. About four and a half hours of work lived in his head for about 40 hours. A plan is faster to write and much faster to review than 3,000 lines of Go. Once it's approved, you commit it to memory as how the system works. The code doesn't exist yet, but it will in about three hours, because there's no other way out of the loop. That costs a couple of hours of attention.
The durations from his example, to scale. He didn't split the "couple of hours" for the plan, so it's drawn as one block.
## A recipe is easier to reject than a meal Standing in the kitchen deciding what to cook, it's easy to say "I don't feel like salad." Once the salad is made, you can't. Telling someone to cook a meal again takes Gordon Ramsay energy, and that's what rejecting a pull request is. It's emotionally hard, which is why it was usually only the most senior engineers who did it. Plans changed that. Overmind is now 2.8 times more likely to reject a plan than it ever was to reject a pull request, and the rejections come from everyone on the team, product included. It's also far cheaper: a rejected plan adds about 40 minutes of lead time, while any comment at all on a pull request added 16 hours. His big worry going in was agents writing the plans and agents reviewing them, with nobody reading anything. He says the numbers show it didn't happen. Plans are short enough that people read them, and they care about them. It also gives juniors a way in. A junior who doesn't follow a plan can talk it through with their agent until they do, and reject whatever still doesn't make sense. Dylan's product manager reviews engineering plans exactly like that, and when he pushes back, he's right about 80% of the time. Dylan counts the 8% of plans they abandon as a win. The what looked good, the how showed it wasn't worth it, and nobody had to build it just because it was already in the sprint.

Before it's cooked

A recipe

"I don't feel like salad" is easy to say

The plan: everyone rejects them now

vs

Once it's cooked

A meal

Sending it back takes Gordon Ramsay energy

The pull request: senior engineers did it

Lead time added by pushback

A plan gets rejected+40 min

A pull request gets any comment+16 h

2.8×more likely to reject a plan than a pull request

80%of the product manager's pushback was right

8%of plans dropped before any code exists

The salad, then the numbers from the talk. The two pushback bars are to scale: 40 minutes against 16 hours.
## The how still matters He closed by disagreeing with a Paul Graham line: "When anyone can make anything, the big differentiator is what you choose to make." Dylan says the differentiator is still how you make it. Two laptops can share the exact same spec sheet and be completely different products. The difference is hundreds of how decisions along the way, and those decide whether a product performs, scales and stays easy to debug. His advice: don't sit at the bottom of a loop where the agent makes those calls for you, and say no to tools that put you there. What changes is the layer you answer them at. You stop caring how the files are organized or what the functions are called. You still decide how data flows, how authentication works and how the system handles failure.

One productSame spec sheetAnother product

You still decide, in the planhow data flowshow authentication workshow it handles failure

Agents decide, in the codehow the files are organizedhow the functions are named

DylanDon't sit at the bottom of the loop, and say no to tools that put you there.

Same spec, two products: the difference is the how. The how questions don't go away, they move up into the plan.
## Try it Overmind published the internal tool as [until](https://until.dev/), "git-backed tickets for people and agents". You write the plan, someone signs it off, each section of the plan becomes a job your agents claim, and a review agent compares the code with the plan, so anything missing or extra comes back to you as a decision. It's waitlist-only right now, and the site lists Claude Code, Cursor, Codex, Pi, Factory Droid, Amp and OpenCode. The argument also exists as a manifesto, [AI Is Not Your Peer](https://notyourpeer.com/), by Dylan Ratcliffe, James Lane and David Schmitt. Its key line: "The artifact worth reviewing is the plan we wrote before the agent wrote anything."

  1. 01You write the plan

    Every feature starts with a plan, and the plan is yours

  2. 02Someone signs it off

    No agent starts until a person says yes

  3. 03Agents take the jobs

    Each section of the plan becomes a named job

  4. 04Check the difference

    Anything missing or extra comes back to you as a decision

Listed harnessesClaude CodeCursorCodexPiFactory DroidAmpOpenCode

WaitlistGit-backed tickets for people and agents.

The four steps as until.dev describes them. The first two are people, the last two are agents.
If you want more: - [Pull Requests Are Dead, Long Live Peer Review](https://www.youtube.com/watch?v=5lq5zp6wYbQ), the talk, 25 minutes - [AI Is Not Your Peer](https://notyourpeer.com/), the manifesto - [until](https://until.dev/), the plan-review tool, on a waitlist - [The agentic software factory is the 2022 one with one box swapped](/notes/software-factory-2022-to-lights-off.html), where review became my bottleneck - [What breaks when an AI agent runs a real business](/notes/what-breaks-when-an-agent-runs-a-business.html), another AGNTCon + MCPCon talk Watch [the talk](https://www.youtube.com/watch?v=5lq5zp6wYbQ), then try reviewing your next agent task as a plan before any code exists. Tell me if your lead time moves.