On Plans as HypothesesPlans are not evidence · Just a very persuasive router

There was a plan that laid out a fix neatly: the problem was clearly identified, the changes were spelled out, the risks were flagged where they belonged, and it ended with words like “fixed” and “settled” that made me want to nod along. I almost approved it after reading it.

Fortunately, I did not. Later, someone actually read through the code that was supposedly “already covered” and found that it did nothing of the sort described in the plan. By design, it only checked things; it would not initiate the crucial action. The route the plan proposed, “insert a record and wait for a background process to take over,” had no one waiting to take over. The resource that should have been restored by a rollback would never come back.

The plan had not lied. It had simply written “I think this approach should fix it” as “fixed.” What went missing in between was the friction of reality. Since then, I have set myself a rule: a plan can only ever be a hypothesis, never evidence. It can tell me where to look for evidence next, but it cannot count as evidence itself.

Plans are good at storytelling, not proof

Models really are good at producing plans, because organizing scattered information into a coherent narrative is what they do well. “I intend to change these parts” becomes a clear route. “I think the risk is here” becomes a measured assessment. “I have already checked” becomes a reassuring state of completion. It reads like an engineering retrospective on work already done.

But narratives do not determine engineering facts. Whether a field is validated does not depend on whether the plan says “add validation”; whether a background job might delete data by mistake does not depend on whether the plan says “added safeguards”; whether a login session can get mixed up does not depend on whether the build logs are green.

Engineering facts have their own locations. They are in a particular line of code, a particular database record, the page a browser actually renders, the traces left by a real request after it passes through the entire system. A plan cannot reach these places. It can only point to them.

So when I receive a plan now, my first response is not “Does this plan make sense?” Instead, I ask: Which places has it routed me to that I can inspect for myself?

“Fixed” is a claim, not a state

There are three words in collaborative work that make it especially easy to let go: “fixed,” “resolved,” and “settled.”

Between people, those three words carry a social commitment. When an engineer tells you “fixed,” you assume they have at least compiled it, looked at it, and run it once. When a model says “fixed,” it is expressing a conclusion inferred from the current context, not the state of the world. The two sound alike, but their reliability differs by an order of magnitude.

This is not deception. The model has simply compressed “given these changes, this should work in theory” into “fixed.” In that compression, the friction of testing the idea against reality has been discarded.

I eventually decided to translate all such words back into their original form:

Fixed    = a claim of a fix; look for the corresponding evidence in the code
Resolved = a claim of a resolution; look for the corresponding runtime evidence
Settled  = a claim that things have settled; look for the corresponding evidence about boundary conditions

It is a small act of translation, but surprisingly effective. It changes none of the technical content. It simply removes that little sense of emotional safety, leaving just one thing: where to go next to verify it.

Three layers of evidence: code, local checks, and the living system

When I verify a plan now, I divide the evidence into three layers. Each costs more than the last, and each is harder evidence.

The first layer is facts in the code. Open that specific file, that specific function, that specific conditional branch. Do not let the plan tell you “this is already covered.” Look at the line yourself and see whether it is. The greatest danger at this layer is skimming: a model is good at describing a small, local change as a fix that runs through the entire chain, and someone skimming will believe it. That was what went wrong with the plan I almost approved. No one had gone down to this layer. Only when someone actually read through the code did the hidden assumption come to light.

The second layer is local facts. Run the narrowest check in an environment you can control: a type check, a unit test, a script, a minimal reproduction. It proves only one thing: this path does not immediately break under controlled conditions. It does not prove that production is fine.

The third layer is the living system. For a frontend, that means a real browser; for transactions, a real database and state machine; for deployment, the full path of a real request through a test or production environment. Many problems only show themselves at this layer: caches, sessions, gateways, environment variables inside containers, the time windows of scheduled jobs, the shape of old data left behind months ago. The first two layers can never cast light on these things.

You do not have to go through all three layers every time. For a narrow change, the first or second may be enough. But once a change touches user assets, authentication state, money, inventory, or background jobs that write data on their own, the third layer is no longer an optional extra. It is the only source of fact.

Independent review matters for its detachment, not its intelligence

At first, I thought the value of “having one model review another” was that the other model might be smarter. Later, I found that was not it at all. The real value was its detachment.

The one doing the implementation has written the plan, read the code, changed the files, and run the tests. Little by little, it begins to take its chosen path for granted. It knows why it made each change, and it remembers where it has already put in effort. This context settles into a gentle kind of inertia: it is more likely to believe its fix has covered the problem because it knows all too well how hard it has tried.

An external reviewer that does not share this context has no such emotional debt. It sees only two things: claims and evidence. It will not lower its skepticism because “this plan has already been discussed for a long time,” or assume that only minor issues remain because “we have already done a round of fixes.” I have seen it overturn a repeatedly confirmed judgment as soon as it arrived. It had no idea how many times that judgment had been confirmed. It simply went to the code running in production, counted for itself, and found that the numbers did not match.

This detachment is not hostility. It strips away the narrative's hold. It pulls the judgment firmly back from “I understand how hard this has been for you” to “Do the facts actually hold up?” Reviewing your own work can never do this. It is a matter of position, not ability: standing inside your own context, you cannot pretend you have never read it. For that same reason, the external review must be physically separate from the implementation. When the reviewer makes its independent judgment, it cannot already have the implementer's conclusions in hand. The moment it reads them, its independence is gone.

GO, minor fix, and blocker are engineering terms

Another habit that developed gradually was reducing review verdicts to three terms: GO, minor fix, and blocker.

The point is not to sound decisive. It is to keep the discussion from branching out forever.

GO does not mean perfect. It means the current risk is within an acceptable range and the work can move to the next stage. Minor fix means there is a real problem, but it does not alter the main path, and fixing it will not require a redesign. Blocker means that moving forward would carry an incorrect state into a more expensive stage. The later it is discovered, the more painful the rollback.

This three-way distinction is especially useful when working with models, because a model can easily give you a long list of “suggestions,” one after another, all quite reasonable. With too many suggestions, people lose their priorities and start going in circles over details. Set the verdict to one of these three terms first, and every detail that follows must answer the same question: does this actually prevent the next step? If it does, it is a blocker, and nothing else matters. If it does not, it is a minor fix. Do not let it hold up the main work.

A plan is a router, not a judge

If I could keep only one sentence from this essay, it would be this: a plan is a router, not a judge.

A router's job is to send traffic to the right place. A plan should do the same: tell me which file to visit, which condition to examine, which check to run, which data to inspect, which page to open. How well it does its job depends on how accurately it points to those places, not how neatly its story holds together.

A judge's job is to reach a verdict. That authority cannot be handed to a plan. Conclusions can only grow out of that three-layer chain of evidence. No other source counts.

This may look like a change to a collaboration process, but what it really changes is how I read. I used to read a plan and weigh up “Is it clever?” Now I weigh up “Has it brought me face to face with the facts?” A plan that cannot point to facts, however complete its account, is only a piece of literature. A plan that points precisely to facts, even if it is only a few lines long, is enough to enter engineering practice.

Bring the facts closer instead of making the model stronger

Many people attribute problems in collaboration to the model's capabilities: switch to a stronger one, open a larger context window, write more elaborate instructions. I cannot say this is useless, but I increasingly feel that something else is what really changes the outcome: bringing the facts closer to the collaboration.

Describe the repository structure clearly. Spell out the verification commands. Turn experience into entries that can be retrieved. Write handoffs as an index of evidence that the next person can follow. Translate every “fixed” into “please go and look at each of these three places.” None of this adds a single parameter to the model, but it makes it harder for the model to quietly drift away inside its own narrative. At every step of that drift, there is a specific place nearby that can be opened and checked against its account.

Looking back at the plan I almost approved, the problem was never that it was not clever enough. It was clever enough to turn an unverified hypothesis into a retrospective that made me want to sign off. What actually prevented an incident was someone willing to close the plan, walk over to the line of code it pointed to, and look at it firsthand, rather than a model that spoke more persuasively. Background processes get no special exemption, and neither do plans. They simply have fewer witnesses.

Translation log: translated from the Chinese original by Codex on 2026-11-03, 17:46–17:48 (US Eastern). Main thread GPT-6 Astra (gpt-6-astra), reasoning effort ultra; it spawned 1 sub-thread: 1 on GPT-6 Astra (effort ultra). Together they read 357,972 tokens (298,368 of them from cache) and wrote 6,827 (1,163 of them reasoning).