On the Cold EyeThe Cold Eye — Why AI Must Review AI

In the previous article (№ 007), I mentioned a plan covering 23 bugs that had “converged” after 3 rounds of Claude self-review, only for 6 rounds of independent review by Codex to turn up 60+ more issues. This article is about why that happens, and how to make independent review a discipline.

The problem is not that “AI is not smart enough.” It is structural.

An agent’s own plan can cloud its judgment

As the lead agent writes a plan, it embeds its own assumptions in it. When it comes back to review its work, what it sees is “the plan in my head” rather than “this plan on the page.”

This clouding of judgment takes several concrete forms:

It tends to trust what it has already written. “I wrote earlier that this function was transactional, so I can keep using it that way” — but that earlier judgment may itself be wrong.

It tends to see what it wants to see. Looking at a piece of code, it sees only the parts that “support my plan,” missing the code alongside them that would overturn it.

It tends to settle for “good enough.” Once it feels the plan more or less works, it stops digging and starts wrapping up.

This is not a failing unique to LLMs. Human engineers have it too. The only difference is that we build code review tools so another pair of eyes can look at the work; having AI review AI follows the same reasoning.

7 kinds of structural blind spots

Over the past few months, I have grouped the issues Claude misses in self-review into several categories. These are structural rather than random bugs: they recur, and we can address them deliberately.

① Misreading where a commit happens. In a 50-100-line Read window, the agent takes “there is a commit() somewhere” to mean the function itself is transactional. In fact, the outer caller handles the commit, and the one inside is a leftover from testing.

② Estimating the number of callers by eye. “This change affects 17 callers,” when grep -c gives 22. That sounds close enough, but the 5 missed callers are often where the bugs come from.

③ Inventing APIs from training data. The most dangerous category. The agent will write “access WalletLockResponse.history_id” — a field that sounds reasonable but does not exist in the code. It is something imagined from training data, plausible-looking and nonexistent.

④ Inventing feature objects and filenames. “The backfill file 001_init.sql” — but the sql/ directory actually starts at 008. “auto_accept_trade is in trading.listings”; it is actually in user.profiles. The agent invents things by intuition; a quick ls gives it away.

⑤ Case and enum mismatches. The Python enum is actually lock_for_trading, but the plan casually writes it as LOCK_FOR_TRADING. As soon as the change goes live, the CHECK constraint triggers an error and a rollback.

⑥ Contradictory state machines. One paragraph says “remove the LOCK_ATTEMPTED state,” while another still references LOCK_ATTEMPTED. The plan’s author does not reconcile the paragraphs, but the code will hit the problem on the day it goes live.

⑦ Ordering at the architectural level. A single Read cannot reveal whether the order of cross-service calls within a transaction will cause inconsistencies. This category is the hardest to spot because finding it requires “running through it in your head,” and an LLM is very prone to skipping steps when it does that.

I now list all 7 categories in the team’s PITFALLS document, and give them to Codex as a checklist before every external review.

How an independent context fills the gaps

The key words are independence, distrust, and direct verification.

Independence: Codex starts a new session with no memory. It has not seen how this plan took shape or participated in any of the arguments. It reads the plan as it would an unfamiliar paper: every claim needs evidence.

Distrust: it does not accept narratives such as “we discussed this earlier, so…” in the plan. Nor does it accept shortcuts such as “obviously, X works this way.” It wants to see file:line.

Direct verification: it reads documents and runs commands. grep / Read / MCP DB query — every conclusion must be checked on the spot, without relying on memory or intuition.

Documentation may drift from production (historical migration drift / missed deployments). Direct inspection of the production schema through MCP is the only authority.

This point is especially important. When writing a plan, we often assume “the code is as the documentation describes it.” In reality, the code changes every day, the documentation may not have been updated for two months, and the schema may have drifted after a hotfix migration.

In that situation, the only reliable approach is to ask the database. MCP tools connect directly to the production schema. Use SELECT to inspect table structures / CHECK constraints / indexes, and what you see is what is actually there.

During EVT-20260427-001, audit-agent used MCP to check production directly and found an issue that had survived 6 rounds of external review + 3 rounds of self-review on the plan: the email dispatch function hardcoded an old notification type enum that did not match the types added in phase 3.3. Had this gone unnoticed, dispatch would have failed for every new order on launch day. Because it was asynchronous, users would not have seen it; failures would simply have piled up quietly in the logs.

This is the blind spot that “mental contamination” hides, and exactly where an independent context + direct verification can fill the gap.

In practice: /audit must run in an independent context

I now have two firm rules for running /audit:

1. It must run through an independent agent. Start it with Agent(subagent_type='general-purpose', run_in_background:true), running in the background to avoid contaminating the lead agent’s context. The lead agent has already seen the plan + research + implementation code. Its judgment is already clouded; it cannot be objective.

2. It must have MCP tools for direct verification. audit-agent must be able to query the production DB schema directly (mcp__ggboys-deploy__tool_db_query). Documents + grep alone are not enough.

The prompt for audit-agent includes an explicit checklist of required checks:

· Security (injection / permissions / logging of sensitive fields)

· Correctness (business logic / state machines / edge cases)

· Resilience (exceptions / retries / idempotency)

· Installation readiness (migrations / configuration / dependencies)

· Documentation consistency (drift between documentation and code)

· Cross-file consistency (field names / enum values)

· Performance (N+1 / indexes / locks)

· System-wide effects (architectural impact)

Produce a structured report: severity (CRITICAL / HIGH / MEDIUM / LOW / INFO) + file:line + suggested fix.

The rule: HIGH issues must be fixed before deployment to prod. Mark MEDIUM/LOW issues for phase 4 followup.

The fundamental difference between the two reviews

Set Claude self-review alongside Codex external review, and the difference is structural:

Claude self-review — context shaped by prior involvement / trusts its own plan / tends to settle for “good enough” / easily anchors on its own line of thought.

Codex external review — a cold eye / trusts no outside account / checks every conclusion with grep against production code / unafraid to expose mistakes / independence through a fresh session with no memory each round.

So this is what you see: Claude starts declaring “convergence” after 3 rounds of self-review; Codex immediately finds 4 blockers in round 1, and finds new issues in every round. Only around rounds 5-6 do the returns begin to diminish.

This does not mean Codex is stronger than Claude. Reverse the roles, with Codex writing the plan and Claude reviewing it, and the result would probably be a mirror image. What matters is independence, rather than which model is better.

An unexpected philosophy

When I first encountered this, my initial reaction was frustration: “I’m using an AI that is supposed to be very capable, and it cannot even review its own plan accurately?”

After more runs, though, I grew calmer. AI’s blind spots are structural rather than random. That is actually good news.

Structural means a process can compensate for them. My workflow now is:

research → makeplan → annotationloop self-review → one broad Codex external review → address annotations → focused / final-go-check external review if needed → implementation → /audit independent review → deployment.

Every link added to this chain closes off a known class of blind spots. It looks cumbersome, but compared with the cost of a production incident, it costs almost nothing.

At a deeper level, this has changed how I understand AI’s limits. AI is not “an all-purpose assistant that can write code.” It is a set of tools with different strengths in different roles. Ask the same model to write, and it produces a certain style of code + certain kinds of blind spots; ask it to review, and it can see some things it missed while writing.

Putting it in the right role matters far more than asking it to do more.

---

The next article (№ 009) offers engineering notes on AI working memory. Getting AI to remember you is a matter of discipline in collaboration, rather than a technical problem.

Stay calm, keep going.

Translation log: translated from the Chinese original by Codex on 2026-11-03, 17:51–17:54 (US Eastern). Main thread GPT-6 Astra (gpt-6-astra), reasoning effort ultra; it spawned 1 sub-thread: 1 on GPT-6 Astra (effort ultra). Together they read 419,597 tokens (389,632 of them from cache) and wrote 8,638 (1,757 of them reasoning).