25 Bugs My AI Reviewer Caught Before They Shipped

In the first three days of rebuilding a platform with AI agents, an independent review caught more than 25 serious defects before a single one reached a commit. The code compiled. The tests were green. The agent said everything worked. Here is what was actually wrong, and the rules that caught it.

The setup: whoever writes the code doesn't approve it

The project is a platform for charitable donations that I'm rebuilding from scratch. Donations are held in a smart contract and paid out in tranches that donors vote on. Real money, real rules, no room for "it mostly works".

I split the work into three roles. This split is the whole method.

Role	Who	Does	Never does
Owner and engineer	Me	Product decisions, branches, every commit, server commands, secrets	Write most of the code
Reviewer and architect	Claude, in a chat	Specs, architecture decisions, tasks, agent prompts, plan reviews, code reviews in the repository	Commit, or ever see a secret
Implementer	An AI coding agent in my IDE	Writes the code, the tests and a feedback file per task	Commit, deploy, touch keys, or change the server without approval

Why three roles? The agent that writes the code is the worst possible judge of it. It knows what it meant to do, so it reads what it meant, not what it wrote. A second AI with a different job, and a human who owns every commit, close that gap.

How every task runs

No code is written before the documents exist. A manifest with the rules for the agent, a product spec, an architecture document and a log of decisions with their reasons. The agent has no memory between sessions. These files are its memory.

Then every task follows the same loop:

1. The reviewer writes the task: scope, tests, what must not be touched, acceptance criteria.
2. I create the branch. The agent never does.
3. The agent reads the documents and answers with a plan. No code yet.
4. The reviewer checks the plan and sends back changes. Only then: go.
5. The agent implements, runs the tests and writes a feedback file: what it did, what it changed, every deviation from the spec.
6. The reviewer reads the code in the repository, not the agent's summary.
7. One to three rounds of fixes.
8. I check that no secret is in the diff, commit, push, and merge after CI passes.

Step 3 alone paid for itself. Almost every plan needed changes before a line of code existed, and changing a plan costs minutes. Changing shipped code costs days.

What the review caught

Every one of these passed the agent's own checks. Most of them would have passed a quick human glance too.

Area	What was wrong	What would have happened
Voting on payouts	A voter's weight was read at the moment of the vote	Anyone could deposit during a vote, outvote the others and move more money than they put in
Voting on payouts	A pool with no voters counted "0 votes of 0" as a pass	The operator could move donated money with zero votes
Donor rules	A donor's "reject" vote went to the admin instead of rejecting	The product's core promise broken, and the agent didn't mention it
Login plan	The server trusted the wallet address sent by the browser	Anyone could claim someone else's donations
Login plan	An admin script would have removed my own access	The owner locked out of his own platform
Admin settings	No limits on configuration values	A success threshold of 0 makes a campaign with €0 "successful"
Tests	A security test accepted any failure as a pass	Green tests, no actual protection
Feedback	The summary described functions that didn't exist in the code	Wrong documentation for the security audit
Dependencies	A library tagged as v5.7 was actually v4.8	Building on the wrong version without knowing it
UI	The logo disappeared in dark mode: black on black	Automated tests passed; only looking at a screenshot caught it
Deploy	The agent skipped the required local Docker test	The bug surfaced only on the server
Secrets	The agent proposed storing the backup's private key on the server	One breach and the backups are readable too

The pattern behind the table: the agent rarely writes nonsense. It writes plausible code that does something slightly different from the rule, and then describes it as if it followed the rule.

The rules I now enforce

1. Questions before plans. An AI that doesn't ask is guessing, and a guess at the start spreads into every task after it.
2. Documents are the agent's memory. Rules, spec, architecture and decisions live in the repository, not in a chat.
3. Whoever writes doesn't approve. One agent implements, a different one reviews, a human commits.
4. Plan before code. Always. Changing a plan is cheap.
5. Review the code, not the summary. Summaries claimed events, functions and wiring that weren't there.
6. Proof by output. Every "it works" comes with the command output that shows it.
7. Deviations must be declared. "Deviations: none" is the most common false sentence in AI feedback.
8. Money and security get boundary tests. Exactly at the limit and one unit below, plus invariants and counters that prove the path actually ran.
9. UI gets screenshots. Light and dark, desktop and mobile, compared with the design by a pair of eyes.
10. Secrets never go into the chat, the repo or the agent. When a password was pasted into the chat by mistake, it was reset the same minute.
11. The owner makes the commits. Whoever controls the history controls the project.

The numbers so far: a working dev environment in 2 days, 8 tasks done, 241 contract tests, about 2 fix rounds per task, and 25+ serious defects caught before commit. The project is still in progress.

What this means for your AI-built product

AI didn't make these mistakes because it's bad. It made them because nobody stood between "the code exists" and "the code is right". That gap is exactly where AI-built MVPs break once real users and real money arrive.

If your product was built fast with AI and you're not sure what's sitting in that gap, that's the work I do. I review the architecture, the database rules and the security, and you get a clear list of what to fix first.

Book a 15-minute intro call

Building with AI yourself and not a developer? The same loop, made simple, is in my guide Vibe Coding 101.