Nine rules with named evidence
Each rule exists because at least two independent teams converged on it, or because a controlled study named its failure mode. Nothing here is ours; it is what the public record supports.
Every finding gets exactly one bucket: fixed, rejected-with-reason, intentional, or doesn't matter. Noise is the rejected plus doesn't-matter share, and you report it with its definition attached.
Why: Uber could gate a launch on 65% of comments addressed against a 51% human baseline, while a critic counting differently could dismiss the same system - the definition decides the comparison (L27). Google publishes its useful rate only alongside the 22 suppressed topics that produced it (L24).
Periodically point the reviewer at changes known to be clean and record what it invents. That false-positive rate on clean input is the trust number; finding counts are not.
Why: Anthropic measures noise this way alongside incident evals (L19), and controlled runs put false positives as high as 7.6 per PR (L40).
Three rounds, then escalate to a human or a re-plan. Re-review only the new diff. Report bug classes, not one-liners. Never re-report a finding the author has dismissed with a reason.
Why: one Anthropic PR cycled 50+ times overnight and 59% of re-flags were the reviewer's own earlier findings (L18). The fixes are exactly these caps. Some vendors still productize the unbounded loop.
Reviewer instructions live on a protected branch and are never read from the PR's own changes. If a PR can edit the rules that review it, the reviewer is the attack.
Why: config injection is the documented failure mode; the guidance is to treat reviewer configuration like CI configuration, as trusted infrastructure.
Never let the same model family be the only reviewer of its own output. Cross-model review is the cheapest blind-spot fix that exists.
Why: models fail to self-correct 64.5% of the time (L56); models recall more bugs in a different vendor's code than their own (L55); iterative self-improvement loops actively degrade security (L57).
"Prior approval", "dependency compatibility", "matches the ticket": descriptions are claims, not context. A reviewer (human or agent) verifies them against the diff and the repo before they count.
Why: on 1,062 vulnerability-reintroducing PRs, weaker review agents rejected under 35% when the malicious change carried a prior-approval or compatibility narrative; the failure was accepting an unverified account (L80). Most unmonitored developers also fail to detect agent sabotage.
Tier on size, paths touched, and sensitivity, never on the author's seniority. Reserve human approval for the tiers that need judgment, and make agent approvals unable to satisfy them.
Why: Anthropic's 0/1/2 tiers (L11), Cloudflare's lite/full tiers (L28), and Duckbill's label floors (L34) are the same design reached separately.
Every incident-causing PR joins a corpus; the reviewer is re-scored against it continuously. Pair it with the clean-PR probe so the reviewer is scored on silence as well as speech.
Why: this is Anthropic's daily loop (L19), and it is the only described method that connects review quality to outcomes anyone can observe.
Usage-metered review makes cost a first-class metric. Report it per useful finding, never per review, and route text-heavy roles to cheaper models before buying a bigger one.
Why: the spread is roughly 15-20x between a tuned homebuilt reviewer and vendor pricing (L28, L15), and useful rate is what separates them.
A 30-day proof, with firing criteria
A pilot without pre-committed numbers becomes a purchase. This checklist derives from the nine rules above and the pack's own baselines; set the targets before day one and keep them written where the pilot runs.
Adopt the four-way response taxonomy (rule 01) and brief the team in one paragraph. Pick the pilot repos and the risk-tier boundaries. Record the pre-AI baselines you already have: median time to first review, revert rate, and current comment volume.
Run the reviewer in comment-only mode on every PR in scope. No merge gating, no required approvals (rule 07 keeps that decision separate). Classify every human response daily; the noise number is meaningless before roughly 50 classified responses.
Point the reviewer at three known-clean changes (rule 02). Count what it invents. That false-positive rate on clean input is the trust number; a reviewer that cannot stay silent on clean code fails here regardless of what it finds elsewhere.
Useful rate (share of comments acted on), comments per PR, clean-PR false positives, and cost per useful finding. Judge them against the pack's own anchors: Uber launched at 65% of comments addressed against a 51% human baseline (L27); the largest field study measured 56.3% of an untuned vendor reviewer's comments rejected (L51); WeTravel declined twice on noise that never justified the cost (L36).
After the pilot, five numbers carry the deployment: useful rate, comments per PR, rounds-to-pass on agent-authored PRs, clean-PR false positives, and cost per useful finding. All five come from responses your team already produces; none requires the vendor's dashboard.
You are the meter
The four-way taxonomy is the instrument every serious deployment measures with, and its edge calls are where honest teams diverge. Classify six author responses. The examples are written to exercise those edges; they are not real reviewer output.
Context. A small PR bumps a retry limit and adjusts backoff timing.
Reviewer finding. "Retries are too aggressive: 3 attempts at 100ms backoff will hammer the API under transient failures."
Author response. The author pushes a new commit: 5 attempts with exponential backoff starting at 250ms.
Where the evidence is thin, contested, or skewed
The full map, published here because nobody else does. This section is the operating manual for the evidence base: the places where it is thin, contested, or skewed.
No controlled production study of AI review's effect on escaped defects exists anywhere in public. Every works-or-fails claim, including the confident ones in favor, extrapolates from proxies: merge rates, rejection rates, self-reported reverts (L78).
The numbers everyone quotes (16% to 54% substantive comments, reverts down 40% while PRs doubled, 19% auto-approved, zero defects on self-reviewed diffs) are credible first-party reports with described methods and unpublished data. Most coverage repeats them as settled fact (L2, L13, L35).
Two credible benchmarks give the same tools opposite precision profiles because they define false positive differently. Every vendor benchmark crowns its publisher; benchmarks skew Python-heavy and single-ecosystem; no neutral head-to-head of 2026 tools exists (L40, L50, L66).
Meta, Netflix, Amazon, Apple, Stripe, and every bank or insurer studied are silent. Intuit is the lone large non-tech-native voice; one Asian team and two European ones appear at all. Regulated-industry practice is essentially undocumented (Q6).
Adopters publish cycle-time wins; industry telemetry shows review times rising sharply under high AI adoption (105% longer first-review waits, time-in-review up). The reconciling variable, probably reviewer quality plus adoption intensity, has not been measured publicly (L5, L6).
Famous numbers distort in transmission: "59% pre-existing bugs" means "59% of repeats of the reviewer's own findings"; a 2-3x verification claim is a practitioner's estimate, not a benchmark; "blocking security review" is Anthropic-internal while the shipped product never blocks; METR's famous slowdown result attenuated in 2026 re-analysis (L18, L22, L17, L60).
Rejection rates, security collapse, convergence cost, and throughput mismatch are quantified. Maintainer burnout and abandonment stories are discourse; the widely-quoted claims that failed verification (an untraceable "27% of orgs banned AI" survey, a "9x vulnerability introduction" stat whose primary says nothing of the sort) are published as killed claims on the evidence page.
The human-addressed baseline conflicts across studies: 60% in one million-PR analysis, 51% addressed-as-bugs in Uber's. Both are honest; the definitions differ. Always state the definition when comparing AI against human reviewers (L27).
What would actually settle this
The debate stays open because these results do not exist. When any of them lands, update the conclusions; until then, treat confident takes (including this research's) as provisional.
The honest current state
AI code review produces measurable busy-work reduction and mixed signal quality; whether it reduces the defects that escape into production is unmeasured in public. The teams getting value treat it as a noise-engineering project with risk-tiered human attention on top. The teams getting burned bought a homepage number. The measurement method on this page is how you tell which one you are getting, and the evidence explorer is how you check every claim this research just made.
Sources and method. Everything here distills a 195-row evidence ledger built from fetched primaries: 18+ first-party team accounts, peer-reviewed studies, tooling benchmarks, counter-evidence, a 2026-11-01 refresh of the delta since the original run plus AI-lab-authored material, and a vendor-by-vendor data-flow pass. Original run collected 2026-10-26, with production-time verification updates on 2026-10-27 (Intercom's primary post, the Sevra-Bench paper) and the refresh folded 2026-11-01. Confidence tiers follow the ledger's own columns; the claims that failed verification are published, not silently dropped.