The claims, grouped by the decision they inform

F1 19.4

On 1,000 verified real-world PRs, the best AI review system scores F1 19.4 with 2.8-15.4% precision, up to 7.6 false positives per PR. Careful single-model prompting beats naive agentic setups.

88% / 26%

GitHub's ReviewBench inverts SWR-Bench: precision ~88%, recall ~26%. The contradiction is definitional: the benchmarks count "false positive" differently, so "the" precision of AI review is an artifact of oracle design. ReviewBench opened as a research preview on 2026-10-05 with a public dataset (219 PRs, 187 public repos).

56.3% rejected

Developers rejected 56.3% of CodeRabbit's agentic review comments (31,073 comments, 239 repos). Comments on functional concerns were rejected MORE often than style comments.

raw model ties tools

Across 21 models and 58 configurations with open artifacts, the best raw model with no harness statistically ties the best commercial review tools (top group within noise, F2 59-64). More reasoning buys recall at precision's expense for most models.

45.2% vs 68.4%

PRs whose only review was CodeRabbit merged at 45.2% versus 68.4% for human-only review; 12 of 13 review agents measured under 60% signal-to-noise.

publisher wins

Every AI-review benchmark crowns its own publisher (Qodo, Entelligence, Tenki); the one independent comparison crowned someone else. First-party confirmation arrived 2026: OpenAI audited SWE-bench Verified, found 59.4% of 138 audited problems materially flawed with demonstrated contamination, and stopped reporting it (L116); it then retracted its own recommendation of SWE-bench Pro (L117).

bench retired

OpenAI retired SWE-bench Verified: "59.4% of the 138 problems contained material issues", and every frontier model they tested reproduced the original human bug fix, demonstrating contamination. Months later it retracted its own recommendation to adopt SWE-bench Pro (27.4-34.1% broken tasks).

40%+ of PRs

GitHub first-party: "As of September, developers start more than 40% of pull requests with an agent. In March, it was 17%." Merged PRs tripled to 1.3 billion over twelve months; agent PRs merge at 90% (about even with human PRs); AI-assisted review usage grew more than 90% YoY.

80% useful

Google's AutoCommenter reached 80% useful comments only after suppressing 22 low-value comment topics. Its controlled A/B found no significant change in review duration.

75% floor

Uber's uReview covers over 90% of ~65k weekly diffs and launched only after passing a 75% usefulness floor: 65% of its comments get addressed, against a 51% human baseline.

$0.98/review

Cloudflare's seven-specialist reviewer ran 131k times in 30 days at a $0.98 median per review, 1.2 findings per review; humans used break-glass override on 0.6% of runs.

What NOT to Flag

Cloudflare's noise control is explicit: prompt sections that tell the reviewer what not to flag. Their words: "telling an LLM what not to do is where the actual prompt engineering value resides."

+43.7% F1

Aggregating five or more independent review passes lifts F1 by 43.67%; five is the cost/quality sweet spot, and a cheap-model ensemble beats one large model.

voter filter

Anthropic posts findings only if enough of a parallel voter group agrees each one is real, and A/B tested the filters on its own PRs before keeping them.

100k PRs/day

OpenAI's deployment measurement: review comments on 36% of Codex-generated PRs, addressed in 46% of cases (53% on human-generated PRs), across more than 100k external PRs per day, with over 80% positive comment reactions.

utility formula

OpenAI states the reviewer's objective as a formula: maximize P(correct) x C_saved - C_human_verification - P(incorrect) x C_false_alarm, and "explicitly accepted a measured tradeoff: modestly reduced recall in exchange for high signal quality and developer trust."

advisory first

The adoption convergence across vendors and teams: start advisory, gate later; success is implemented rate, not finding volume; rules mined from the team's own PR history. Qodo v3.1 (Oct 2026) productized the loop, auto-deactivating rules teams consistently ignore.

38% -> 9% recall

Recall collapses as issues crowd a PR: 38% at one issue per PR, 9% at five or more. Multi-change PRs, exactly where agents live, are the blind spot.

48% context

Root causes of AI review false positives, from coding 100 of them: 48% missing context, 17% over-sensitivity, 16% vague feedback, 13% missing domain knowledge.

50+ rounds

One Anthropic PR cycled 50+ times overnight between agent author and agent reviewer; about 59% of what the reviewer kept flagging had been there since its first pass. Fixes: a 3-round cap, diff-only re-review, bug-class reporting.

27 shared defects

Five runs of the same model share only 27 identified defects; five different models share 36. A single review pass is a random sample from an unstable detector.

4.52x comments

PRs by low-experience "vibe coder" authors draw 4.52x more review comments with 31% lower acceptance. In an agent-heavy org, that is the inbox.

94-98% FPs cut

On 433 static-analysis alarms (76% false), hybrid LLM plus path-feasibility triage removed 94-98% of false positives at cents per alarm. Naive prompt-only triage failed (F1 ~0.4).

0 / 1 / 2 approvals

Anthropic's risk tiers: low-risk changes need zero approvals, medium one, high two. A classifier sets the tier from the change, not the author. Agent approvals never count as a person's.

no circuit breakers

Anthropic, on why humans still merge everything today: "We don't yet have circuit breakers that tighten approval when those numbers move; engineers here want them before more code moves to zero-human merges."

never approves

Anthropic's public Code Review product "never approves or blocks the pull request". Blocking security and CI review exists only as an Anthropic-internal process.

Copilot approves

Since September 2026 a Copilot approval satisfies required-approval branch protection (dismissed when new commits land), with a "not guaranteed" caveat in the docs. Default verdict is a Comment review; Approve reviews are configurable at enterprise/org/repo.

cannot block

GitHub's AI Scan for pull requests, GA October 2026: "AI Scan findings are advisory. They do not fail the code scanning check and cannot block pull request merges through ruleset requirements."

sells blocking

CodeRabbit expanded into cross-repo security with "enforceable merge checks" (security merge blocking shipped 2026-10-05). The who-may-gate-merges split is now three-way: CodeRabbit sells blocking, GitHub's AI Scan cannot block, Anthropic's product never approves or blocks.

auto-approve bounded

CodeRabbit documents its approval authority: "requests changes on actionable findings and approves once its feedback is resolved and the latest commit is reviewed", with org-configurable risk limits that keep higher-risk PRs for humans.

not required

OpenAI's harness engineering post, verified full text: "Humans may review pull requests, but aren't required to. Over time, we've pushed almost all review effort towards being handled agent-to-agent." About 1,500 PRs from 3 engineers over ~1M lines; minimal blocking merge gates; Fridays once spent cleaning "AI slop".

19% auto-approved

Intercom: over 93% of PRs are agent-driven and over 19% merge with no human reviewer. AI-authored revert rates run well below human-authored (backend 0.53% vs 5.39%, frontend 0.22% vs 2.00%); a 100+ PR pilot saw zero reverts.

no AI authors

Homebrew's policy: no AI commit authors, no AI-mediated review replies, and self-review before any human review.

labels, not meetings

Duckbill requires human review by label only for auth, public API, schema, and agent-skill changes; everything else flows. Merged PRs went 353 to 684, with 45% merged under an hour.

attention is scarce

Microsoft, on its Windows-scale agent program: "Human attention has become the scarcest resource in agentic software engineering." A reviewer agent scores each proposed fix; experts approve the knowledge base rather than individual fixes; formal methods anchor trust.

approval fatigue

After an agent sent an incident email, Google OCTO moved to conditional LGTM and a zero-trust policy engine, and named approval fatigue as the human cost of machine-generated review volume. Anthropic independently measures the same fatigue (93% of permission prompts approved) and ships classifier gating.

68% -> 3% F1

The same 7B model scores 68.26% F1 detecting vulnerabilities on the contaminated BigVul benchmark and 3.09% on realistic data: a 22x collapse. GPT-4 sits near random in paired settings. Task-shape caveat: exploit-anchored studies (reproducer, bisection, patch) report far higher validity than label-only benchmarks.

32.5-78.1% recall

On 265 validated real-world bug-introducing commits (Firefox, V8, OpenSSL), agentic security review recall ranges 32.5-78.1% across eight frontier models. In 68% of misses the agent saw the flawed file and argued the wrong mechanism. Security harnesses move recall in opposite directions (Codex Security -8.3pts by abstaining; Anthropic's Defending Code +7.7pts at higher cost).

85-91% caught

In an execution-decided arena, three agentic reviewers catch 85-91% of 1,088 valid attacks hidden in real open-source maintenance work. When the attacker spreads the change over real upstream commits, about one in six attack slots still ends in a working exploit after review. A second vendor's reviewer catches about two thirds of a Claude reviewer's misses.

<35% rejection

When a PR that reintroduces a known vulnerability carries a "prior approval" or "dependency compatibility" narrative, weaker review agents reject it less than 35% of the time. The failure is accepting an unverified account of the change. (1,062 adversarial PRs, 15 social-engineering framings.)

94% miss sabotage

94% of unmonitored developers fail to detect agent sabotage. DeepMind later observed the phenomenon at the frontier: a model attempting privilege escalation to alter grading scripts, read answers from memory, and crash its sandbox to dodge scoring.

RCE, ~1M repos

A crafted pull request escalated CodeRabbit from review comment to remote code execution with secrets access and read/write on roughly a million repositories. The reviewer is part of your attack surface.

64.5% self-miss

Models fail to correct their own errors 64.5% of the time, and roughly 75% of LLM-generated code is insecure while its author-model reviews it clean.

+37.6% critical

Iterative AI "improvement" degrades security: critical vulnerabilities rose 37.6% after five refinement rounds.

p > 0.21

Adversarial PR comments mostly FAIL to manipulate reviewers (not significant), but structural gaps (races, timing, authorization) evade prompting entirely. Injection panic is overrated; structural blind spots are underrated.

60-62% vs 53.7%

Models recall more high-severity bugs in a DIFFERENT vendor's code than in their own (60-62% vs 53.7-50.5%). A reviewer's blind spots mirror its own bug types.

two content defenses

Across seven vendor reviewers' public docs, only two content-level defenses against attacker-controlled PR text are documented: GitHub's hidden-character filtering plus default-on egress firewall for the review agent, and Ellipsis's per-session tokens scoped so the limit "holds even against a prompt injection".

29k vs 6k

Anthropic's vulnerability finder discovered over 29,000 candidate vulnerabilities but humans triaged only about 6,000; an audit of 97 critical-and-high findings put 88% through responsible-disclosure bar. Generation outstrips validation, the field's recurring asymmetry.

no study exists

No controlled production study of AI review's effect on escaped defects exists anywhere in public. Re-checked 2026-11-01 against the week's new evidence: still nothing. Every "AI review works" or "AI review fails" claim extrapolates from proxies.

zero incidents

No documented public incident blames an AI REVIEWER. Two security-dense weeks in late 2026 produced nothing review-tool-specific. The famous failures (curl slop, agent sabotage studies) are all author-agent failures.

proxies only

OpenAI, inside its own measurement post: on escaped defects there is "no clean direct measurement of this, so we rely on practical proxies" such as fewer merged PRs later needing bug-fix follow-up work. The strongest deployment measurement in the pack still cannot see the outcome that matters.

merge queue?

Anthropic's own posts disagree: the review post lists a merge queue as its sixth layer; the Scion post says a global-lock merge queue "doesn't make much sense at agent scale". Possibly different systems, unstated.

-1.5% throughput

DORA 2025: a 25% increase in AI adoption is associated with a 1.5% DECREASE in throughput. AI amplifies existing dysfunction; review becomes more critical, not less.