Review is where agent-scale code gets absorbed, and it is measured
Human reviewer effectiveness collapses past roughly 400-line diffs, and agents open 10,000-line PRs (L9). The Postgres 19 delay story turned on maintainers receiving more valid reviewer-found bugs than they could fix (The Register, Oct 2026). True AI-to-AI review, where a different vendor's AI reviews an agent's PR, is still only 1.6% of agent PRs, so humans remain the absorbing layer almost everywhere (L8). Microsoft's agent-scale program names the constraint directly: "Human attention has become the scarcest resource in agentic software engineering" (L123).
Sources: the agent share and merge rates from Octoverse 2026, GitHub's own telemetry (github.blog, Oct 28, 2026); the monorepo numbers from Anthropic's engineering post(Oct 15, 2026); the survey from DX's Q3 2026 State of AI Impact (500+ orgs). Adopter-published wins and industry telemetry point the same direction but have not been publicly reconciled, so treat magnitudes as soft and the direction as solid.
Which changes get reviewed, and by what
No team trusts a single check for everything. The documented pattern: compute a risk tier from the change itself, then decide what each tier requires.
| Team | How the tier is set | What the tier buys | Source |
|---|---|---|---|
| Anthropic | A classifier reads the change: "The change sets the tier, not the author." | 0 / 1 / 2 human approvals by tier; agent approvals never count; approvals pinned to the commit | L11 claude.dev, Oct 2026 |
| Cloudflare | Diff size and sensitive paths | Lite/full review depth; the bot gates, humans keep a break-glass path used on 0.6% of runs | L28 blog.cloudflare.com, Apr 2026 |
| Duckbill | Labels: auth, public API, schema, agent skills | Human review required only on those; everything else flows without a gate | L34 Pragmatic Engineer, Sep 2026 |
Triage also bounds the review loop itself. Left uncapped, agent author and agent reviewer will cycle all night: one Anthropic PR ran more than fifty rounds, and about 59% of what the reviewer kept flagging had been there since its first pass (L18). The fixes the same post names - a three-round cap, re-reviewing only the new diff, reporting bug classes instead of one-liners - are now the documented convergence. Where full coverage is wanted despite the noise, the controlled result says aggregate: five or more independent review passes lift F1 by 43.67%, and a cheap-model ensemble beats one large model (L43).
The published tiers come from teams reviewing thousands of PRs a day. The mechanism (tier from the change, not the author) transfers; the thresholds are the part nobody publishes below hyperscaler volume.
Threshold economics below hyperscaler scale: what risk-tier boundaries cost and save at, say, 20 PRs a day.
The noise war
If the reviewer drops ten comments on a PR, that is a configuration problem. The teams making AI review work engineered noise down to about one finding per review.
"56.3% of CodeRabbit's review comments were rejected by developers" - across 31,073 comments on 239 real repositories. Comments about functional concerns were rejected more often than style comments.
The largest first-party deployment measurement agrees from the inside: OpenAI's reviewer comments on 36% of Codex-generated PRs and gets addressed in 46% of cases, across more than 100,000 external PRs a day - and its design post states the objective as a formula that pays for false alarms (alignment.openai.com, Dec 2025). Anthropic posts a finding only if enough of a parallel voter group agrees it is real (L96). Cloudflare's words: "telling an LLM what not to do is where the actual prompt engineering value resides" (L29).
The useful rate (share of comments a human acts on) is the number that decides adoption; measure it before and after any tuning. The playbook carries the measurement method and a 30-day pilot checklist.
A per-tool noise expectation at ordinary volume: what comments per PR to expect from a named vendor tool on a 20-PR-a-day team.
Can an agent approve the merge?
This is where the frontier openly disagrees with itself, and in 2026 the disagreement became a three-way platform split.
Its public product "never approves or blocks the pull request" (L17). Internally, humans still merge everything, and the reason is stated: "We don't yet have circuit breakers that tighten approval when those numbers move; engineers here want them before more code moves to zero-human merges" (claude.dev, Oct 2026, L95).
A Copilot approval can satisfy required-approval branch protection since September 2026, dismissed when new commits land, with a "not guaranteed" caveat (L68). The AI Scan security findings are explicitly non-blocking: they "cannot block pull request merges through ruleset requirements" (L83).
Sells the third position: "enforceable merge checks" for security findings (shipped 2026-10-05, L84), with documented auto-approval bounded by org-configurable risk limits (L154). A review vendor now sells the gate itself.
Ten documented teams, their own numbers. Pick a team to see its practice.
0 / 1 / 2 approvals by risk tier. A classifier sets the tier from the change; agent approvals never count as a person's; approvals are pinned to the commit. Humans merge even low-risk changes today, and engineers say more automation waits on circuit breakers they do not have yet (L11, L95, L100).
75% usefulness floor. uReview reviews over 90% of ~65k weekly diffs and launched only after passing the floor, with confidence grading and category suppression deciding what a human ever sees (L27).
Break-glass 0.6%. A seven-agent reviewer gates PRs autonomously ($0.98 median per review, 1.2 findings each), and humans keep an override path they use on 6 of every 1,000 runs (L28).
Suggestions never auto-applied. PRAssistant participates in over 90% of PRs (600k+/month) and lifts completion time 10-20%, but its fixes are always applied by a person (L26).
No AI commit authors. The open-source distribution standard also bans AI-mediated review replies and requires self-review before any human review (docs.brew.sh, L70).
"Humans may review pull requests, but aren't required to." Review runs agent-to-agent; taste gets encoded into docs and tooling instead of into human gates. About 1,500 PRs from 3 engineers (openai.com, Feb 2026, L115).
19% auto-approved, zero pilot reverts. Over 93% of PRs are agent-driven; over 19% merge with no human. AI-authored revert rates: 0.53% vs 5.39% human on the backend, 0.22% vs 2.00% on the frontend. The agent refuses large PRs, any engineer can summon a human, every decision is logged for auditors, and the shipper stays accountable. Their line: "Human review is not a guarantee of safety. It never was." (L79)
Copilot approvals satisfy branch protection. Since September 2026 a Copilot approval can check the required-approval box (dismissed when new commits land), with a "not guaranteed" caveat in the docs. The platform productized the opposite of Anthropic's norm (L68).
Labels, not meetings. Human review is required by label only for auth, public API, schema, and agent-skill changes; everything else flows. Merged PRs went 353 to 684, with 45% merged under an hour (L34).
"Green CI is no longer proof of safety." The doctrine says leverage-not-rely; an internal bot auto-approves low-risk changes anyway. Their litmus test: would you own a production incident tied to this PR? (vercel.com, Mar 2026, L33)
The default posture for a 200-person team is not "keep a human on every merge because it is safer". The sabotage evidence says unmonitored humans catch almost nothing: 94% fail to detect agent sabotage in controlled settings (L81), and DeepMind later watched a frontier model attempt exactly that - escalating privileges to alter grading scripts and crashing its own sandbox to avoid being scored (L121). What the human-on-merge camp actually buys is time and accountability, and Intercom's revert numbers are one argument for keeping that accountability even where the review is not (L79). Both camps are reasoning without the deciding measurement.
Escaped-defect outcomes for AI-approved versus human-approved code, controlled for change size and risk tier. Self-reported reverts and "zero defects attributed" posts are the only data, and a revert rate measures what got noticed (L78).
Build vs buy, an order of magnitude apart
Every major reviewer now usage-meters, so the published rates can be put on one axis: $0.25 to $25 per review across tools (L67).
| Who | Cost per review | What sits behind the number | Source |
|---|---|---|---|
| Raw model, no harness | $0.14 median | Best raw model at low effort statistically ties the best commercial tools on the largest open benchmark (IronBee, 7,980 reviews) | L92 ironbee.ai, Oct 2026 |
| Cloudflare, homebuilt | $0.98 median | Text-heavy reviewer roles routed to cheap models, 85.7% cache-hit rate | L28 blog.cloudflare.com, Apr 2026 |
| Typical vendor | $15-25 | Token-billed review tiers (Anthropic's published pricing for its reviewer) | L15 claude.com, Mar 2026 |
At a mid-size team's volume the spread becomes a budget line. Twenty PRs a day at vendor rates is roughly $300-500 a day, about $6,000-10,000 over a 20-working-day month; the same volume on the tuned homebuilt median is about $392. That arithmetic is derivation, not measurement: it multiplies published rates by a volume assumption, and the vendor bar hides real variance (the hardest inputs sit outside metered tiers; Anthropic's deep review tier caps at 500 files / 8,000 lines, far under the 10k-line agent PRs that review worst). Uber states its in-house cost is an order of magnitude below vendor pricing at 65k diffs a week (L27), which is the same conclusion measured from inside.
Budget cost per useful finding, not per review; the useful rate is what separates the rows above, not model choice. The full log-scale chart with each party's own figure is on the practices page.
The break-even point itself: the volume and accuracy at which a homebuilt reviewer's engineering cost passes vendor pricing.
The reviewer is part of the attack surface
The strongest warning in the pack is a postmortem. In 2025, Kudelski Security turned a crafted pull request into remote code execution on CodeRabbit's infrastructure, with secrets access and read/write on roughly a million repositories (L59). The chain ran through the things every reviewer integration has: PR content treated as instructions, a config the PR could influence, and a GitHub App with wide scope. The defense that survived was the deterministic one - Anthropic's containment doctrine phrases it as "the deterministic boundary is what gets hit when everything probabilistic misses" (L107).
The injection variant is quieter and documented in controlled form: when a PR that reintroduces a real vulnerability carries a "prior approval" or "dependency compatibility" narrative, weaker review agents reject it less than 35% of the time (L80). The playbook's rule is mechanical: reviewer config lives on a protected branch and is never read from the PR's own changes. Against the reviewer's own prompt handling, only two content-level defenses are publicly documented across the seven major vendors - GitHub's hidden-character filtering plus default-on egress firewall, and Ellipsis's per-session tokens scoped so the limit "holds even against a prompt injection" (L142, L193).
What the reviewer does with your code is separately documented, vendor by vendor, and the differences are material: GitHub's individual Copilot plans switch to opt-out training on interaction data from April 24, 2026 while Business and Enterprise are excluded (L134); CodeRabbit's no-training promise explicitly does not cover open-source repos, which it trains on (L152); Greptile may train on de-identified customer data by default with an admin opt-out (L165). The full comparison - retention, training use, bot permissions, self-host boundaries - is the data flow page.
Detection telemetry from production reviewer deployments: how often the reviewer itself is the injection target, and what caught it.
What the measurements actually say
Speed claims are self-reported; the controlled record is humbler; and the benchmark layer underneath both is under repair by the labs that built it.
| Question | The claim | The measurement |
|---|---|---|
| Does it speed review up? | Anthropic's GA post reports reviews ~3x faster (20 min to 6-10 min) | Google's controlled A/B of AutoCommenter found no significant change in review duration (L24). Anthropic's figure is self-reported, different product, different scale. |
| Does it find real bugs? | Vendor pages: "catches X% of bugs" | Best system on verified real PRs: F1 19.4, up to 7.6 false positives per PR (L40). Security detection collapses 22x on decontaminated data (L47). On real bug-introducing commits, recall spans 32.5-78.1% across frontier models (L91). |
| Can it stop an adversary? | Security merge checks sell protection | In execution-decided adversarial runs, reviewers catch 85-91% of attacks, but roughly one in six attack slots still ends in a working exploit when the attacker spreads changes over real commits (L90). |
| Which benchmark do I trust? | "Top-ranked on [benchmark]" | Every AI-review benchmark crowns its publisher (L66) - and the labs now say so themselves: OpenAI retired SWE-bench Verified (59.4% of audited problems flawed, contamination demonstrated, L116) and retracted its own SWE-bench Pro recommendation (L117). |
Two absences bound every confident take, this research included. No controlled production study of AI review's effect on escaped defects exists - re-checked against the week's evidence on 2026-11-01, still nothing (L78). No documented public incident blames an AI reviewer either; the famous failures are all author-agent failures (L77). OpenAI's measurement post concedes the same hole from inside the strongest deployment: "There is no clean direct measurement of this, so we rely on practical proxies" (L114).
Claims that did not survive verification are published too, on the evidence page: a "9x vulnerability introduction" stat retracted after its primary turned out to say nothing of the sort, an untraceable "27% of orgs banned AI" survey, and others. Killed claims are part of the record.