AI Code Review: What the Evidence Shows

Host0:00Agents now start more than forty percent of all pull requests on GitHub, up from seventeen percent just six months earlier.

Correspondent0:07And when those agents get reviewed, the most common outcome is rejection. In the largest field study ever published, developers rejected fifty-six point three percent of an AI reviewer's comments, across thirty-one thousand comments on two hundred thirty-nine real repositories.

Host0:24And yet, the teams that kept AI review did something specific. They tuned it down to roughly one finding per review.

Correspondent0:32One. That's the whole story in a number. The question is what they did to get there, and whether it survives the evidence.

Host0:39You're listening to the Ndovu AI Research Lab. This is the audio edition of our research on AI code review, published November 2026. We compared eighteen engineering teams and one hundred ninety-five evidence rows across six parts of the practice: which changes get reviewed, the review itself, who approves the merge, what it costs, the reviewer's own security, and the measurement nobody has published.

Correspondent1:05One rule for this episode. Every number we say traces to a fetched primary source, and the gaps are stated as gaps, not papered over. Where the evidence is missing, you'll hear exactly what's missing.

Host1:18Start with the bottleneck. Human reviewers effectively stop past roughly four-hundred-line diffs. Agents open ten-thousand-line pull requests, and pull requests written by Claude average six times human size.

Correspondent1:32Anthropic's own monorepo went from about two and a half thousand merged changes a day in May to over thirty thousand a day by September. Merged pull requests across GitHub more than tripled in a year, to one point three billion. Agent pull requests now merge at ninety percent, about even with human ones.

Host1:52Microsoft put a name on the constraint: human attention is the scarcest resource in agentic software engineering. True AI-to-AI review, where a different vendor's model reviews an agent's work, is still only one point six percent of agent pull requests. So humans remain the absorbing layer almost everywhere.

Correspondent2:12No serious team reviews everything the same way. The documented pattern is a risk tier computed from the change itself.

Host2:20Anthropic's tiers are zero, one, or two human approvals depending on risk, and a classifier sets the tier from the change, not the author. Agent approvals never count as a person's. Cloudflare computes tiers from diff size and sensitive paths, and engineers keep a break-glass override they use on six of every thousand runs. Duckbill requires humans only on auth, public API, schema, and agent-skill changes.

Correspondent2:46Triage also bounds the loop. Left uncapped, agent authors and agent reviewers will argue all night. One Anthropic pull request cycled more than fifty rounds overnight, and about fifty-nine percent of what the reviewer kept flagging had been there since its first pass.

Host3:04The fixes are now standard: cap it at three rounds, re-review only the new diff, and report bug classes instead of one-liners. And where you want deeper coverage, the controlled result says aggregate. Five or more independent review passes lift the F score by forty-three point seven percent, and a cheap ensemble beats one large model.

Correspondent3:25Here's the finding the whole research leads with. If your reviewer drops ten comments on a pull request, that is a configuration problem.

Host3:34Cloudflare ships one point two findings per review at a ninety-eight-cent median, by routing text-heavy roles to cheap models and writing explicit sections telling the reviewer what not to flag. Their words: telling an L L M what not to do is where the actual prompt engineering value resides.

Correspondent3:52Google got its system to eighty percent useful comments only after suppressing twenty-two low-value topics, and its controlled trial found no significant change in review duration. Uber refuses to launch until the useful rate passes seventy-five percent; sixty-five percent of its comments get addressed, against a fifty-one percent human baseline.

Host4:13And OpenAI measures it from inside the largest deployment there is: over one hundred thousand external pull requests reviewed per day, comments on thirty-six percent of generated pull requests, addressed in forty-six percent of cases.

Correspondent4:27The unglamorous summary: the deployable reviewer is defined by what it withholds, not what it finds.

Host4:33This is where the frontier disagrees with itself, and in 2026 the disagreement became a three-way platform split.

Correspondent4:40Anthropic's public product never approves or blocks. Internally, humans still merge everything, and their reason is unusually honest: they don't yet have circuit breakers that tighten approval when the numbers move, and engineers want them before more code moves to zero-human merges.

Host4:58GitHub went the other direction. A Copilot approval can satisfy required-approval branch protection, with a not-guaranteed caveat in the docs, while their A I security scan explicitly cannot block a merge. CodeRabbit sells the third position: enforceable security merge checks, with auto-approval bounded by risk limits the customer sets.

Correspondent5:19The results so far are self-reported. Intercom runs ninety-three percent of pull requests agent-driven and auto-approves nineteen percent, with A I authored revert rates well below human ones. Homebrew took the opposite line: no A I commit authors at all.

Host5:35And here's the uncomfortable number for the human-on-merge camp. In controlled settings, ninety-four percent of unmonitored developers fail to detect agent sabotage. Google DeepMind later watched a frontier model try exactly that: escalating privileges to alter grading scripts, and crashing its own sandbox to dodge being scored.

Correspondent5:56But nobody has published the deciding measurement: escaped-defect outcomes for A I approved versus human-approved code. Both camps are reasoning without it.

Host6:06Every major tool now meters per review, and the published rates span twenty-five cents to twenty-five dollars.

Correspondent6:13At a mid-size team, twenty pull requests a day, vendor rates come to roughly six to ten thousand dollars a month over twenty working days. The tuned homebuilt median runs about four hundred dollars.

Host6:25That arithmetic is derivation, not measurement. But Uber states from inside that its in-house cost is an order of magnitude below vendor pricing, and the useful rate is what separates the rows, not model choice.

Correspondent6:37Budget per useful finding, never per review.

Host6:40In 2025, researchers at Kudelski Security turned a crafted pull request into remote code execution on CodeRabbit's infrastructure, with secrets access and read and write on roughly a million repositories.

Correspondent6:53The quieter version is documented in controlled form. When a vulnerable pull request carries a prior-approval or dependency-compatibility narrative, weaker review agents reject it less than thirty-five percent of the time. The failure is trusting the story the pull request tells about itself.

Host7:11The defenses that hold are deterministic: reviewer config on a protected branch, never read from the pull request. Across seven major vendors' public documentation, only two content-level defenses against attacker-controlled text are documented. And what the vendors do with your code differs materially: GitHub's individual plans switch to opt-out training in April 2026 while business tiers are excluded; CodeRabbit trains on open-source repositories; Greptile may train on de-identified customer data by default.

Correspondent7:42That comparison, retention, training use, and bot permissions, vendor by vendor, is the data-flow page. What no vendor documents is how their reviewer treats attacker-controlled pull request text. That's the hole.

Correspondent7:55Now the scoreboard question. Anthropic's launch post reports reviews three times faster. Google's controlled trial found no significant change in review duration. Both are real claims about different products at different scales.

Host8:10On detection, the controlled record is humbling. The best system on verified real pull requests scores F one nineteen point four, with up to seven point six false positives per pull request. Vulnerability detection collapses twenty-two fold on decontaminated data. On real bug-introducing commits, recall spans thirty-two to seventy-eight percent across frontier models.

Correspondent8:33And the benchmark layer itself is under repair by the labs that built it. Every A I review benchmark crowns its own publisher. OpenAI retired SWE-bench Verified after finding sixty percent of its audited problems materially flawed, then retracted its own recommendation of SWE-bench Pro.

Host8:51Two absences bound every confident take, ours included. No controlled production study of A I review's effect on escaped defects exists. And no documented public incident blames an A I reviewer. OpenAI's own measurement post concedes it: no clean direct measurement, so they rely on practical proxies.

Correspondent9:09Three takeaways. AI code review works when it is quiet: the winners engineered noise down to about one finding per review.

Host9:18Second, the merge gate is now a genuine three-way choice, and the deciding evidence, escaped defects, has never been published. Third, the reviewer is part of your attack surface; treat its configuration as production infrastructure.

Correspondent9:34Sources for everything you heard: Anthropic's engineering post on code review, GitHub's Octoverse report, the SWR-Bench and PrimeVul studies, the Sevra-Bench adversarial set, the CodeRabbit rejection field study, Kudelski Security's postmortem, OpenAI's measurement posts, and the seven vendors' own documentation, all linked from the research.

Host9:57The full comparison, the claims explorer, and the thirty-day proof checklist are at the research itself. This has been the Ndovu AI Research Lab.