AI code review tools: what they actually catch
Othman Shareef · July 8, 2026 · 6 min read · AI and Code Review
AI code reviewers are the hottest aisle in dev tooling, to the point where one of the vendors wrote an essay titled “There is an AI code review bubble” and 250 commenters mostly agreed. Strip away both the hype and the backlash and a real question remains: what do these tools actually catch?
What they’re genuinely good at
The 9-in-47 is real value, and it clusters: forgotten error paths, mismatches between a comment and the code below it, obvious-but-easy-to-skim bugs in tedious code, missed null cases in languages without strict checking. AI reviewers read every line with identical attention, the one thing humans reliably don’t do. As a mechanical first pass that runs before any human looks, that’s legitimately useful, which is the role we recommended in our tools comparison.
Where the noise comes from
- Linter overlap: the bulk of suggestions duplicate what deterministic tools already encode, except slower, non-deterministically, and phrased as a paragraph you have to read.
- Speculative findings: the most-echoed complaint in the bubble thread. Paraphrasing one commenter, it’s hard to stop the tool giving you twenty speculative concerns wrapped around the one real error.
- Missing context: the tool doesn’t know your schema invariants, your deliberate design exceptions, or the incident that explains the weird retry logic, so it flags exactly those things, repeatedly.
- Run-to-run variance: the same diff reviewed twice yields different comments, which corrodes trust in subtle ways a static analyzer never does.
The benchmark problem
Every vendor publishes accuracy numbers; the numbers disagree wildly; and each benchmark’s winner tends to be its publisher. This isn’t (necessarily) bad faith: catch rate trades against false-positive rate, so any tool can look like the leader by choosing where on that curve to measure. The only benchmark that predicts your experience is your own repo: take a dozen merged PRs whose bugs you later found in production, run the candidate tool on them, and count three things: real catches, linter-duplicates, and confident nonsense. An afternoon of that beats every comparison page, including this one.
The economics test
Dedicated AI reviewers run roughly $24 to $40 per seat monthly. The honest comparison isn’t “versus nothing.” It’s versus a tightened ESLint config (free, deterministic, zero hallucinations) plus the same model called directly in CI with your own rules. And one step further out: the bottleneck these tools address sits next to a bigger one: the speed at which humans can read and judge a change. That one doesn’t improve by adding more comments to triage; it improves by making the reading faster, which is the bet we took instead. More on that division of labor in the final piece of this series.