Alert fatigue comes for code review

Operations teams learned this lesson a decade ago and wrote it on the wall: a pager that cries wolf doesn’t get silenced; it gets ignored, and then the real page scrolls by unread. AI code review is now relearning it, one speculative comment at a time. False positives are the #1 complaint against every tool in the category, and the damage isn’t the wasted minute per dismissal: it’s what the dismissals do to the reader.

The mechanism, not the moral

When a bot leaves nine comments on a PR and two matter, the author must spend their own judgment to find the two, which is precisely the work the bot was bought to reduce. Worse, the triage is repetitive: the same speculative concerns recur on every PR (“consider handling the case where this config is undefined”; it can’t be), so developers build the only rational defense: a mental filter labeled probably nothing. The team that built one popular reviewer documented this exact arc: they had to engineer nitpicking out of their own bot because noisy comments were burying the signal ones. We saw the same dynamic with human nitpicks; machines just produce them at scale.

Tuning for trust

• Deterministic first. Anything a linter or type checker can decide should never reach the AI layer: it’s free precision, and it shrinks the surface the bot can be wrong about.
• Narrow the bot’s beat. Security-sensitive paths, error handling, concurrency: places where a speculative flag is worth reading. Silence elsewhere is a feature.
• Demand confidence, suppress hedging. Most tools accept custom instructions; “only comment when you can name the failing input” filters out the twenty-speculative-reasons genre.
• Measure dismissal rate per bot, visibly. What gets measured gets tuned; a bot nobody audits drifts noisy.
• One reviewer’s voice, not three. Stacking multiple AI reviewers multiplies noise faster than coverage: their false positives don’t overlap; their true positives mostly do.

Protect the scarce resource

The thing alert fatigue actually destroys is the thing review runs on: reviewer attention. Every unit spent dismissing speculation is a unit not spent on the core files of the change. That’s the lens we’d apply to any addition to the review loop, our own product included: does it add attention (by making reading faster, by triaging what matters first) or does it tax attention and call the tax a feature? Tools that can’t answer that cleanly end up in the mental spam folder, no matter what their benchmark said.