We Let Fable 5 Audit Vortex vs Greptile on 805 Real PRs. Here's the Report.

Benchmarks for AI code reviewers are usually vibes. A vendor picks three flattering examples, screenshots them, and calls it a comparison. We wanted something harder to argue with.

So we built a corpus: 805 real pull requests across 24 production repositories where MergeStorm Vortex and Greptile both reviewed the same PR — same diff, same moment, same context. Then we handed the raw data to Fable 5 and asked it to audit us. Not to write marketing copy. To find every place Greptile beat us.

The pipeline was simple and fully automated:

1. Scrape every review comment both bots posted, PR by PR.
2. Match findings across bots with DeepSeek v4 flash — same underlying issue, different wording, counted once.
3. Audit — Fable 5 read the matched data, spot-checked findings against the raw comments, dug through our reviewer's actual source code, and wrote the report below.

3,442 distinct findings later, here is what an independent model concluded — including the parts that stung.

The headline numbers

Vortex caught 2,542 findings Greptile never mentioned. Greptile caught 727 Vortex never mentioned. That is a 3.5-to-1 ratio on unique catches, on identical code — and Vortex did it with less noise: an estimated 0.7% invalid-finding rate versus Greptile's 1.5%.

The severity split matters more than the raw counts. Among findings only Vortex caught: 935+ were high or critical bugs — race conditions, unchecked transaction receipts, precision loss on money math, security issues like IP-spoofing bypasses in rate limiters. Among findings only Greptile caught, 78% were medium severity or below, and roughly 30% were style nits, test suggestions, and docs tweaks that Vortex skips on purpose.

CodeRabbit appeared on a subset of the corpus and contributed 18 total catches — too few to change any conclusion.

Explore the full breakdown yourself:

What Greptile actually won

An honest audit has to name the gap, so here it is. Fable 5 clustered the 158 validated high/critical findings that only Greptile caught into five recurring themes:

1. Input and CLI edge cases — flags that accept 0 or NaN and silently produce empty arrays, spin loops, or crashes three calls later.
2. Cross-file config and schema drift — a migration creating snake_case columns while the code reads camelCase keys; deployment configs updated in one environment but not the base.
3. Silent fallbacks — failure paths that quietly fall through to a wrong-but-plausible branch instead of erroring loudly.
4. Platform and vendor constraints — the kind of knowledge no diff contains: a framework's in-memory queue being documented as unsupported in production, or a search engine's 60px logo limit for rich results.
5. Caching and TTL semantics — error responses cached for a full TTL, flags that quietly negate intended caching.

Real findings, worth catching. But notice what is not on that list: there is no category where Vortex is structurally blind. Every theme is a checklist item, not an architecture problem.

The twist the audit found in our own source code

This is our favorite part of the report, because we did not see it coming.

Fable 5 read Vortex's reviewer source and found that the comparison was rigged against Vortex — by us. Vortex feeds prior bot comments into every review pass with an explicit instruction: if another reviewer already raised a theme, skip it — a reworded version of a prior finding is still a repeat. Greptile's bot account ships in that prior-reviewer list by default.

In other words, on every one of the 805 PRs, Vortex read Greptile's comments first and deliberately declined to repeat them. It was designed to complement other reviewers, not compete with them. A chunk of that 727-finding "gap" is territory Vortex was actively yielding. Head-to-head, with deference off, the gap shrinks before we change a single prompt.

What we're doing with it

The audit ends with a recommendation and we are taking it:

• Close the five themes at the prompt level. Each one becomes an explicit checklist item in Vortex's review instructions — input edge cases, config drift, silent fallbacks, vendor constraints, cache semantics.
• Stop deferring to reviewers that are no longer in the room.
• Keep the noise floor. The audit's acceptance gate: recover at least 70% of the validated Greptile-only high/critical findings on a held-out sample, without letting the invalid rate rise above 2%.

The same scrape-match-audit pipeline keeps running as a regression monitor, so "did we actually close the gap" is a number, not a feeling.

The takeaway

If you are evaluating AI code reviewers, ask vendors for this: same PRs, same diffs, findings matched issue-by-issue, judged by a model with no stake in the answer, gaps published including the ones that hurt.

That is the benchmark that matters — not a screenshot of one good comment.

Vortex reviews your first 100 PRs free — install MergeStorm and run your own head-to-head. Methodology note: findings were matched and validity-judged by DeepSeek v4 flash with manual spot-checks; the audit report and analysis were generated by Fable 5. Corpus: July 2026.