We Blind A/B Tested Haiku 4.5 vs DeepSeek V4 Flash on Real PR Reviews. Then We Went Bigger.

Picking the model behind a code review bot is usually vibes. Someone runs two prompts on one diff, likes one answer better, and ships it. We wanted an actual answer, so we turned Vortex's own initial reviews into a blind A/B experiment and let production traffic decide.

This is a small study and we will not pretend otherwise. But small and honest beats large and imaginary, and the signal was clear enough to act on.

The setup

Vortex posts an initial review on every pull request it monitors. For experiment vortex-initial-ab-v1, we ran both arms on the same work: Claude Haiku 4.5 and DeepSeek V4 Flash each reviewed the same diff, with the same context and the same prompt. One of the two reviews was delivered to the PR, chosen at random, with nothing in the output saying which model wrote it. The reader could vote for the review they preferred without knowing whose work they were grading.

For every pair we logged what actually matters when you run reviews at volume: relative model cost on identical work, how many inline findings each model posted, and whether the two arms reached the same overall verdict.

The corpus: 73 assignments across 12 repositories and 5 users, 66 delivered ballots, and 72 pairs where both arms completed. All numbers below were queried from production on 2026-08-13.

What 72 paired runs showed

Cost was not close. On paired runs of the same diffs, DeepSeek V4 Flash came in at 85.6% lower mean model cost than Haiku 4.5. Median cost moved in the same direction. We are not publishing raw dollars per review.

DeepSeek said more. It averaged 1.74 inline findings per review to Haiku's 1.04. In head-to-head pairs, DeepSeek posted more inlines 31 times, Haiku 6 times, with 35 ties. Volume is not quality, and we flag that honestly below. But an initial review's job is to surface things worth looking at, and one arm consistently surfaced more.

They disagree often enough for the choice to matter. The two arms reached the same overall verdict on 43 of 72 pairs (60%). If the models agreed on everything, the cheap one would win by default. A 40% disagreement rate means model choice is a real decision, which is why we measured instead of guessing.

Humans voted DeepSeek, but barely anyone voted. We collected 3 blind votes, all from our own dogfooding, and all 3 preferred the DeepSeek review. We are reporting that because we said we would report everything. Three votes is an anecdote, not a win rate, and we make no statistical claim from it.

Explore the full numbers yourself:

What this study does not show

An honest study names its own limits, so here are ours:

• The vote sample is tiny. Three votes, all internal. The preference signal is directional at best.
• More inlines is not automatically better. A chattier model could just be noisier. What we can say is that the extra findings went through the same delivery pipeline and the same output constraints, and the readers who did vote preferred the denser reviews.
• The corpus is 12 repositories and 5 users. Real production traffic, but skewed toward our own orbit. This is a dogfood-heavy pilot, not a benchmark paper.

None of these caveats touches the cost result, though. The 85.6% saving is measured on paired runs of identical work, and it does not need a p-value to change your unit economics.

The decision: not one cheap agent, a fleet

The naive move is to swap Core to DeepSeek and pocket the 86%. We are doing something better with the margin: spending a slice of it on coverage.

Vortex is expanding from one initial reviewer to an N-agent DeepSeek fleet, starting at four:

1. Core reviews the diff for correctness and bugs, exactly as today.
2. Security reads the same change looking only for vulnerabilities, injection surfaces, and authorization gaps.
3. Performance hunts for hot-path regressions, N+1 patterns, and pathological allocations.
4. Architecture checks the change against the shape of the codebase: boundaries, coupling, drift.

Each agent makes its own pass with its own focus, and the passes run in parallel, so a four-agent review does not pay a serial latency tax over a one-agent review.

The arithmetic is the point. At the relative costs measured in this A/B, four DeepSeek V4 Flash passes still cost less than one Haiku 4.5 pass. Four specialist reviews for less compute than one generalist review is not a trade-off. It is strictly more coverage for less money.

In credit terms: a Core initial review stays at 1 credit, and the full four-agent fleet is 4 credits. For comparison, a single Tempest DeepSeek V4 Pro deep review is 6 credits and Tempest Sonnet is 15. The entire fleet costs less than one deep pass.

The takeaway

If you run an AI reviewer, or you are choosing one, the method transfers: run both models on the same diffs, blind the output, log cost and findings per pair, and publish the caveats next to the wins. It took one experiment flag and a vote widget, and it turned a vibes decision into a numbers decision.

We came out of it cheaper per review, denser per review, and with budget left over to put four reviewers on your PR where one used to stand.

Vortex reviews your first 50 PRs free: install MergeStorm and run your own comparison. Methodology note: experiment vortex-initial-ab-v1 randomly assigned blind paired runs of Claude Haiku 4.5 and DeepSeek V4 Flash on Vortex initial reviews (the A/B Core arm is DeepSeek V4 Flash, not Tempest DeepSeek V4 Pro). Aggregates were queried from production on 2026-08-13: 73 assignments, 66 delivered ballots, 72 variant pairs, 12 repositories, 5 users. Latency was not instrumented in this experiment and no latency numbers are claimed. The analysis and this writeup were generated by Fable (Claude Fable 5), an AI model, and reviewed by the team before publication. Small honest study, not a mega-benchmark.