A new frontier model ships. It benchmarks higher than the model below it. The obvious move is to point your code reviewer at it and accept the bill.

GPT-6 Astra costs $10 per million input tokens and $50 per million output. GPT-5.6 Sol costs $4 and $20. Astra is 2.5 times the price. The question nobody answers with data is whether it finds 2.5 times the bugs.

We tested it. Astra did not find more bugs than Sol. It found fewer, and it cost more per bug it did find.

That result comes with a caveat we will not bury: Astra was the more accurate reviewer by a wide margin. Almost everything it said was correct. It just said less.

The short answer

Across 50 real pull requests, GPT-5.6 Sol produced 107 confirmed bugs to GPT-6 Astra's 91, at $0.039 per bug against $0.062. Astra was far more precise: 95% of its findings survived verification versus 85% of Sol's. Neither model is a superset of the other.

Why most code review benchmarks are not worth much

The standard way to report a model's review quality is to count what it flagged. That number is close to meaningless on its own.

A reviewer that comments on everything will "find" more bugs than a careful one, and it will also bury the real findings under noise the team learns to scroll past. A 2026 study of more than 31,000 CodeRabbit comment-and-feedback pairs found that 56.3% were rejected by developers, against 36.4% accepted. Read the study.

So a review benchmark has to answer two questions, not one:

1. How many real bugs did the reviewer find?

2. How much wrong or irrelevant commentary came attached?

CodeRabbit's own GPT-6 Astra evaluation reports a single metric, actionable bug coverage, at 61.3% for Astra against 59.0% for Sol. It does not report how many pull requests were tested, or the false-positive rate that came with that coverage. Those are the numbers that decide whether a team keeps the reviewer switched on.

We built this evaluation to report both.

What we measured

We used the public benchmark pull requests from the AI-Code-Review-Evals organization: 50 pull requests drawn from five large open-source codebases, each one a real change with real defects introduced against a clean base branch.

Both models received the same prompt on the same diffs: report correctness errors, security vulnerabilities, races, leaks, data loss, and broken error handling, and explicitly do not report style, naming, documentation, or test suggestions. Both returned structured findings. One pass each, single seed.

How a finding becomes a confirmed bug

Counting a model's own claims would just measure confidence. So every finding went through verification.

1. Pool. For each pull request, all findings from every reviewer were collected into one anonymized list, labeled R1, R2 and so on, with no tool names attached.

2. Deduplicate. A judge model grouped findings describing the same underlying defect into a single issue, so two reviewers describing one bug count once.

3. Verify. The judge decided, from the diff alone, whether each issue was a real defect a maintainer would need to fix, or noise.

4. Require agreement. The entire pool was judged twice, by two different models. A bug counts as confirmed only when both judges independently call it real.

The two judges agreed on 91% of findings. Of 196 distinct issues in the pool, 132 cleared both judges.

Result: Sol raised more, and more of it was real

Read the precision column before the confirmed column, because it explains the shape of the result.

Astra is the more trustworthy reviewer per comment. At 95%, almost everything it raised held up under verification. If a team's main complaint about AI review is noise, that number matters more than any coverage figure.

Sol found more real defects anyway. It raised more, and although a larger share washed out, the surviving count was still higher.

Why the volume gap is a threshold, not a reasoning gap

The obvious explanation for Sol's extra findings is that a weaker model is less disciplined. The severity mix says something more specific.

On high and medium severity the two models are close. The entire volume gap is concentrated in low-severity findings, where Sol reported 4.9 times as many.

Astra is not reasoning better about serious defects here. It is applying a higher bar for what deserves a comment at all. That is a real product difference, and which one you want depends on whether your reviewers are drowning or your bugs are escaping.

Result: the cheaper model was cheaper per bug, twice over

Astra's 2.5x token price does not translate into 2.5x the bugs. It translates into fewer bugs at a higher price, on this workload.

There is a real trade the other way. Astra was faster, at 36 seconds per review against 51, because it wrote far less. For a reviewer that blocks a pull request, latency is a feature, and a team that values a terse, high-confidence comment stream may accept fewer findings to get it.

Result: neither model is a superset of the other

This is the finding with the most practical consequence. 32 confirmed bugs were reported only by Sol. 16 were reported only by Astra. Same diffs, same prompt.

Two of the bugs only Sol caught were security defects, and both are worth looking at because they show what "cross-file reasoning" actually costs a reviewer that is not looking hard enough.

A substring origin check in Discourse's embed script. The change validates the sender of a postMessage like this:

if (discourseUrl.indexOf(e.origin) === -1) { return; }

if (discourseUrl.indexOf(e.origin) === -1) { return; }

That asks whether the sender's origin appears anywhere inside the forum's URL. An attacker who registers a domain that is a prefix of the real one passes the check. Astra read the same lines and did not flag it.

A redirect that escapes a host allowlist in Discourse's topic embedding. The retriever validates the host of the URL it was given, then fetches it with a library that follows redirects without revalidating the destination. The allowlist is enforced once, at the front door, and the redirect walks around it. Catching this requires connecting a validation in one file to a fetch in another.

The lesson is not that Sol is the better security reviewer. It is that two strong models given identical inputs disagree substantially about what is worth reporting, which is a poor foundation for picking a reviewer on one leaderboard number.

What this benchmark does not prove

We would rather state the limits than have a reader find them.

1. Single seed, one pass, and no variance estimate. We did not run repeated seeds, so we cannot say how much of the 107 to 91 gap in confirmed bugs would survive a rerun. Treat the count difference as directional. The cost-per-bug gap is the sturdier result, because token price does not vary between runs.

2. The judges are also contestants. One judge was more lenient across the board, and its leniency was largest on its own output. Sol-as-judge accepted Sol's findings at 96% where Astra-as-judge accepted the same findings at 87%, a gap of 9 points. On Astra's findings the two judges differed by only 3. Requiring both to agree removes the findings this disagreement covers, but a judge with no entry in the race would be better.

3. These are seeded defects, not organic ones. The benchmark's bugs were introduced deliberately. Real regressions are often subtler and less well contained.

4. Diff-only review. Both models saw the diff and nothing else. No repository history, no dependency graph, no production signal.

5. One workload. Five codebases and 50 pull requests is a real sample, not a universal ranking. Your languages and review conventions will move these numbers.

We also ran the shipping Entelligence reviewer's public comments through the same verification, but we are not scoring them against the two models. Those comments were produced by the shipping product under its own configuration and its own scope, not by this evaluation's bug-only prompt. Comparing them directly would measure the difference in instructions, not the difference in reviewers.

Where picking a model stops helping

Both models in this test read a diff in isolation. Every defect they could possibly have found was already inside the change.

Real review is not like that. The expensive failures come from changes that are locally correct and wrong in context:

• A query that is fine until it runs against the production table size.

• A retry that is correct until it meets a caller that already retries.

• A migration that passes review and locks a table under real write volume.

• A config change that is safe in every environment except the one with the legacy flag.

No amount of model upgrade fixes a context problem. A reviewer that has never seen the repository's history, the service's dependency graph, or what actually broke in production last quarter is guessing at the part that matters most.

This is the gap Entelligence code review is built around. It reviews a pull request with full-repo awareness, maps the change to the system architecture it touches, and feeds production behavior back into later reviews, so the reviewer learns from the alerts and regressions a diff-only model never sees.

Model choice is one variable in review quality. Context is the other one, and it is the one you cannot buy by upgrading a model.

How to run this evaluation on your own code

The method matters more than our numbers. To get a defensible answer for your codebase:

1. Collect 30 to 50 merged pull requests that contained real defects, ideally ones that caused an incident or a follow-up fix.

2. Give every model the same prompt, and be explicit about what is out of scope. Most of the noise gap between models is a threshold difference you control.

3. Never score a model on its own claims. Verify each finding against the diff, and use a second verifier that did not produce the findings.

4. Report precision next to coverage. A model that finds more and is wrong more may be a worse tool for your team.

5. Divide cost by confirmed bugs, not by tokens. Token price tells you almost nothing about what a review costs.

6. Check the disagreement. If two candidates find substantially different bugs, the interesting question is what your current setup misses that both of them caught.

Frequently asked questions

Is GPT-6 Astra better than GPT-5.6 Sol for code review?

Not on this benchmark, if your metric is bugs found per dollar. Sol produced 107 confirmed bugs to Astra's 91 at lower cost. Astra was clearly better on precision at 95% against 85%, and it was faster. Choose Astra if reviewer trust and latency matter more than coverage.

Does Astra's cross-file reasoning advantage show up in review?

We did not reproduce it. The two security defects in this evaluation that required connecting code across files were both caught by Sol and missed by Astra. That is a small sample and not a refutation, but it is evidence against treating the claim as settled.

How much does an AI code review actually cost per pull request?

On this workload, $0.083 per pull request with Sol and $0.113 with Astra, for a single review pass on diffs averaging a few hundred lines. Cost scales with diff size, so a repository with large changes will pay considerably more.

Should AI review comments block a merge?

No. On this benchmark the best precision was 95%, and the model with the most confirmed findings ran at 85%, so a meaningful share of its comments were still wrong. Treat AI review as a first pass that a human still signs off.

Can I reproduce these results?

Yes. The pull requests are public in the AI-Code-Review-Evals organization, and the review prompt, per-pull-request outputs, judge verdicts, and scoring script are committed alongside this article.

The takeaway

The reflex when a new frontier model ships is to move the expensive workload onto it. On code review, that reflex cost us money and findings at the same time.

The number that should drive the decision is not the price per million tokens or the coverage percentage in a launch post. It is the cost of one bug a maintainer actually fixes, measured on your code, with the findings verified by something that did not produce them.

Run the test before you move the budget.

See how Entelligence reviews a pull request with full repository context.

Methodology: 50 pull requests from the public AI-Code-Review-Evals benchmark repositories, one review pass per model, single seed, September 2026. Findings pooled per pull request, deduplicated, and independently verified by two judge models; a bug is counted only where both judges agreed. Prices are the published rates of $10/$50 per million tokens for GPT-6 Astra and $4/$20 for GPT-5.6 Sol. Prompts, raw outputs, judge verdicts, and the scoring script are in the article's bench/ directory.