Gemini Flash 2.0 vs OpenAI O1, O1-Mini, and O3-Mini PR Reviews

TL;DR

Gemini Flash 2.0

• Generated 187 review comments detecting 96 critical bugs (a 51.3% critical bug ratio)
• Found 2.67× more critical bugs than OpenAI O1, which flagged 36 critical issues from 56 comments
• Offers comprehensive review with 20 nitpicks (10.7%) and 71 additional feedback items (38.0%), adding valuable context and runtime insight

OpenAI O1-Mini & O3-Mini

• O1-Mini: 107 comments with 85 critical bugs (79.4% ratio), plus minimal nitpicks and other feedback
• O3-Mini: 69 comments with 55 critical bugs (79.7% ratio), demonstrating highly focused feedback
• High critical bug ratios indicate strong precision on core issues, though overall volume is lower

OpenAI O1

• Produced 56 comments with 36 critical bugs (64.3% ratio)
• While precise, output volume is modest compared to Gemini's extensive coverage

OpenAI's Track Record in PR Reviews

Over time, while developing Entelligence's PR review bot, we rigorously tested multiple models—including GPT-4, Claude Sonnet, and earlier versions of Gemini Flash. OpenAI O1 consistently emerged as a strong performer, reliably identifying edge cases and critical issues with minimal false positives. Even when models evaluated each other's reviews, O1's focus on critical bug detection was frequently validated.

However, with the advent of Gemini Flash 2.0, we sought to answer one key question: Can it not only match but extend the depth of OpenAI's models—especially when it comes to complex code reasoning and runtime behavior?

Evaluating Gemini Flash 2.0

Using our open-source PR review evaluation framework (see our detailed guide), we analyzed a dataset that produced a total of 419 review comments. Each bot was run on its own subset of pull requests, and the results were classified using our LLM-based evaluation system focused on both functional and security issues.

{
  "comment_index": "",
  "Comment": "",
  "category": "CRITICAL_BUG|NITPICK|OTHER",
  "reasoning": "Brief explanation for the chosen category"
}

For every PR comment, we captured the comment text, related code snippet, and developer response. This structured approach—now open-sourced—ensures industry-standard consistency and helps focus on feedback that truly impacts functionality.

Gemini Flash 2.0 vs. OpenAI Models

Gemini Flash 2.0

• Output: 187 review comments
• Critical Bugs: 96 (51.3% of comments)
• Nitpicks: 20 (10.7%)
• Other Feedback: 71 (38.0%)

Gemini Flash 2.0 delivered a high volume of actionable insights. Although its critical bug ratio is lower compared to the OpenAI Mini models, the raw numbers tell a compelling story: Gemini flagged 2.67× more critical bugs than OpenAI O1. Moreover, its extensive feedback—spanning both deep functional issues and additional contextual suggestions—demonstrates an enhanced understanding of code runtime behavior.

OpenAI Models

OpenAI O1:

• Output: 56 comments
• Critical Bugs: 36 (64.3%)
• While precise, the total number of critical detections is modest relative to Gemini Flash 2.0

OpenAI O1-Mini:

• Output: 107 comments
• Critical Bugs: 85 (79.4%)
• Highly focused on flagging core issues with minimal extra commentary

OpenAI O3-Mini:

• Output: 69 comments
• Critical Bugs: 55 (79.7%)
• Similar to O1-Mini in precision, delivering nearly 80% critical bug content

Key Takeaways

PR Examples

PR #2847 - feat: Implement new authentication flow

o3-mini & o1-mini Findings (Better at Bug Detection)

• ✅ o3-mini correctly flagged a security vulnerability where user tokens were stored in localStorage, recommending the use of httpOnly cookies for improved security
• ✅ o3-mini identified a missing event.preventDefault() in form submission, which could lead to unexpected page reloads
• ✅ o1-mini found a missing try-catch block in API calls, leading to potential unhandled errors
• ✅ o1-mini caught a missing dependency in a useEffect hook, which could cause stale closures

o1 Findings (Limited Detection)

• ⚠️ Flagged inconsistent CSS theming, recommending CSS variables for maintainability
• ⚠️ Identified missing ARIA attributes in menu items, improving accessibility but not directly related to functionality or security

Gemini Findings (Minimal Detection)

• ⚠️ Suggested lazy loading for performance improvement, which is a good optimization but not a critical bug or security issue
• ⚠️ Flagged outdated installation instructions, which is useful but not related to the core functionality or correctness of the PR

Conclusion

Our latest evaluation reveals a clear trade-off between focused precision and comprehensive coverage:

• OpenAI O1-Mini and O3-Mini deliver a high percentage of critical bug feedback, making them excellent for pinpointing the most severe issues
• OpenAI O1 offers a balanced but modest output
• Gemini Flash 2.0, while exhibiting a lower critical bug ratio (51.3%), produced a much larger volume of feedback—identifying 96 critical bugs versus O1's 36—thus catching 2.67× more critical issues overall. Its broader scope, enriched with additional context and runtime insights, makes it a powerful tool for comprehensive code reviews

Gemini Flash 2.0's ability to combine in-depth analysis with actionable, contextual feedback is a game-changer for PR reviews. Whether you need pinpoint precision or an expansive review, these insights can help you choose the right tool to improve code quality and reduce production bugs.