EntelligenceAI: State-of-the-Art PR Reviews

I'm Aiswarya Sankar, founder of Entelligence.ai. We build AI-powered developer tools that help engineering teams ship better code faster. While building our AI code review system, we ran into a fundamental problem:

We set out to build the first OSS PR Review eval platform and are excited to announce that we hit 81% bug detection rate higher than others in the space.

The Problem

Every major AI company has their own code review bot now. The market is getting crowded, but there's no standardized way to compare them. Everyone claims to catch 3x more bugs (a meaningless metric), others about time saved (hard to verify).

We needed answers to basic questions:

• Are these bots catching real bugs or just style nits?
• How many of these issues matter to developers?
• How many critical issues do they actually find?

Our Approach: Build an Eval Framework

We decided to create the first open evaluation framework for code review bots. Here's how we did it:

• Set up multiple bots (including our own) on the same OSS codebases
• Collected 597 review comments across 5 different bots
• Built a classification system to categorize comments

The magic was in having multiple bots review identical PRs. This created a natural A/B test environment - we could see how different bots commented on exactly the same code changes, and more importantly, which comments developers actually addressed. We gathered data from over 200 PRs across these codebases, giving us a unique opportunity to compare bot effectiveness on a level playing field.

Challenges in Building Good Evals

Building effective evaluation systems for AI code review presents several significant challenges. One of the most fundamental issues is subjectivity - what constitutes a critical bug versus a simple nitpick can vary dramatically between different engineering teams and organizations. This variability makes it difficult to create standardized evaluation criteria that work across different contexts.

Core challenges we identified:

• Subjectivity: What's a critical bug vs a nitpick varies by team
• Context Matters: Code reviews don't make sense in isolation
• False Positives: Bots often identify "issues" that aren't actually problems
• Measurement Complexity: Simple metrics like "number of comments" aren't meaningful

To address these challenges, we implemented a systematic evaluation framework using large language models to ensure consistent classification across all reviews. This automated approach allowed us to apply identical criteria to every comment, eliminating the variability that often comes with human reviewers.

Evaluation Methodology

To ensure consistent classification of code review comments, we developed an LLM-based evaluation system focused on functional and security issues. Here's our approach:

{
    "comment_index": "<index>",
    "Comment": <comment>,
    "category": "CRITICAL_BUG|NITPICK|OTHER",
    "reasoning": "Brief explanation of why this category was chosen"
}

For each PR comment, we captured the comment text, related code, and developer response. This structured approach helped maintain consistency and focus on functional impact rather than style. We've open-sourced these evaluation criteria to help standardize industry practices.

Key learnings:

• Quantity ≠ Quality: More comments often meant more noise
• Context is Crucial: Bots need codebase-wide context to make meaningful suggestions
• Team Adaptation: Good review systems learn from team patterns
• False Positive Cost: Bad suggestions quickly lead to developers ignoring the bot

What We Found

Our analysis revealed significant differences in how effectively different code review tools identify critical bugs:

• EntelligenceAI leads with a 67.1% critical bug detection rate
• Other tools range from 8.1% to 29.8% in critical bug detection
• Most tools focus heavily on style issues rather than critical bugs

The data clearly shows that most code review tools are not optimized for finding critical issues that actually matter to developers.

Building a Better System

We went through multiple iterations to improve our own system:

The Results

Our current system achieves:

• 83.1% critical bug ratio (up from 40%)
• 1.1% style comments (down from 30%)

Open Sourcing Everything

We've open-sourced our entire evaluation framework at https://github.com/Entelligence-AI/code_review_evals including:

• Full dataset of 597 classified comments
• Evaluation methodology
• Classification system
• Team response tracking

We've open sourced this evaluation system because the industry and customers need a standard way to measure code review quality. We hope this helps teams make informed decisions about their tooling.