AI PR Review Evals

I'm Aiswarya Sankar, founder of Entelligence.ai. We build AI-powered developer tools that help engineering teams ship better code faster. While building our AI code review system, we ran into a fundamental problem:

The Problem

Every major AI company has their own code review bot now. The market is getting crowded, but there's no standardized way to compare them. Everyone claims to catch 3x more bugs (a meaningless metric), others about time saved (hard to verify).

We needed answers to basic questions:

• Are these bots catching real bugs or just style nits?
• How many of these issues matter to developers?
• How many critical issues do they actually find?

Our Approach: Build an Eval Framework

We decided to create the first open evaluation framework for code review bots. Here's how we did it:

• Set up multiple bots (including our own) on the same OSS codebases
• Collected 597 review comments across 5 different bots
• Built a classification system to categorize comments

The magic was in having multiple bots review identical PRs. This created a natural A/B test environment - we could see how different bots commented on exactly the same code changes, and more importantly, which comments developers actually addressed. We gathered data from over 200 PRs across these codebases, giving us a unique opportunity to compare bot effectiveness on a level playing field.

Challenges in Building Good Evals

Building effective evaluation systems for AI code review presents several significant challenges. One of the most fundamental issues is subjectivity - what constitutes a critical bug versus a simple nitpick can vary dramatically between different engineering teams and organizations. This variability makes it difficult to create standardized evaluation criteria that work across different contexts.

Core challenges we identified:

• Subjectivity: What's a critical bug vs a nitpick varies by team
• Context Matters: Code reviews don't make sense in isolation
• False Positives: Bots often identify "issues" that aren't actually problems
• Measurement Complexity: Simple metrics like "number of comments" aren't meaningful

To address these challenges, we implemented a systematic evaluation framework using large language models to ensure consistent classification across all reviews. This automated approach allowed us to apply identical criteria to every comment, eliminating the variability that often comes with human reviewers.

Evaluation Methodology

To ensure consistent classification of code review comments, we developed an LLM-based evaluation system focused on functional and security issues. Here's our approach:

                                {
                                    "comment_index": "",
                                    "Comment": ,
                                    "category": "CRITICAL_BUG|NITPICK|OTHER",
                                    "reasoning": "Brief explanation of why this category was chosen"
                                }

For each PR comment, we captured the comment text, related code, and developer response. This structured approach helped maintain consistency and focus on functional impact rather than style. We've open-sourced these evaluation criteria to help standardize industry practices.

Key learnings:

• Quantity ≠ Quality: More comments often meant more noise
• Context is Crucial: Bots need codebase-wide context to make meaningful suggestions
• Team Adaptation: Good review systems learn from team patterns
• False Positive Cost: Bad suggestions quickly lead to developers ignoring the bot

What We Found

Most code review bots have a major noise problem:

• 45% of comments (on average) were pure style nits
• Some "leading" tools had up to 70% style comments
• Only 20-40% of comments identified critical bugs