SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback

Dimensions overall score 7.0

GitHub Code Pulse

No public code linked for this paper yet.

• SWE-PRBench is a benchmark comprising 350 pull requests with human-annotated ground truth for evaluating AI code review quality.method 100%
  Evidencepartial

  We introduce SWE-PRBench, a benchmark of 350 pull requests with human-annotated ground truth for evaluating AI code review quality.

  Implicationpartial

  Directly stated in the abstract and introduction.

  Verificationpartial

  partial

• Eight frontier AI models detect only 15-31% of human-flagged issues on the diff-only configuration.result 100%
  Evidencepartial

  8 frontier models detect only 15–31% of human-flagged issues on the diff-only configuration

  Implicationpartial

  Explicitly stated in the abstract with a numerical range.

  Verificationpartial

  partial

• AI code review performance is significantly below human expert performance.result 90%
  Evidencepartial

  demonstrating that AI code review remains far below human expert performance

  Implicationpartial

  Stated in the abstract as a conclusion drawn from the benchmark results.

  Verificationpartial

  partial

• All 8 evaluated AI models show a monotonic degradation in performance as the context provided increases from diff-only to full context.result 100%
  Evidencepartial

  All 8 models degrade monotonically from config_A to config_C, even when context is provided via structured semantic layers

  Implicationpartial

  Directly stated in the abstract and supported by the analysis of context configurations.

  Verificationpartial

  partial

• The dominant reason for performance degradation with increased context is a collapse in Type2_Contextual issue detection, consistent with attention dilution.result 90%
  Evidencepartial

  The dominant mechanism is a collapse of Type2_Contextual issue detection at config_B, consistent with attention dilution in long contexts

  Implicationpartial

  The abstract identifies this as the 'dominant mechanism' and provides a supporting explanation.

  Verificationpartial

  partial

• A structured 2,000-token diff-with-summary prompt outperforms a 2,500-token full-context prompt enriched with execution context, behavior mapping, and test signatures across all 8 models.result 100%
  Evidencepartial

  a structured 2,000-token diff-with-summary prompt outperforms a 2,500-token full-context prompt enriched with execution context, behaviour mapping, and test signatures across all 8 models.

  Implicationpartial

  Explicitly stated in the abstract as a key finding regarding prompt engineering.

  Verificationpartial

  partial

• The top four AI models evaluated are statistically indistinguishable in their performance, while a clear performance gap separates them from the remaining four models.result 90%
  Evidencepartial

  The top four models are statistically indistinguishable (mean score 0.147-0.153) while a clear tier gap separates them from the remaining four (mean score <= 0.113).

  Implicationpartial

  The abstract describes a 'tier gap' and provides mean scores for the top and bottom groups.

  Verificationpartial

  partial

• The Repository Quality Score (RQS) used for filtering repositories includes components like review culture, PR recency, test quality, PR volume, and contamination.method 90%
  Evidencepartial

  Repository Quality Score (RQS) components. Component What it measures Max pts Review culture Share of substantive human review comments 30 PR recency Merged PRs in last 90 days 25 Test quality Test files, CI presence, coverage tooling 20

  Implicationpartial

  The abstract mentions filtering using RQS, and the provided text details its components and weighting.

  Verificationpartial

  partial

• SWE-PRBench is a benchmark consisting of 350 pull requests with human-annotated ground truth for evaluating AI code review quality.method 100%
  Evidencepartial

  We introduce SWE-PRBench, a benchmark of 350 pull requests with human-annotated ground truth for evaluating AI code review quality.

  Implicationpartial

  Directly stated in the abstract and introduction.

  Verificationpartial

  partial

• Eight frontier AI models detect only 15-31% of human-flagged issues on the diff-only configuration.result 100%
  Evidencepartial

  8 frontier models detect only 15–31% of human-flagged issues on the diff-only configuration

  Implicationpartial

  Directly stated in the abstract and supported by baseline results.

  Verificationpartial

  partial

• All 8 evaluated AI models show a monotonic degradation in performance as the context provided increases from diff-only to full context.result 100%
  Evidencepartial

  All 8 models degrade monotonically from config_A to config_C, even when context is provided via structured semantic layers

  Implicationpartial

  Explicitly stated in the abstract and supported by the description of configurations.

  Verificationpartial

  partial

• The primary reason for performance degradation with increased context is a collapse in Type2_Contextual issue detection.result 90%
  Evidencepartial

  The dominant mechanism is a collapse of Type2_Contextual issue detection at config_B, consistent with attention dilution in long contexts

  Implicationpartial

  Stated as the dominant mechanism in the abstract, explaining the observed degradation.

  Verificationpartial

  partial

Startup potential card