Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering

Dimensions overall score 5.0

• StackRepoQA is the first multi-project, repository-level question answering dataset constructed from 1,318 real developer questions and accepted answers across 134 open-source Java projects.method 95%
  Evidencepartial

  In this work, we introduce StackRepoQA, the first multi-project, repository-level question answering dataset constructed from 1,318 real developer questions and accepted answers across 134 open-source Java projects.

  Implicationpartial

  This is explicitly stated in the abstract and introduction as a novel contribution.

  Verificationpartial

  partial

• LLMs achieve moderate accuracy (around 58%) on repository-level QA without augmentation.result 90%
  Evidencepartial

  Our findings indicate that while LLMs achieve moderate accuracy (around58%) on repository-level QA, much of this success can be attributed to memorization of previously seen Stack Overflow content rather than genuine reasoning over source code.

  Implicationpartial

  This is a direct result reported in the abstract and further elaborated in the introduction.

  Verificationpartial

  partial

• Retrieval-Augmented Generation (RAG) with graph-based retrieval provides the largest performance gains for repository-level question answering.method 90%
  Evidencepartial

  RAG provided measurable improvements, with graph-based retrieval yielding the largest gains; however, even the best configuration only increased accuracy to approximately64%.

  Implicationpartial

  The abstract and introduction highlight the effectiveness of graph-based RAG.

  Verificationpartial

  partial

• A significant portion of high LLM scores in repository-level QA results from verbatim reproduction of answers rather than genuine reasoning.result 95%
  Evidencepartial

  The analysis reveals that high scores often result from verbatim reproduction of Stack Overflow answers rather than genuine reasoning.

  Implicationpartial

  This is a key finding explicitly stated in the abstract and supported by the ablation study description.

  Verificationpartial

  partial

• LLMs exhibit performance degradation on unseen questions and sensitivity to irrelevant context in repository-level tasks.limitation 85%
  Evidencepartial

  We discuss key limitations of LLMs on repository-level tasks, such as performance degradation on unseen questions and sensitivity to irrelevant context

  Implicationpartial

  This limitation is mentioned in the abstract and further detailed in the 'Insights and Resources' section.

  Verificationpartial

  partial

• Flat text-based retrieval is insufficient for source-code RAG because it fails to capture inherent structural and semantic relationships.technical 90%
  Evidencepartial

  In source-code RAG setups, retrieval solely based on flat textual similarity is insufficient, because code inherently encodes structural and semantic relationships—such as function calls, class hierarchies, and data-flow dependencies—that extend beyond what text similarity can capture [59].

  Implicationpartial

  This is a technical justification for the proposed graph-based approach, stated in the introduction.

  Verificationpartial

  partial

• The best LLM configuration evaluated, incorporating graph-based retrieval, achieved an accuracy of approximately 64% on repository-level QA.result 90%
  Evidencepartial

  RAG provided measurable improvements, with graph-based retrieval yielding the largest gains; however, even the best configuration only increased accuracy to approximately64%.

  Implicationpartial

  This is a specific quantitative result reported in the abstract.

  Verificationpartial

  partial

• StackRepoQA is the first multi-project, repository-level question answering dataset constructed from 1,318 real developer questions and accepted answers across 134 open-source Java projects.method 95%
  Evidencepartial

  In this work, we introduce StackRepoQA, the first multi-project, repository-level question answering dataset constructed from 1,318 real developer questions and accepted answers across 134 open-source Java projects.

  Implicationpartial

  This is explicitly stated in the abstract and introduction, highlighting its novelty and scope.

  Verificationpartial

  partial

• LLMs achieve moderate accuracy (around 58%) on repository-level QA without augmentation.result 90%
  Evidencepartial

  Our findings indicate that while LLMs achieve moderate accuracy (around58%) on repository-level QA, much of this success can be attributed to memorization of previously seen Stack Overflow content rather than genuine reasoning over source code.

  Implicationpartial

  The abstract and analysis excerpt provide a specific percentage for baseline LLM accuracy.

  Verificationpartial

  partial

• Retrieval-Augmented Generation (RAG) methods, particularly graph-based retrieval, improve LLM performance on repository-level question answering.method 90%
  Evidencepartial

  RAG provided measurable improvements, with graph-based retrieval yielding the largest gains; however, even the best configuration only increased accuracy to approximately64%.

  Implicationpartial

  The abstract and analysis excerpt state that RAG provides improvements, with graph-based retrieval yielding the largest gains.

  Verificationpartial

  partial

• Even with the best RAG configuration (graph-based retrieval), LLM accuracy on repository-level QA only reaches approximately 64%.result 90%
  Evidencepartial

  RAG provided measurable improvements, with graph-based retrieval yielding the largest gains; however, even the best configuration only increased accuracy to approximately64%.

  Implicationpartial

  The analysis excerpt provides a specific upper bound for accuracy with augmentation.

  Verificationpartial

  partial

• A significant portion of high LLM scores on repository-level QA is due to verbatim reproduction of Stack Overflow answers, indicating memorization rather than genuine reasoning.result 95%
  Evidencepartial

  Our results show that LLMs achieve moderate accuracy at baseline, with performance improving when structural signals are incorporated. Nonetheless, overall accuracy remains limited for repository-scale comprehension. The analysis reveals that high scores often result from verbatim reproduction of Stack Overflow answers rather than genuine reasoning.

  Implicationpartial

  This is a key finding explicitly stated in the abstract and analysis.

  Verificationpartial

  partial

Competitive landscape

A new dataset and evaluation framework to benchmark LLMs on understanding entire code repositories, revealing limitations in current reasoning capabilities.