Ranking Reasoning LLMs under Test-Time Scaling

Dimensions overall score 8.0

• Scorio is a library that implements statistical ranking methods for evaluating reasoning LLMs under test-time scaling.method 95%
  Evidencepartial

  We formalize dense benchmark ranking under test-time scaling and introduce Scorio, a library that implements statistical ranking methods such as paired-comparison models, item response theory (IRT) models, voting rules, and graph- and spectral-based methods.

  Implicationpartial

  Directly stated in abstract with specific method names and purpose.

  Verificationpartial

  partial

• Most full-trial rankings (with up to N=80 trials) agree closely with the Bayesian gold standard, with mean Kendall's τ_b = 0.93–0.95.result 93%
  Evidencepartial

  Across $20$ reasoning models on four Olympiad-style math benchmarks (AIME'24, AIME'25, HMMT'25, and BrUMO'25; up to $N=80$ trials), most full-trial rankings agree closely with the Bayesian gold standard $\mathrm{Bayes}_{\mathcal{U}}@80$ (mean Kendall's $τ_b = 0.93$--$0.95$)

  Implicationpartial

  Direct numeric evidence provided in abstract with specific statistical measure.

  Verificationpartial

  partial

• In the single-trial regime, the best ranking methods reach τ_b ≈ 0.86.result 92%
  Evidencepartial

  In the single-trial regime, the best methods reach $τ_b \approx 0.86$.

  Implicationpartial

  Direct numeric evidence provided in abstract.

  Verificationpartial

  partial

• Using greedy decoding as an empirical prior reduces variance at N=1 by 16–52%.result 91%
  Evidencepartial

  Using greedy decoding as an empirical prior ($\mathrm{Bayes}_{\mathbf{R}_0}@N$) reduces variance at $N=1$ by $16$--$52\%$

  Implicationpartial

  Direct numeric evidence with specific percentage range provided.

  Verificationpartial

  partial

• Using greedy decoding as an empirical prior can bias rankings when greedy and stochastic sampling disagree.limitation 88%
  Evidencepartial

  but can bias rankings when greedy and stochastic sampling disagree.

  Implicationpartial

  Directly stated limitation with clear causal relationship.

  Verificationpartial

  partial

• 19–34 methods recover exactly the same ordering as the Bayesian gold standard in full-trial rankings.result 90%
  Evidencepartial

  and $19$--$34$ methods recover exactly the same ordering.

  Implicationpartial

  Direct numeric evidence with specific range provided.

  Verificationpartial

  partial

• The paper identifies reliable ranking methods for both high- and low-budget test-time scaling.result 85%
  Evidencepartial

  These results identify reliable ranking methods for both high- and low-budget test-time scaling.

  Implicationpartial

  Direct statement of contribution in abstract, though 'reliable' requires some interpretation.

  Verificationpartial

  partial

• Scorio is released as an open-source library at a specific GitHub repository.technical 95%
  Evidencepartial

  We release Scorio as an open-source library at https://github.com/mohsenhariri/scorio.

  Implicationpartial

  Direct statement with specific URL provided.

  Verificationpartial

  partial

Competitive landscape

Scorio is an open-source library for ranking reasoning LLMs under test-time scaling using advanced statistical methods.