Ranking Reasoning LLMs under Test-Time Scaling

Dimensions overall score 8.0

GitHub Code Pulse

No public code linked for this paper yet.

• Scorio is a library that implements statistical ranking methods for evaluating reasoning LLMs under test-time scaling.method 95%
• Most full-trial rankings (with up to N=80 trials) agree closely with the Bayesian gold standard, with mean Kendall's τ_b = 0.93–0.95.result 93%
• In the single-trial regime, the best ranking methods reach τ_b ≈ 0.86.result 92%
• Using greedy decoding as an empirical prior reduces variance at N=1 by 16–52%.result 91%
• Using greedy decoding as an empirical prior can bias rankings when greedy and stochastic sampling disagree.limitation 88%
• 19–34 methods recover exactly the same ordering as the Bayesian gold standard in full-trial rankings.result 90%
• The paper identifies reliable ranking methods for both high- and low-budget test-time scaling.result 85%
• Scorio is released as an open-source library at a specific GitHub repository.technical 95%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card