Solver-in-the-Loop: MDP-Based Benchmarks for Self-Correction and Behavioral Rationality in Operations Research

Dimensions overall score 9.0

GitHub Code Pulse

No public code linked for this paper yet.

• Existing LLM benchmarks evaluate Operations Research as one-shot translation from problem description to solver code, ignoring the diagnostic debugging loop entirely.limitation 95%
• The ORDebug benchmark evaluates iterative self-correction through 5,000+ problems spanning 9 error types, with each repair action triggering solver re-execution and IIS recomputation.method 98%
• The ORBias benchmark evaluates behavioral rationality through 2,000 newsvendor instances (1,000 in-distribution + 1,000 out-of-distribution), measuring systematic deviations from closed-form optimal policies.method 98%
• Domain-specific RLVR training enables an 8B model to surpass frontier APIs with a 95.3% vs 86.2% recovery rate (+9.1%), 62.4% vs 47.8% diagnostic accuracy (+14.6%), and 2.25 vs 3.78 steps to resolution (1.7× faster).result 96%
• On ORBias, curriculum training achieves the only negative ID→OOD bias drift among models evaluated (-9.6%), reducing systematic bias by 48% (from 20.0% to 10.4%).result 96%
• Process-level evaluation with verifiable oracles enables targeted training that outperforms scale.result 92%
• The benchmarks place the solver in the evaluation loop, providing deterministic, verifiable feedback.technical 94%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card