Solver-in-the-Loop: MDP-Based Benchmarks for Self-Correction and Behavioral Rationality in Operations Research

Dimensions overall score 9.0

• Existing LLM benchmarks evaluate Operations Research as one-shot translation from problem description to solver code, ignoring the diagnostic debugging loop entirely.limitation 95%
  Evidencepartial

  Yet existing LLM benchmarks evaluate OR as one-shot translation -- given a problem description, generate solver code -- ignoring this diagnostic loop entirely.

  Implicationmissing

  Implication not extracted yet.

  Verificationpartial

  partial

• The ORDebug benchmark evaluates iterative self-correction through 5,000+ problems spanning 9 error types, with each repair action triggering solver re-execution and IIS recomputation.method 98%
  Evidencepartial

  \textbf{\ORDebug{}} evaluates iterative self-correction through 5,000+ problems spanning 9 error types; each repair action triggers solver re-execution and \IIS{} recomputation, providing deterministic, verifiable feedback.

  Implicationmissing

  Implication not extracted yet.

  Verificationpartial

  partial

• The ORBias benchmark evaluates behavioral rationality through 2,000 newsvendor instances (1,000 in-distribution + 1,000 out-of-distribution), measuring systematic deviations from closed-form optimal policies.method 98%
  Evidencepartial

  \textbf{\ORBias{}} evaluates behavioral rationality through 2,000 newsvendor instances (1,000 ID + 1,000 OOD), measuring systematic deviations from closed-form optimal policies.

  Implicationmissing

  Implication not extracted yet.

  Verificationpartial

  partial

• Domain-specific RLVR training enables an 8B model to surpass frontier APIs with a 95.3% vs 86.2% recovery rate (+9.1%), 62.4% vs 47.8% diagnostic accuracy (+14.6%), and 2.25 vs 3.78 steps to resolution (1.7× faster).result 96%
  Evidencepartial

  Across 26 models and 12,000+ samples, we find that domain-specific RLVR training enables an 8B model to surpass frontier APIs: 95.3\% vs 86.2\% recovery rate (+9.1\%), 62.4\% vs 47.8\% diagnostic accuracy (+14.6\%), and 2.25 vs 3.78 steps to resolution (1.7$\times$ faster).

  Implicationmissing

  Implication not extracted yet.

  Verificationpartial

  partial

• On ORBias, curriculum training achieves the only negative ID→OOD bias drift among models evaluated (-9.6%), reducing systematic bias by 48% (from 20.0% to 10.4%).result 96%
  Evidencepartial

  On \ORBias{}, curriculum training achieves the only negative ID$\rightarrow$OOD bias drift among models evaluated (-9.6\%), reducing systematic bias by 48\% (from 20.0\% to 10.4\%).

  Implicationmissing

  Implication not extracted yet.

  Verificationpartial

  partial

• Process-level evaluation with verifiable oracles enables targeted training that outperforms scale.result 92%
  Evidencepartial

  These results demonstrate that process-level evaluation with verifiable oracles enables targeted training that outperforms scale.

  Implicationmissing

  Implication not extracted yet.

  Verificationpartial

  partial

• The benchmarks place the solver in the evaluation loop, providing deterministic, verifiable feedback.technical 94%
  Evidencepartial

  We introduce two benchmarks that place the \textbf{solver in the evaluation loop}.

  Implicationmissing

  Implication not extracted yet.

  Verificationpartial

  partial

• The 8B model with RLVR training achieves 62.4% diagnostic accuracy on ORDebug, compared to 47.8% for frontier APIs.result 95%
  Evidencepartial

  62.4% vs 47.8% diagnostic accuracy (+14.6%)

  Implicationpartial

  Directly stated in the abstract with explicit percentages.

  Verificationpartial

  partial

• The ORDebug benchmark evaluates iterative self-correction through 5,000+ problems spanning 9 error types.method 95%
  Evidencepartial

  We introduce two benchmarks that place the solver in the evaluation loop. ORDebug evaluates iterative self-correction through 5,000+ problems spanning 9 error types

  Implicationpartial

  Directly stated in the abstract with specific numbers.

  Verificationpartial

  partial

• Domain-specific RLVR training enables an 8B model to surpass frontier APIs with 95.3% vs 86.2% recovery rate on ORDebug.result 95%
  Evidencepartial

  domain-specific RLVR training enables an 8B model to surpass frontier APIs: 95.3% vs 86.2% recovery rate (+9.1%)

  Implicationpartial

  Directly stated in the abstract with explicit percentages.

  Verificationpartial

  partial

• On ORBias, curriculum training achieves the only negative ID→OOD bias drift among models evaluated (-9.6%), reducing systematic bias by 48%.result 95%
  Evidencepartial

  On ORBias, curriculum training achieves the only negative ID→OOD bias drift among models evaluated (-9.6%), reducing systematic bias by 48% (from 20.0% to 10.4%)

  Implicationpartial

  Directly stated in the abstract with specific numbers.

  Verificationpartial

  partial

• Existing LLM benchmarks evaluate OR as one-shot translation, ignoring the diagnostic loop of debugging infeasible models.limitation 90%
  Evidencepartial

  Yet existing LLM benchmarks evaluate OR as one-shot translation — given a problem description, generate solver code — ignoring this diagnostic loop entirely.

  Implicationpartial

  Directly stated in the abstract as a limitation of prior work.

  Verificationpartial

  partial

Competitive landscape

A new benchmark suite for iterative self-correction and bias reduction in operations research, outperforming existing methods in speed and accuracy.