Stabilizing Rubric Integration Training via Decoupled Advantage Normalization

Dimensions overall score 7.0

• Process-Aware Policy Optimization (PAPO) integrates process-level evaluation into Group Relative Policy Optimization (GRPO) using decoupled advantage normalization.method 95%
  Evidencepartial

  We propose Process-Aware Policy Optimization (PAPO), a method that integrates process-level evaluation into Group Relative Policy Optimization (GRPO) through decoupled advantage normalization

  Implicationpartial

  This is the core methodological contribution explicitly stated in the abstract and introduction.

  Verificationpartial

  partial

• PAPO resolves issues with Outcome Reward Models (ORM) that treat all correct responses identically and Process Reward Models (PRM) that can lead to reward hacking.method 95%
  Evidencepartial

  PAPO resolves both by composing the advantage from an outcome component Aout, derived from ORM and normalized over all responses, and a process component Aproc, derived from a rubric-based PRM and normalized exclusively among correct responses.

  Implicationpartial

  The abstract clearly outlines the problems PAPO aims to solve with existing reward models.

  Verificationpartial

  partial

• PAPO consistently outperforms ORM on OlympiadBench, achieving 51.3% accuracy compared to ORM's 46.3%.result 95%
  Evidencepartial

  Experiments across multiple model scales and six benchmarks demonstrate that PAPO consistently outperforms ORM, reaching 51.3% vs. 46.3% on OlympiadBench

  Implicationpartial

  This is a specific quantitative result presented in the abstract and supported by a figure.

  Verificationpartial

  partial

• PAPO continues to improve as ORM plateaus and declines.result 95%
  Evidencepartial

  while continuing to improve as ORM plateaus and declines.

  Implicationpartial

  This is a direct comparison of performance trends between PAPO and ORM, supported by figures.

  Verificationpartial

  partial

• GRPO with ORM evaluates only final-answer correctness and provides no signal on reasoning process quality.limitation 90%
  Evidencepartial

  However, GRPO with ORM evaluates only final-answer correctness, providing no signal on reasoning process quality, which leads to two critical issues.

  Implicationpartial

  This is a limitation of the baseline method explicitly stated in the introduction.

  Verificationpartial

  partial

• Naive integration of PRM scores into GRPO leads to training instability and reward hacking.limitation 90%
  Evidencepartial

  However, as we demonstrate in §3, naive integration of PRM scores into GRPO leads to training instability.

  Implicationpartial

  The abstract and analysis sections highlight the issues with direct PRM integration.

  Verificationpartial

  partial

• PAPO's decoupled advantage normalization ensures that the outcome component anchors training on correctness while the process component differentiates reasoning quality without distorting the outcome signal.technical 95%
  Evidencepartial

  This decoupled design ensures that Aout anchors training on correctness while Aproc differentiates reasoning quality without distorting the outcome signal.

  Implicationpartial

  This describes the technical mechanism and benefit of PAPO's design, as stated in the abstract.

  Verificationpartial

  partial

• Naive multiplicative combination of ORM and PRM (ORM×PRM) barely improves over ORM and fails to resolve signal exhaustion.result 90%
  Evidencepartial

  Naive multiplicative combination (ORM×PRM) barely improves over ORM, while PAPO’s decoupled normalization yields substantial gains on all benchmarks.

  Implicationpartial

  The analysis section and figures demonstrate the ineffectiveness of a simple multiplicative combination.

  Verificationpartial

  partial

Competitive landscape

A novel training method that improves LLM reasoning quality by decoupling outcome and process rewards, outperforming existing approaches on complex benchmarks.