Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training

Dimensions overall score 7.0

• I-PPO consistently outperforms SFT and PPO baselines in experiments.result 85%
  Evidencepartial

  Our experiments demonstrate that I-PPO consistently outperforms SFT and PPO baselines.

  Implicationpartial

  Directly stated as an experimental result, though specific metrics are not provided in the given text.

  Verificationpartial

  partial

• The filtering process in I-PPO acts as an intrinsic early stopping mechanism, accelerating training efficiency.result 80%
  Evidencepartial

  We show that our filtering process acts as an intrinsic early stopping mechanism, accelerating training efficiency

  Implicationpartial

  Directly stated as a demonstrated benefit, though the specific acceleration factor is not quantified in the given text.

  Verificationpartial

  partial

• I-PPO effectively reduces unfaithful Chain-of-Thought (CoT) reasoning.result 80%
  Evidencepartial

  while effectively reducing unfaithful CoT reasoning.

  Implicationpartial

  Directly stated as a key result of the method's application.

  Verificationpartial

  partial

• Traditional PPO algorithms train on the entire rollout buffer, assuming all generated episodes provide beneficial optimization signals.technical 90%
  Evidencepartial

  Traditional RL algorithms like Proximal Policy Optimization (PPO) typically train on the entire rollout buffer, operating under the assumption that all generated episodes provide a beneficial optimization signal.

  Implicationpartial

  Directly and explicitly stated in the abstract as foundational premise.

  Verificationpartial

  partial

• Episodes generated during PPO training frequently contain noisy or unfaithful reasoning, which can degrade model performance and slow down training.limitation 85%
  Evidencepartial

  However, these episodes frequently contain noisy or unfaithful reasoning, which can degrade model performance and slow down training.

  Implicationpartial

  Directly stated in the abstract as a key problem motivating the research.

  Verificationpartial

  partial

• The proposed Influence-Guided PPO (I-PPO) framework integrates data attribution into the RL post-training loop.method 95%
  Evidencepartial

  In this paper, we propose \textbf{Influence-Guided PPO (I-PPO)}, a novel framework that integrates data attribution into the RL post-training loop.

  Implicationpartial

  Explicitly and precisely stated as the core contribution of the paper.

  Verificationpartial

  partial

• I-PPO calculates an influence score for each episode using a gradient-based approximation to identify episodes anti-aligned with a validation gradient.method 90%
  Evidencepartial

  By calculating an influence score for each episode using a gradient-based approximation, I-PPO identifies and eliminates episodes that are anti-aligned with a validation gradient.

  Implicationpartial

  Directly describes the core technical mechanism of the proposed method.

  Verificationpartial

  partial

Competitive landscape

Accelerate LLM training and reduce unfaithful reasoning by intelligently filtering training data using influence scores.