Training In-Context and In-Weights Mixtures Via Contrastive Context Sampling

Dimensions overall score 3.0

• The similarity structure between target inputs and context examples plays an important role in the emergence of in-context learning after IC-Train.result 90%
  Evidencepartial

  In this paper we show that the similarity structure between target inputs and context examples also plays an important role.

  Implicationpartial

  Directly stated in abstract as a key finding of the paper

  Verificationpartial

  partial

• Random context examples during fine-tuning lead to loss of in-context learning and dominance of in-weights learning.result 85%
  Evidencepartial

  Random context leads to loss of ICL and IWL dominance

  Implicationpartial

  Directly stated in abstract with clear causal relationship

  Verificationpartial

  partial

• Using only similar examples in context causes in-context learning to degenerate to copying labels without regard to relevance.limitation 85%
  Evidencepartial

  only similar examples in context causes ICL to degenerate to copying labels without regard to relevance

  Implicationpartial

  Directly stated in abstract with clear causal relationship

  Verificationpartial

  partial

• Contrastive-Context sampling enforces two types of contrasts: (1) mix of similar and random examples within a context, and (2) varying grades of similarity across contexts.method 90%
  Evidencepartial

  we propose a simple Contrastive-Context which enforces two types of contrasts: (1) mix of similar and random examples within a context to evolve a correct form of ICL, and (2) varying grades of similarity across contexts to evolve ICL-IWL mixtures.

  Implicationpartial

  Directly stated in abstract as the proposed method

  Verificationpartial

  partial

• Contrasted contexts yield stable ICL-IWL mixtures, avoiding collapse into pure ICL, IWL, or copying.result 85%
  Evidencepartial

  Diagnostic probes confirm that contrasted contexts yield stable ICL-IWL mixtures, avoiding collapse into pure ICL, IWL, or copying.

  Implicationpartial

  Directly stated in abstract as a key result of the method

  Verificationpartial

  partial

• Standard task-specific fine-tuning often erodes in-context learning ability in LLMs.limitation 80%
  Evidencepartial

  standard task-specific fine-tuning often erodes ICL

  Implicationpartial

  Directly stated in abstract as motivation for the work

  Verificationpartial

  partial

• The emergence of in-context learning after IC-Train depends on factors such as task diversity and training duration.technical 80%
  Evidencepartial

  Prior work has shown that emergence of ICL after IC-Train depends on factors such as task diversity and training duration.

  Implicationpartial

  Directly stated in abstract as established prior knowledge

  Verificationpartial

  partial

• The paper validates its findings with extensive empirical evaluation on four LLMs and several tasks.method 85%
  Evidencepartial

  We validate with extensive empirical evaluation on four LLMs and several tasks.

  Implicationpartial

  Directly stated in abstract as part of the methodology

  Verificationpartial

  partial

Competitive landscape

A novel training strategy for LLMs that balances in-context and in-weights learning by using contrastive context sampling to improve performance and prevent label copying.