One-for-All Model Initialization with Frequency-Domain Knowledge

Dimensions overall score 8.0

• A model's foundational, task-agnostic knowledge (its 'learngene') is encoded within the low-frequency components of its weights.technical 85%
  Evidencepartial

  we empirically demonstrate that a model's foundational, task-agnostic knowledge, its 'learngene', is encoded within the low-frequency components of its weights

  Implicationpartial

  Directly stated in abstract as empirical finding, though specific evidence details not provided in given text

  Verificationpartial

  partial

• FRONT framework achieves state-of-the-art performance in knowledge transfer tasks.result 90%
  Evidencepartial

  Extensive experiments demonstrate that FRONT achieves the state-of-the-art performance

  Implicationpartial

  Explicitly stated in abstract with mention of extensive experiments

  Verificationpartial

  partial

• FRONT accelerates convergence by up to 15 times in vision tasks.result 90%
  Evidencepartial

  accelerates convergence by up to 15 times in vision tasks

  Implicationpartial

  Clear numeric claim directly stated in abstract

  Verificationpartial

  partial

• FRONT reduces training FLOPs by an average of 40.5% in language tasks.result 90%
  Evidencepartial

  reduces training FLOPs by an average of 40.5% in language tasks

  Implicationpartial

  Clear numeric claim directly stated in abstract

  Verificationpartial

  partial

• FRONT uses Discrete Cosine Transform (DCT) to isolate low-frequency 'learngene' components.method 95%
  Evidencepartial

  we propose FRONT (FRequency dOmain kNowledge Transfer), a novel framework that uses the Discrete Cosine Transform (DCT) to isolate the low-frequency 'learngene'

  Implicationpartial

  Directly and explicitly stated in abstract as core method

  Verificationpartial

  partial

• The learngene can be adapted to initialize models of arbitrary size via simple truncation or padding without training.method 90%
  Evidencepartial

  This learngene can be seamlessly adapted to initialize models of arbitrary size via simple truncation or padding, a process that is entirely training-free

  Implicationpartial

  Directly stated in abstract with clear description of process

  Verificationpartial

  partial

• Existing parameter selection approaches fail to capture the interdependent structure of pre-trained model knowledge.limitation 85%
  Evidencepartial

  recent approaches typically resort to either parameter selection, which fails to capture the interdependent structure of this knowledge

  Implicationpartial

  Directly stated as limitation of previous approaches in abstract

  Verificationpartial

  partial

• Parameter prediction using generative models depends on impractical access to large network collections.limitation 85%
  Evidencepartial

  or parameter prediction using generative models that depend on impractical access to large network collections

  Implicationpartial

  Directly stated as limitation of alternative approaches in abstract

  Verificationpartial

  partial

Competitive landscape

FRONT enables efficient transfer learning by extracting and transferring task-agnostic knowledge from pre-trained models via frequency domain analysis, allowing for faster convergence and reduced training costs.