ExFusion: Efficient Transformer Training via Multi-Experts Fusion

Dimensions overall score 4.0

• ExFusion introduces multi-expert characteristics into Transformer training while incurring only marginal computational cost compared to standard dense training.method 90%
  Evidencepartial

  ExFusion introduces multi-expert characteristics into the training process while incurring only marginal computational cost compared to standard dense training.

  Implicationpartial

  Directly stated in the abstract with clear methodology description

  Verificationpartial

  partial

• ExFusion eliminates additional overhead in storage and deployment by integrating multiple experts into a single unified expert after training.technical 90%
  Evidencepartial

  After training, the learned weights are used to integrate multi-experts into a single unified expert, thereby eliminating additional overhead in storage and deployment.

  Implicationpartial

  Explicitly stated in the abstract with clear mechanism description

  Verificationpartial

  partial

• Traditional MoE models require substantially increased parameter counts and higher training costs compared to dense models.limitation 85%
  Evidencepartial

  For example, SwinV2-MoE-S (128 experts) [15], trained on a large-scale dataset (e.g., ImageNet-22k [16]), surpasses SwinV2-S by roughly 1% on the ImageNet-1k benchmark [17]. However, this improvement requires nearly a 30× increase in parameters

  Implicationpartial

  Directly stated with specific example comparing SwinV2-MoE-S to SwinV2-S

  Verificationpartial

  partial

• ExFusion upcycles the feed-forward network (FFN) of the Transformer into a multi-expert configuration during initialization.method 90%
  Evidencepartial

  Specifically, during the initialization phase, ExFusion upcycles the feed-forward network (FFN) of the Transformer into a multi-expert configuration

  Implicationpartial

  Directly stated in the abstract with clear technical description

  Verificationpartial

  partial

• ExFusion demonstrates effectiveness on a variety of computer vision and natural language processing tasks.result 85%
  Evidencepartial

  Extensive experiments on a variety of computer vision and natural language processing tasks demonstrate the effectiveness of the proposed method.

  Implicationpartial

  Directly stated in abstract with supporting experimental sections

  Verificationpartial

  partial

• During training, ExFusion fuses multiple experts into a single unified expert equivalent to the original FFN for forward computation.method 90%
  Evidencepartial

  During training, these weights allow multiple experts to be fused into a single unified expert equivalent to the original FFN, which is subsequently used for forward computation.

  Implicationpartial

  Directly stated in the abstract with clear operational mechanism

  Verificationpartial

  partial

• ExFusion addresses the critical challenge of leveraging MoE benefits while maintaining parameter count and computational complexity comparable to dense models.technical 80%
  Evidencepartial

  Could we harness the benefits of MoE during training yet retain a parameter count and computational complexity comparable to dense models, all without a significant rise in training expenses?

  Implicationpartial

  Directly stated in the methodology section as the core research question

  Verificationpartial

  partial

Competitive landscape

A novel pre-training approach that fuses multiple experts in Transformers to enhance performance with minimal additional cost during training and deployment.