Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Models

Dimensions overall score 7.0

• Expert-choice (EC) routing provides deterministic load balancing by design for diffusion language models, yielding higher throughput and faster convergence than token-choice (TC) routing.method 90%
• Allocating more expert capacity to low-mask-ratio steps consistently achieves the best performance under matched FLOPs in diffusion language models.result 90%
• Tokens in low-mask-ratio contexts exhibit an order-of-magnitude higher learning efficiency compared to other contexts in diffusion language models.technical 85%
• Existing pretrained token-choice diffusion language models can be retrofitted to expert-choice routing by replacing only the router.method 90%
• Retrofitting TC DLMs to EC routing achieves faster convergence and improved accuracy across diverse downstream tasks.result 85%
• Concentrating compute on low-mask-ratio steps yields the largest marginal return in diffusion language models.technical 80%
• Computation in diffusion language models can be treated as an adaptive policy rather than a fixed architectural constant.technical 85%
• Existing DLM mixture-of-experts models inherit token-choice routing from autoregressive systems, leading to load imbalance and rigid computation allocation.limitation 80%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card