Dimensions overall score 8.0

• Mixture-of-depths attention (MoDA) allows each attention head to attend to sequence KV pairs at the current layer and depth KV pairs from preceding layers.method 95%
• A hardware-efficient algorithm for MoDA achieves 97.3% of FlashAttention-2's efficiency at a sequence length of 64K.technical 90%
• Experiments on 1.5B-parameter models show MoDA improves average perplexity by 0.2 across 10 validation benchmarks.result 90%
• MoDA increases average performance by 2.11% on 10 downstream tasks with a negligible 3.7% FLOPs computational overhead.result 90%
• Combining MoDA with post-norm yields better performance than using it with pre-norm.result 85%
• MoDA consistently outperforms strong baselines in experiments on 1.5B-parameter models.result 85%
• Scalability of MoDA to larger models (e.g., 100B+ parameters) is unproven.limitation 80%
• MoDA addresses signal degradation in deeper LLM layers where informative features from shallow layers are diluted by repeated residual updates.technical 90%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card