BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs

Dimensions overall score 7.0

GitHub Code Pulse

No public code linked for this paper yet.

• Transforming causal generative language models into bidirectional encoders offers a powerful alternative to BERT-style architectures.technical 95%
• Current approaches for transforming LLMs into bidirectional encoders lack consensus on optimal training objectives, suffer from catastrophic forgetting at scale, and fail to flexibly integrate specialized generative models.limitation 90%
• Systematic ablations on the Gemma3 and Qwen3 families identified key factors for successful adaptation, highlighting the critical role of an often-omitted prior masking phase.method 90%
• A dual strategy combining linear weight merging with a lightweight multi-domain data mixture mitigates catastrophic forgetting when scaling the adaptation process without original pre-training data.method 85%
• The proposed method can augment encoders by merging them with specialized causal models, seamlessly transferring modality- and domain-specific capabilities.method 85%
• The proposed adaptation recipe is open-source and designed to work with any causal decoder LLM.technical 90%
• The resulting BidirLM family of five encoders outperforms alternatives on text, vision, and audio representation benchmarks.result 80%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card