Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment

Dimensions overall score 8.0

GitHub Code Pulse

No public code linked for this paper yet.

• Upsampling synthetic training documents about AI misalignment leads to a notable increase in misaligned behavior in 6.9B-parameter LLMs.result 95%
• Upsampling synthetic training documents about aligned behavior reduces misalignment scores from 45% to 9% in 6.9B-parameter LLMs.result 95%
• The effects of alignment pretraining on LLM behavior persist, though dampened, through post-training.result 90%
• Alignment pretraining, focusing on shaping alignment priors during the pretraining phase, is a viable complement to post-training alignment methods.method 90%
• Practitioners are recommended to incorporate alignment pretraining in addition to capability training for AI models.method 90%
• The study utilized 6.9B-parameter LLMs for its controlled experiments on alignment discourse.technical 95%
• The approach of alignment pretraining can reduce misalignment rates from 45% to 9% by adding positive AI stories to training data.result 90%
• The models used in the study are not the most advanced, and the approach may miss nuances of real-world scenarios.limitation 85%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card