Self-Improving Pretraining: using post-trained models to pretrain better models

Self-Improving Pretraining: using post-trained models to pretrain better models explores A pretraining method using reinforcement learning to enhance model safety and factuality by streaming documents and evaluating generated token sequences, improving quality from the ground up.. Commercial viability score: 5/10 in LLM Training.