Dimensions overall score 8.0

GitHub Code Pulse

No public code linked for this paper yet.

• Next-token prediction in language models favors hypotheses that enable shorter and more internally consistent descriptions of the training data.technical 95%
• Language models exhibit a 'truth bias' primarily when false alternatives are structurally harder to compress than correct statements.technical 95%
• Small GPT-2-style character-level transformers (3.5M--86M parameters) were used to test the Compression-Consistency Principle on synthetic math corpora.method 95%
• In a random-error setting with balanced data, models strongly preferred correct completions with 83.1% accuracy.result 95%
• Even when correct rules constituted only 10% of the corpus in a random-error setting, models still showed a strong preference for correct completions with 67.0% accuracy.result 95%
• Replacing random errors with a coherent but mathematically incorrect rule system largely eliminated the preference for correctness, resulting in near-chance accuracy.result 95%
• Embedding verification steps into the model can restore its preference for correctness, even at small scales.result 90%
• The observed 'truth bias' in language models is largely a consequence of compression pressure and a preference for internal consistency, not an intrinsic drive towards truth.technical 95%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card