LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws

Dimensions overall score 3.0

• A theoretical framework models LLM training as information transmission over a noisy channel, explaining non-monotonic scaling phenomena. We propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon-Hartley theorem.problem 35%
  Evidencepartial

  A theoretical framework models LLM training as information transmission over a noisy channel, explaining non-monotonic scaling phenomena. We propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon-Hartley theorem.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

• Existing scaling laws for Large Language Models (LLMs), predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such as catastrophic overtraining and quantization-induced degradation, where performance deteriorates despite increased compute. We propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon-Hartley theorem.method 35%
  Evidencepartial

  Existing scaling laws for Large Language Models (LLMs), predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such as catastrophic overtraining and quantization-induced degradation, where performance deteriorates despite increased compute. We propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon-Hartley theorem.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

• ScienceToStartup currently rates this 3.0/10 on the public viability pass. It also extrapolates: fitted on $\leq$6.9B Pythia models with $\leq$180B tokens, it predicts the unseen 12B model up to 307B tokens at pooled $R^2{=}0.847$, while monotonic baselines collapse.result 35%
  Evidencepartial

  ScienceToStartup currently rates this 3.0/10 on the public viability pass. It also extrapolates: fitted on $\leq$6.9B Pythia models with $\leq$180B tokens, it predicts the unseen 12B model up to 307B tokens at pooled $R^2{=}0.847$, while monotonic baselines collapse.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

• LLM Training moved forward this cycle; last verified May 2026. Public score 3.0/10.why now 35%
  Evidencepartial

  LLM Training moved forward this cycle; last verified May 2026. Public score 3.0/10.

  Implicationpartial

  Abstract-backed fallback claim; anchored extraction has not materialized a public claim row yet.

  Verificationpartial

  partial

Competitive landscape

A theoretical framework models LLM training as information transmission over a noisy channel, explaining non-monotonic scaling phenomena.