Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory

Dimensions overall score 3.0

• Muon significantly exceeds the storage capacity of SGD on the logistic regression loss under a power law frequency distribution.result 95%
  Evidencepartial

  Muon storesd 1+ 1/2α items (Theorem 3), whereas GD stores onlyd 1/2α items (Theorem 5).

  Implicationpartial

  This is a central finding explicitly stated in the abstract and supported by Figure 2a and Theorem 3 vs Theorem 5.

  Verificationpartial

  partial

• Muon saturates at a larger critical batch size than SGD for the first step.result 90%
  Evidencepartial

  Muon saturates at a larger critical batch size.

  Implicationpartial

  Stated in the abstract and supported by Figure 1b and the text discussing critical batch sizes.

  Verificationpartial

  partial

• Muon achieves a substantially faster initial recovery rate than SGD under a thresholded gradient approximation for multi-step dynamics.result 90%
  Evidencepartial

  Muon achieves a substantially faster initial recovery rate than SGD

  Implicationpartial

  This is a key finding from the multi-step analysis, explicitly mentioned in the abstract and supported by Theorem 10.

  Verificationpartial

  partial

• The one-step Muon update can recover all items up to a certain threshold, which is dependent on the embedding dimension and frequency distribution exponent.result 95%
  Evidencepartial

  the one-step Muon update W1 ∝h λ(G0)recovers all items up to i≲min{n i⋆, B 1/α (logd) − 1/α }

  Implicationpartial

  This is a specific theoretical result with a precise bound provided in Theorem 3.

  Verificationpartial

  partial

• The spectral map update rule W←W+ηh(G) defines a class of spectral optimizers, where gradient descent corresponds to h(z)=z and exact Muon corresponds to h(z)=sign(z).method 95%
  Evidencepartial

  Within this scheme, gradient descent corresponds to h(z) =z, while exact Muon corresponds to h(z) = sign(z).

  Implicationpartial

  This defines the general framework for spectral optimizers and specifies the forms for GD and Muon.

  Verificationpartial

  partial

• The storage capacity of Muon scales as d^(1 + 1/(2α)) in the population limit, where d is the embedding dimension and α is the exponent of the power law frequency distribution.result 95%
  Evidencepartial

  Muon storesd 1+ 1/2α items (Theorem 3)

  Implicationpartial

  This is a precise theoretical result derived in the paper, stated in the abstract and supported by Theorem 3.

  Verificationpartial

  partial

• SGD exhibits a suboptimal scaling law for any learning rate schedule in multi-step recovery, with its recovery exponent increasing linearly for the first 2α steps.limitation 90%
  Evidencepartial

  for the first⌈2α⌉steps, the recovery exponent of SGD increases linearly u

  Implicationpartial

  This is a specific theoretical limitation identified for SGD in multi-step recovery, detailed in Theorem 10.

  Verificationpartial

  partial

Competitive landscape

This paper theoretically analyzes spectral optimizers for associative memory recall in language models, showing potential capacity advantages over SGD.