MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration

Dimensions overall score 4.0

• MuonEq introduces a lightweight family of pre-orthogonalization equilibration schemes for Muon optimizers, available in three forms: two-sided row/column normalization (RC), row normalization (R), and column normalization (C).method 95%
  Evidencepartial

  We introduce {\method}, a lightweight family of pre-orthogonalization equilibration schemes for Muon in three forms: two-sided row/column normalization (RC), row normalization (R), and column normalization (C).

  Implicationpartial

  Explicitly stated in the abstract and repeated in the analysis section as the core contribution.

  Verificationpartial

  partial

• The row-normalized variant (R) of MuonEq is the natural default for hidden matrix weights and preserves the O~(T^{-1/4}) stationarity guarantee of Muon-type methods.technical 90%
  Evidencepartial

  For the hidden matrix weights targeted by {\method}, the row-normalized variant R is the natural default and preserves the $\widetilde{\mathcal{O}}(T^{-1/4})$ stationarity guarantee of Muon-type methods.

  Implicationpartial

  Directly stated in the abstract as a theoretical property of the proposed method.

  Verificationpartial

  partial

• In LLaMA2 pretraining on C4, the default R variant of MuonEq consistently outperforms Muon on 130M and 350M models, yielding faster convergence and lower validation perplexity.result 90%
  Evidencepartial

  In LLaMA2 pretraining on C4, the default R variant consistently outperforms Muon on 130M and 350M models, yielding faster convergence and lower validation perplexity.

  Implicationpartial

  Explicitly stated in the abstract and supported by experimental results in Figures 3 and 4.

  Verificationpartial

  partial

• Two-sided row/column normalization (RC) provides the fastest decay of orthogonalization error across Newton-Schulz steps compared to single-sided variants.result 85%
  Evidencepartial

  Two-sided row/column normalization decays fastest.

  Implicationpartial

  Strongly supported by the caption and data trend in Figure 2, though the exact claim wording is inferred.

  Verificationpartial

  partial

• MuonEq variants only require O(m+n) auxiliary state, making them lightweight compared to heavier whitening-based preconditioners.technical 90%
  Evidencepartial

  These variants rebalance the momentum matrix before finite-step Newton--Schulz using row/column squared-norm statistics and only $\mathcal{O}(m+n)$ auxiliary state.

  Implicationpartial

  Directly stated in the abstract as a key characteristic of the method.

  Verificationpartial

  partial

• In ablation studies on CIFAR-10/ResNet-18 and FineWeb/GPT2-small, the R variant performs best, reaching 94.53% test accuracy and 24.94 validation perplexity, respectively.result 90%
  Evidencepartial

  R performs best on both CIFAR-10/ResNet-18 and FineWeb/GPT2-small, reaching 94.53% test accuracy on CIFAR-10 and 24.94 validation perplexity on FineWeb/GPT2-small.

  Implicationpartial

  Explicitly stated in the analysis section with specific numeric results.

  Verificationpartial

  partial

• Finite-step orthogonalization is governed by input spectral properties, especially stable rank and condition number.technical 85%
  Evidencepartial

  We show that finite-step orthogonalization is governed by input spectral properties, especially stable rank and condition number

  Implicationpartial

  Directly stated in the abstract as a finding from the analysis.

  Verificationpartial

  partial

• Row/column normalization acts as a zeroth-order whitening surrogate that removes marginal scale mismatch.technical 85%
  Evidencepartial

  and that row/column normalization is a zeroth-order whitening surrogate that removes marginal scale mismatch.

  Implicationpartial

  Directly stated in the abstract as a theoretical interpretation of the method's mechanism.

  Verificationpartial

  partial

Competitive landscape

A lightweight pre-orthogonalization technique for optimizers that accelerates LLM pretraining and reduces perplexity.