On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer

On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer explores MOGA introduces a new family of width-aware optimizers for stable learning-rate transfer across model widths.. Commercial viability score: 3/10 in Optimizer Research.