AdamW optimizer

Gold definitionUpdated Apr 2, 2026

Definition

AdamW is an optimization algorithm that refines the Adam optimizer by decoupling weight decay from the adaptive gradient updates. This modification improves generalization performance and training stability, particularly in deep learning models, by correctly applying L2 regularization.

At a glance

Executive summary

AdamW is an advanced optimization algorithm for training AI models that improves upon the standard Adam optimizer. It achieves better results by applying a technique called weight decay more effectively, which helps models learn more general patterns and avoid memorizing training data. This makes AI models trained with AdamW more reliable and accurate in real-world situations.

TL;DR

AdamW is a smarter version of the Adam optimizer that helps AI models learn better and generalize more effectively by correctly applying a regularization technique called weight decay.

Key points

Decouples weight decay from adaptive gradient updates, unlike original Adam.
Solves the problem of Adam's incorrect L2 regularization, leading to better generalization.
Widely used by researchers and engineers in NLP, CV, and RL, especially for large models like Transformers.
Main alternative (Adam) applies L2 regularization via the loss function, which interacts poorly with adaptive learning rates.
Standard optimizer for large language models, with ongoing research into its specific dynamics in complex training regimes.

Use cases

Training large language models (LLMs) like GPT-3/4 or BERT variants for natural language understanding and generation.
Optimizing Transformer-based models in computer vision for tasks like image classification and object detection.
Fine-tuning pre-trained models across various domains to adapt them to specific downstream tasks.
Developing reinforcement learning agents for complex environments, ensuring stable policy updates.
Training generative adversarial networks (GANs) to produce high-quality synthetic data.

Also known as

Adam with Weight Decay, Adam-W