Optimizers

The engine of learning — from SGD to Adam and beyond

Gradient Descent Fundamentals

All neural network training is based on gradient descent: move parameters in the direction that decreases the loss.

θ^{n e w} = θ^{o l d} - α \nabla_{θ} L

Batch vs Stochastic vs Mini-batch

Type	Gradient Computed On	Trade-offs
Batch GD	Entire dataset	Accurate but slow, no escape from local minima
Stochastic GD	Single sample	Fast, noisy, helps escape local minima
Mini-batch GD	Batch of samples	Best of both worlds, parallelizable

Mini-batch is standard — typically 32, 64, 128, or 256 samples.

Momentum

Momentum accelerates gradient descent by accumulating velocity in consistent directions.

Intuition

Imagine a ball rolling down a hill:

Accelerates in consistent downhill direction
Dampens oscillations in other directions

Algorithm

v_{t} = β v_{t - 1} + \nabla_{θ} L

θ = θ - α v_{t}

Typical $β = 0.9$ (90% of previous velocity retained).

SGD vs Momentum

Benefits

Faster convergence in consistent gradient directions
Dampens oscillations in high-curvature directions
Helps escape shallow local minima (momentum carries through)

Adaptive Learning Rates

Different parameters may need different learning rates. Adaptive methods adjust per-parameter.

Adagrad

Adapts learning rate based on historical gradients:

G_{t} = G_{t - 1} + g_{t}^{2}

θ = θ - \frac{α}{\sqrt{G_{t}} + ϵ} g_{t}

Problem: Learning rate only decreases — eventually too small.

RMSprop

Fixes Adagrad with exponential moving average:

v_{t} = β v_{t - 1} + (1 - β) g_{t}^{2}

θ = θ - \frac{α}{\sqrt{v_{t}} + ϵ} g_{t}

Typical $β = 0.99$ .

Adam (Adaptive Moment Estimation)

Adam combines momentum (first moment) with RMSprop (second moment).

Algorithm

First moment (mean) estimate:

m_{t} = β_{1} m_{t - 1} + (1 - β_{1}) g_{t}

Second moment (variance) estimate:

v_{t} = β_{2} v_{t - 1} + (1 - β_{2}) g_{t}^{2}

Bias correction (important early in training):

{\hat{m}}_{t} = \frac{m_{t}}{1 - β_{1}^{t}}

{\hat{v}}_{t} = \frac{v_{t}}{1 - β_{2}^{t}}

Update:

θ = θ - α \frac{{\hat{m}}_{t}}{\sqrt{{\hat{v}}_{t}} + ϵ}

Default Hyperparameters

Parameter	Default	Role
$α$	0.001	Learning rate
$β_{1}$	0.9	First moment decay
$β_{2}$	0.999	Second moment decay
$ϵ$	1e-8	Numerical stability

Why Bias Correction?

At $t = 1$ : $m_{1} = 0.1 \cdot g_{1}$ (severely underestimated if $β_{1} = 0.9$ )

Bias correction: ${\hat{m}}_{1} = \frac{0.1 \cdot g_{1}}{1 - 0.9} = g_{1}$ (correct scale)

Adam Components

AdamW (Decoupled Weight Decay)

AdamW fixes a subtle bug in how Adam implements L2 regularization.

The Problem

L2 regularization adds $λ | | θ | |^{2}$ to loss. In vanilla Adam, this becomes:

g_{t} = \nabla L + λ θ

But Adam's adaptive scaling interferes with the regularization!

The Solution

Decouple weight decay from the gradient:

θ = θ - α \frac{{\hat{m}}_{t}}{\sqrt{{\hat{v}}_{t}} + ϵ} - α λ θ

Weight decay is applied directly, not through gradients.

When to Use

AdamW for transformers — standard in BERT, GPT, etc.
Adam for most other cases — simpler, usually works fine
If using weight decay with Adam, prefer AdamW

Learning Rate Schedules

The optimal learning rate often changes during training.

Step Decay

Reduce LR by factor at specific epochs:

python

lr = initial_lr * (0.1 ** (epoch // 30))  # Decay by 10x every 30 epochs

Exponential Decay

α_{t} = α_{0} \cdot e^{- k t}

Cosine Annealing

α_{t} = α_{m i n} + \frac{1}{2} (α_{m a x} - α_{m i n}) (1 + \cos (\frac{t}{T} π))

Smooth decay from max to min over T steps.

Learning Rate Schedules

Warmup

Start with very low LR, gradually increase:

python

if step < warmup_steps:
    lr = initial_lr * (step / warmup_steps)
else:
    lr = schedule(step - warmup_steps)

Why warmup?

At initialization, gradients may be unstable
High LR early can cause divergence
Warmup lets model find stable region before large steps
Essential for large batch training and transformers

OneCycleLR

Learning rate increases then decreases over training:

Start low (warmup)
Peak in middle
Decay to very low at end

Often achieves better results in fewer epochs.

Optimizer Comparison

Optimizer	Pros	Cons	Best For
SGD	Simple, generalizes well	Slow, sensitive to LR	Vision with LR schedule
SGD + Momentum	Faster than SGD	Still needs tuning	ImageNet training
Adam	Fast, adaptive, works out of box	May generalize worse	Default, prototyping
AdamW	Proper weight decay	Slightly more complex	Transformers
RAdam	Robust early training	Less common	When Adam diverges early

Decision Guide

Transformers?
  └─ AdamW with warmup + cosine decay

CNNs for final training?
  └─ SGD + momentum + step/cosine decay

Quick prototyping?
  └─ Adam with default settings

Training unstable?
  └─ Add warmup, reduce LR, try RAdam

Optimizer Paths

Practical Recommendations

Learning Rate Selection

Start with defaults: Adam 1e-3, SGD 0.1
LR finder: Train for short time with increasing LR, find where loss starts exploding
Grid search: Try [1e-2, 1e-3, 1e-4] and pick best

Batch Size Considerations

Larger batch → can use larger LR (linear scaling rule)
Larger batch → may generalize worse (sharper minima)
Gradient accumulation simulates large batches with small memory

Common Issues

Issue	Likely Cause	Solution
Loss = NaN	LR too high	Reduce LR, add gradient clipping
Loss plateaus	LR too low or stuck	Increase LR or use scheduler
Loss oscillates	LR too high	Reduce LR
Training diverges	No warmup	Add warmup period

Interview Questions

Q1: "Explain how Adam works."

Adam combines two ideas:
First moment (momentum):
$m_{t} = β_{1} m_{t - 1} + (1 - β_{1}) g_{t}$
Exponential moving average of gradients — accelerates in consistent directions.
Second moment (adaptive LR):
$v_{t} = β_{2} v_{t - 1} + (1 - β_{2}) g_{t}^{2}$
Exponential moving average of squared gradients — scales LR by gradient magnitude.
Bias correction:
${\hat{m}}_{t} = m_{t} / (1 - β_{1}^{t}), {\hat{v}}_{t} = v_{t} / (1 - β_{2}^{t})$
Corrects initialization bias (moments start at 0).
Update:
$θ = θ - α \frac{{\hat{m}}_{t}}{\sqrt{{\hat{v}}_{t}} + ϵ}$
Result: Each parameter gets adapted learning rate based on its gradient history. Parameters with large, consistent gradients get smaller effective LR; parameters with small gradients get larger effective LR.

Q2: "When would you use SGD over Adam?"

SGD with momentum often generalizes better for computer vision:
ImageNet: Best results typically with SGD + momentum + LR schedule
Adam can find sharper minima, which may generalize worse
SGD's implicit regularization through noise may help
When to use SGD:
Final training when you want best generalization
Large-scale vision models (ResNet, EfficientNet on ImageNet)
When you have time to tune LR schedule
When to use Adam:
Prototyping (works out of box)
Transformers/NLP (AdamW is standard)
GANs (more stable than SGD)
When SGD training is unstable
Practical approach: Start with Adam for fast iteration, try SGD for final training if needed.

Q3: "Why is learning rate warmup important?"

At initialization, gradients are unreliable:
Based on random weights
May point in wrong directions
Large steps can cause divergence
Warmup addresses this:
Start with very low LR (e.g., 0.1x final LR)
Gradually increase over warmup period (e.g., 1000 steps)
Allows model to find stable region before taking large steps
When warmup is critical:
Large batch training: Gradients have lower variance, can "overshoot" more easily
Transformers: Deep models with many attention layers are sensitive to early training
Adam's bias correction issues: Even with correction, early updates can be unstable
Typical warmup:
Linear increase over first 5-10% of training
Or fixed number of steps (1000-10000)

Key Takeaways

Momentum accelerates convergence by accumulating velocity in consistent directions.
Adam adapts learning rates per-parameter using first and second moment estimates.
AdamW decouples weight decay — use it for transformers.
Learning rate schedules (warmup + decay) are crucial for good results.
SGD may generalize better than Adam for vision; Adam is standard for transformers.
Warmup prevents early divergence in large batch training and deep models.

Optimizers ​

Gradient Descent Fundamentals ​

Batch vs Stochastic vs Mini-batch ​

Momentum ​

Intuition ​

Algorithm ​

Benefits ​

Adaptive Learning Rates ​

Adagrad ​

RMSprop ​

Adam (Adaptive Moment Estimation) ​

Algorithm ​

Default Hyperparameters ​

Why Bias Correction? ​

AdamW (Decoupled Weight Decay) ​

The Problem ​

The Solution ​

When to Use ​

Learning Rate Schedules ​

Step Decay ​

Exponential Decay ​

Cosine Annealing ​

Warmup ​

OneCycleLR ​

Optimizer Comparison ​

Decision Guide ​

Practical Recommendations ​

Learning Rate Selection ​

Batch Size Considerations ​

Common Issues ​

Interview Questions ​

Q1: "Explain how Adam works." ​

Q2: "When would you use SGD over Adam?" ​

Q3: "Why is learning rate warmup important?" ​

Key Takeaways ​

Optimizers

Gradient Descent Fundamentals

Batch vs Stochastic vs Mini-batch

Momentum

Intuition

Algorithm

Benefits

Adaptive Learning Rates

Adagrad

RMSprop

Adam (Adaptive Moment Estimation)

Algorithm

Default Hyperparameters

Why Bias Correction?

AdamW (Decoupled Weight Decay)

The Problem

The Solution

When to Use

Learning Rate Schedules

Step Decay

Exponential Decay

Cosine Annealing

Warmup

OneCycleLR

Optimizer Comparison

Decision Guide

Practical Recommendations

Learning Rate Selection

Batch Size Considerations

Common Issues

Interview Questions

Q1: "Explain how Adam works."

Q2: "When would you use SGD over Adam?"

Q3: "Why is learning rate warmup important?"

Key Takeaways