Choosing the Learning Rate and Momentum, and Line Search

Start here The learning rate $\eta$ decides how big every step is. Too small and training crawls; too big and it jumps around wildly. This page gives practical tips to pick $\eta$, ways to shrink it over time (annealing), a schedule for momentum, and line search — trying several $\eta$ at every step.

Idea: gentle slopes give tiny gradients, so multiply them by a large $\eta$ to make the steps bigger.

Lecture example — $\eta = 10$ On gentle regions this helps. But on steep regions, the already large gradient is blown up even more. The parameters jump far past the minimum and the loss jumps around.

What we’d like is a learning rate that adjusts itself to the gradient — big where the slope is gentle, small where it is steep. Such adaptive algorithms come next in the course (AdaGrad, RMSProp, Adam).

1. Tune on a log scale: try 0.0001, 0.001, 0.01, 0.1, 1.0.
2. Run a few epochs with each; find the one that works best.
3. Do a finer search around it. If 0.1 was best, now try 0.05, 0.2, 0.3.

Disclaimer from the lecture: these are heuristics — there is no clear winning strategy.

Start with a reasonable $\eta$ and reduce it as training goes on.

Step decay

• Halve the learning rate after every 5 epochs, or
• Halve it after an epoch if the validation error is higher than it was at the end of the previous epoch.

Exponential decay $\eta = \eta _{0} \cdot e^{-kt}$, where $\eta _{0}$ and k are hyperparameters and t is the step number.

(The slide prints this as “$\eta = \eta _{0}^{-kt}$”; the standard exponential decay formula is $\eta _{0} \cdot e^{-kt}$.)

1/t decay $\eta = \eta _{0} / (1 + kt)$, where $\eta _{0}$ and k are hyperparameters and t is the step number.

Worked numbers$(\eta _{0} = 0.1, k = 0.1)$

t	Exponential $\eta _{0}e^{-kt}$	1/t decay $\eta _{0}/(1 + kt)$	Step decay (halve every 5)
0	0.1	0.1	0.1
1	$0.1 \times e^{-0.1} = 0.0905$	$0.1/1.1 = 0.0909$	0.1
5	$0.1 \times e^{-0.5} = 0.0607$	$0.1/1.5 = 0.0667$	0.05
10	$0.1 \times e^{-1} = 0.0368$	$0.1/2 = 0.05$	0.025

A schedule suggested by Sutskever et al., 2013:

$\beta _t = \min (1 - 2^{-1 - log_{2}(⌊t/250⌋ + 1)}, \beta _{\mathrm{max}})$

where $\beta _{\mathrm{max}}$ is chosen from {0.999, 0.995, 0.99, 0.9, 0}.

What it does: $\beta$ starts at 0.5 and grows every 250 steps:

So momentum is small at the start (careful) and larger later (confident), capped at $\beta _{\mathrm{max}}$.

In practice a line search is often done to find a better $\eta$ at each step.

• Update w using different values of $\eta$.
• Keep the updated value that gives the lowest loss.
• So at each step we use the best $\eta$ from the available choices.

Worked numbers (first step from $w = b = -2$, lecture data; checked in code) Loss after the step: $\eta = 0.1 \to 0.41581, \eta = 0.5 \to 0.41577, \eta = 1 \to 0.41573, \eta = 2 \to 0.41564, \eta = 10 \to$ 0.41488. Line search keeps $\eta = 10$ here, because we start on a very flat region.

Plus: convergence is faster than vanilla gradient descent with a fixed $\eta$. Minus: many more computations in each step (one loss evaluation per candidate $\eta$).

Lecture example — line search in action Blue: gradient descent with fixed $\eta = 1$. Black: gradient descent with line search. Line search converges faster. It shows some oscillations too, but these are different from momentum/NAG oscillations — they come from picking $a big \eta$ when it happens to give the lowest loss for that one step.

In one line: Tune $\eta$ on a log scale, shrink it over time (step, exponential or 1/t decay), grow momentum over time, or use line search to pick the best $\eta$ at every step at extra cost.

Exam words: learning rate, log-scale search, annealing, step decay, exponential decay, 1/t decay, hyperparameter, validation error, Sutskever momentum schedule, $\beta _{\mathrm{max}}$, line search.