SECTION 02 / 4

Momentum-Based Gradient Descent and Nesterov Accelerated Gradient

On this page 9 headings

BUILD INTUITION WITH NUMBERS

Compare momentum with a look-ahead gradient

Use L(w) = ½w², current w = 3, previous accumulator u = 2, β = 0.9 and η = 0.1. Here η is outside the accumulator, including in the look-ahead distance.

Ordinary momentum uses the gradient at the current point.
$\begin{aligned}g&=3\\u_{new}&=0.9(2)+3=4.8\\w_{new}&=3-0.1(4.8)=2.52\end{aligned}$
For this NAG convention, first find the look-ahead point.
$w_{look}=3-0.1(0.9)(2)=2.82$
Use the gradient there, then update from the original current point.
$\begin{aligned}g_{look}&=2.82\\u_{new}&=0.9(2)+2.82=4.62\\w_{new}&=3-0.1(4.62)=2.538\end{aligned}$

What changed? NAG changes where the gradient is measured. This example names its scaling convention explicitly; do not mix it with code that absorbs η into the accumulator.

Quick check: Why is the NAG accumulator a little smaller here?

The look-ahead point is closer to the bowl’s minimum, so its gradient is 2.82 instead of 3.

Swipe long formulas, tables or code sideways to see the rest.

Start here If someone keeps telling you “go left… go left… go left”, you start walking faster to the left. Momentum does exactly this: it remembers past gradients and builds up speed. It crosses flat regions quickly — but it can overshoot. Nesterov Accelerated Gradient (NAG) fixes the overshooting by looking ahead before stepping.

1. Momentum: the intuition

If I am repeatedly asked to move in the same direction, I should gain confidence and take bigger steps in that direction — just like a ball rolling down a slope gains momentum.

A ball rolling down a slope speeds up, overshoots the valley, and swings back — Momentum = remember the past direction; speed builds up, but it may overshoot.

2. The update rule

$u_t = \beta u_{t-1} + \nabla w_t$ $w_{t+1} = w_t - \eta u_t$

with $u_{-1} = 0, w_{0}$ random, and $0 \le \beta < 1$ ( $\beta$ is the momentum factor, often 0.9).

So besides the current gradient, we also look at the history of updates.

3. Unrolling: an exponentially weighted average

$u_{0} = \beta \cdot 0 + \nabla w_{0} = \nabla w_{0}$
$u_{1} = \beta u_{0} + \nabla w_{1} = \beta \nabla w_{0} + \nabla w_{1}$
$u_{2} = \beta u_{1} + \nabla w_{2} = \beta (\beta \nabla w_{0} + \nabla w_{1}) + \nabla w_{2} = \beta ^{2}\nabla w_{0} + \beta \nabla w_{1} + \nabla w_{2}$
In general: $u_t = \sum _{\tau =0}^{t} \beta ^{t-\tau } \nabla w_\tau$

$u_t$ is an exponentially weighted sum of the current and all past gradients. Recent gradients get weight close to 1; old ones fade $(\beta , \beta ^{2}, \beta ^{3}, \ldots )$ .

Feel it: if the gradient is always 1 and $\beta = 0.9$ :

t	$u_t = 0.9 u_{t-1} + 1$
0	1
1	$0.9 \times 1 + 1 = 1.9$
2	$0.9 \times 1.9 + 1 = 2.71$
3	$0.9 \times 2.71 + 1 = 3.439$
…	grows towards $\frac{1}{1 - 0.9} =$ 10

So on a long gentle slope, momentum can take steps up to 10 times bigger than plain gradient descent.

4. The code (and a small fix)

def do_mgd(max_epochs):
    w, b, eta = -2, -2, 1.0
    prev_uw, prev_ub, beta = 0, 0, 0.9
    for i in range(max_epochs):
        dw, db = 0, 0
        for x, y in zip(X, Y):
            dw += grad_w(w, b, x, y)
            db += grad_b(w, b, x, y)
        uw = beta * prev_uw + eta * dw
        ub = beta * prev_ub + eta * db
        w = w - uw          # the slide writes "w - vw" — a typo; it must be uw
        b = b - ub          # likewise "b - vb" should be ub
        prev_uw = uw
        prev_ub = ub

Note: the code puts $\eta$ inside $u$ (u = βu + η∇w, then $w = w - u$ ). The formula above puts $\eta$ outside $(w = w - \eta u)$ . With a fixed $\eta$ these are the same thing — the code’s u is just $\eta$ times the formula’s u.

5. Is moving fast always good?

Good: even on gentle slopes, momentum takes large steps because the built-up momentum carries it along.

Bad: it can overshoot the goal. In a narrow valley (error high on both sides of the minimum), momentum oscillates in and out of the valley and takes many U-turns before settling.

Lecture example — the race Start $w_{0} = -2, b_{0} = -4, \eta = 1$ , on a surface with a narrow valley. After 50 iterations, momentum has reached an error of 0.00001, while plain gradient descent is still stuck at 0.16. So, despite the U-turns, momentum still converges much faster than plain gradient descent.

6. Nesterov Accelerated Gradient (NAG): look before you leap

Can we reduce the oscillations? Yes.

From $u_t = \beta u_{t-1} + \nabla w_t$ we know we are going to move at least by $\beta u_{t-1}$ , and then a bit more by the gradient. So why not first jump to where momentum will take us, measure the gradient there, and then decide?

NAG update:

$u_t = \beta u_{t-1} + \nabla (w_t - \beta u_{t-1})$ $w_{t+1} = w_t - \eta u_t$

(In the lecture’s code form, where $\eta$ is inside u, the look-ahead point is $w - \beta \cdot$ prev_v; in the formula form above, it is $w_t - \eta \beta u_{t-1}$ . Same idea: step to where momentum is taking you, then measure.)

NAG takes the momentum step first, measures the gradient at the look-ahead point, then corrects; momentum measures at the current point — Momentum measures the slope where it stands; NAG measures it where it is about to land.

def do_nag(max_epochs):
    w, b, eta = -2, -2, 1.0
    prev_vw, prev_vb, beta = 0, 0, 0.9
    for i in range(max_epochs):
        dw, db = 0, 0
        # partial update: where momentum alone would take us
        v_w = beta * prev_vw
        v_b = beta * prev_vb
        for x, y in zip(X, Y):
            # look ahead: gradient at the look-ahead point
            dw += grad_w(w - v_w, b - v_b, x, y)
            db += grad_b(w - v_w, b - v_b, x, y)
        vw = beta * prev_vw + eta * dw
        vb = beta * prev_vb + eta * db
        w = w - vw
        b = b - vb
        prev_vw = vw
        prev_vb = vb

Why it helps: if momentum is about to overshoot, the gradient at the look-ahead point already points back, so NAG corrects its course earlier. Its oscillations are smaller, and it is less likely to escape the minimum valley.

7. Comparing the three

Three paths on the same contour map: plain GD barely moves, momentum overshoots with big U-turns, NAG makes smaller U-turns — Same start, same η: GD crawls, momentum overshoots, NAG overshoots less.

Method	Uses	Strength	Weakness
Gradient descent	current gradient only	stable	very slow on gentle slopes
Momentum	current + past gradients	fast on gentle slopes	overshoots, many U-turns
NAG	gradient at the look-ahead point + past	fast, shorter U-turns	slightly more bookkeeping

Worked comparison ( $L = w^{2}$ , start $w = 1, \eta = 0.1, \beta = 0.9$ , so $\nabla w = 2w$ ):

Step	GD	Momentum	NAG
1	0.8	0.8	0.8
2	0.64	0.46	0.496
3	0.512	0.062	0.178
4	0.410	−0.309 (overshot past 0)	−0.087 (smaller overshoot)

(Full arithmetic for these steps is on the How to Solve page.)

In one line: Momentum adds a decaying memory of past gradients to speed through flat regions (but overshoots); NAG measures the gradient at the look-ahead point to cut down the overshooting.

Exam words: momentum, $\beta , u_t = \beta u_{t-1} + \nabla w_t$ , exponentially weighted average, overshoot, oscillation, U-turn, Nesterov Accelerated Gradient, look-ahead, look before you leap.

Quick check

$\beta = 0.5$ , gradients $\nabla w_{0} = 2, \nabla w_{1} = 2, \nabla w_{2} = 2$ . Find $u_{2}$ .

u_{0} = 2; u_{1} = 0.5\times 2 + 2 = 3; u_{2} = 0.5\times 3 + 2 = 3.5.

What is the weight of $\nabla w_{0}$ inside $u_{3}$ when $\beta = 0.9$ ?

\beta ^{3} = 0.729.

What happens with $\beta = 0$ ?

u_t = \nabla w_t

, so momentum becomes plain gradient descent.

Where does NAG compute the gradient?

At the look-ahead point — where the momentum step alone would take the parameters — not at the current point.

Why can momentum leave the minimum valley?

Its accumulated velocity carries it past the minimum and up the other side before the gradient can slow it down.

YOUR STUDY PROGRESS

Finished this section? Save your progress.