Backpropagation — Hidden Units, Weights, Biases and the Full Algorithm

Start here We know the gradient at the output: $\hat{y} - e(\ell )$. Now we push it down the network. Three short formulas do all the work: one to pass blame from a layer to the hidden units below, one to go through the activation, and one to get the gradient for the weights and biases. Put them in a loop and you have backpropagation.

In a wide network, one hidden unit $h_{ij}$ (layer i, unit j) feeds every pre-activation $a_{i+1,m}$ in the layer above. So its effect on the loss travels along many paths.

Multi-path chain rule: if p(z) depends on z through several intermediate values $q_m(z)$, then

$\partial p/\partial z = \sum _m (\partial p/\partial q_m) \cdot (\partial q_m/\partial z)$

Here: $p = L(\theta ), z = h_{ij}, q_m = a_{i+1,m}$.

Since $a_{i+1} = W_{i+1} h_{i} + b_{i+1}$, we have $\partial a_{i+1,m}/\partial h_{ij} = W_{i+1,m,j}$. So

$\partial L/\partial h_{ij} = \sum _{m=1}^{k} (\partial L/\partial a_{i+1,m}) \cdot W_{i+1,m,j}$

This is the dot product of the j-th column of $W_{i+1}$ with $\nabla a_{i+1} L$:

$\partial L/\partial h_{ij} = (W_{i+1,\cdot ,j})^{T} \nabla a_{i+1} L$

Stack all j into a vector:

$\nabla h_{i} L = W_{i+1}^{T} (\nabla a_{i+1} L)$

(transpose of the weight matrix times the gradient of the layer above)

Since $h_{ij} = g(a_{ij})$, each unit only depends on its own pre-activation:

$\partial L/\partial a_{ij} = (\partial L/\partial h_{ij}) \cdot g'(a_{ij})$

As a vector:

$\nabla a_{i} L = \nabla h_{i} L \odot [g'(a_{i1}), \ldots , g'(a_{in})]$

⊙ means element-wise multiplication (multiply position by position).

Now we can compute $\nabla a_{i}$ for every layer: start from $\nabla a_L$ (known), $get \nabla h_{L-1}$, then $\nabla a_{L-1}$, then $\nabla h_{L-2}, \ldots$ down to layer 1.

Recall $a_k = b_k + W_k h_{k-1}$, so $a_{\mathrm{ki}} = b_{\mathrm{ki}} + \sum _{j} W_{\mathrm{kij}} h_{k-1,j}$. Then $\partial a_{\mathrm{ki}}/\partial W_{\mathrm{kij}} = h_{k-1,j}$, and

$\partial L/\partial W_{\mathrm{kij}} = (\partial L/\partial a_{\mathrm{ki}}) \cdot h_{k-1,j}$

Lecture example — a 3 × 3 weight matrix

$\nabla W_k L, row$ by row:

• $Row 1: \partial L/\partial a_k1 \cdot h_{k-1,1}, \partial L/\partial a_k1 \cdot h_{k-1,2}, \partial L/\partial a_k1 \cdot h_{k-1,3}$
• $Row 2: \partial L/\partial a_k2 \cdot h_{k-1,1}, \partial L/\partial a_k2 \cdot h_{k-1,2}, \partial L/\partial a_k2 \cdot h_{k-1,3}$
• $Row 3: \partial L/\partial a_k3 \cdot h_{k-1,1}, \partial L/\partial a_k3 \cdot h_{k-1,2}, \partial L/\partial a_k3 \cdot h_{k-1,3}$

Row i uses $\partial L/\partial a_{\mathrm{ki}}$; column j uses $h_{k-1,j}$. This is exactly an outer product.

$\nabla W_k L = \nabla a_k L \cdot h_{k-1}^{T}$

(column vector times row vector $= a$ matrix of the same shape as $W_k$)

$a_{\mathrm{ki}} = b_{\mathrm{ki}} + \sum _{j} W_{\mathrm{kij}} h_{k-1,j}$, so $\partial a_{\mathrm{ki}}/\partial b_{\mathrm{ki}} = 1$, and

$\nabla b_k L = \nabla a_k L$

Training loop

t ← 0;  max_iterations ← 1000
initialise θ₀ = [W₁, …, W_L, b₁, …, b_L]
while t++ < max_iterations:
    a₁, h₁, …, a_{L−1}, h_{L−1}, a_L, ŷ = forward_propagation(θ_t)
    ∇θ_t = backward_propagation(h₁ … h_{L−1}, a₁ … a_L, y, ŷ)
    θ_{t+1} ← θ_t − η ∇θ_t

Forward propagation

for k = 1 to L−1:
    a_k = b_k + W_k h_{k−1}
    h_k = g(a_k)
a_L = b_L + W_L h_{L−1}
ŷ = O(a_L)

Backward propagation

∇a_L L = −(e(y) − ŷ)                         # output gradient
for k = L down to 1:
    ∇W_k L = ∇a_k L · h_{k−1}ᵀ                # weights
    ∇b_k L = ∇a_k L                           # biases
    ∇h_{k−1} L = W_kᵀ ∇a_k L                  # pass to layer below
    ∇a_{k−1} L = ∇h_{k−1} L ⊙ [… g′(a_{k−1,j}) …]   # through activation

($At k = 1$ the last two lines are not needed — $h_{0}$ is the input x, which we don’t update.)

The only missing piece is g′.

Logistic:$g(z) = \sigma (z) = \frac{1}{1 + e^{-z}}$

$g'(z) = (-1)/(1 + e^{-z})^{2} \cdot (-e^{-z}) = [\frac{1}{1 + e^{-z}}] \cdot [e^{-z}/(1 + e^{-z})] = [\frac{1}{1 + e^{-z}}] \cdot [(1 + e^{-z} - 1)/(1 + e^{-z})]$

$\sigma '(z) = g(z)(1 - g(z))$

tanh:$g(z) = (e^{z} - e^{-z})/(e^{z} + e^{-z})$

$g'(z) = [(e^{z} + e^{-z})(e^{z} + e^{-z}) - (e^{z} - e^{-z})(e^{z} - e^{-z})] / (e^{z} + e^{-z})^{2} = 1 - (e^{z} - e^{-z})^{2}/(e^{z} + e^{-z})^{2}$

$\tanh '(z) = 1 - (g(z))^{2}$

Nice property: both derivatives use the output g(z) we already stored in the forward pass — no extra exponentials needed.

Network: 2 inputs → 2 sigmoid hidden units → 2 softmax outputs. All biases 0. $x = [1, 0.5], W_{1} = [[0.1, 0.2], [0.3, -0.1]], W_{2} = [[0.5, -0.4], [0.2, 0.3]]$, true class $\ell = 1$.

Forward:

1. $a_{1} = W_{1}x = [0.1\times 1 + 0.2\times 0.5, 0.3\times 1 + (-0.1)\times 0.5] = [0.2, 0.25]$
2. $h_{1} = \sigma (a_{1}) = [0.5498, 0.5622]$
3. $a_{2} = W_{2}h_{1} = [0.5\times 0.5498 - 0.4\times 0.5622, 0.2\times 0.5498 + 0.3\times 0.5622] = [0.0500, 0.2786]$
4. $\hat{y} = \operatorname{softmax}(a_{2}) = [0.4431, 0.5569]$
5. $L = -\log 0.4431 = 0.814$

Backward:

1. $\nabla a_{2} = \hat{y} - e(1) = [0.4431 - 1, 0.5569] = [-0.5569, 0.5569]$
2. $\nabla W_{2} = \nabla a_{2} h_{1}^{T} = [[-0.5569\times 0.5498, -0.5569\times 0.5622], [0.5569\times 0.5498, 0.5569\times 0.5622]] = [[-0.3062, -0.3131], [0.3062, 0.3131]]$
3. $\nabla b_{2} = [-0.5569, 0.5569]$
4. $\nabla h_{1} = W_{2}^{T}\nabla a_{2} = [0.5\times (-0.5569) + 0.2\times 0.5569, -0.4\times (-0.5569) + 0.3\times 0.5569] = [-0.1671, 0.3898]$
5. $g'(a_{1}) = h_{1}(1 - h_{1}) = [0.2475, 0.2461]$
6. $\nabla a_{1} = \nabla h_{1} \odot g'(a_{1}) = [-0.0414, 0.0959]$
7. $\nabla W_{1} = \nabla a_{1} x^{T} = [[-0.0414, -0.0207], [0.0959, 0.0480]]$

A numerical check (nudging each weight by $10^{-6}$ and re-computing the loss) gives the same $\nabla W_{1}$.

In one line: Backprop $=$ start with $\nabla a_L = \hat{y} - e(\ell )$, then for each layer going down: $\nabla W = \nabla a h^{T}, \nabla b = \nabla a, \nabla h_{\mathrm{below}} = W^{T}\nabla a, \nabla a_{\mathrm{below}} = \nabla h_{\mathrm{below}} \odot g'$.

Exam words: multi-path chain rule, $\nabla h_{i} L = W_{i+1}^{T}\nabla a_{i+1} L$, element-wise product $\odot , \nabla W_k L = \nabla a_k L h_{k-1}^{T}$ (outer product), $\nabla b_k L = \nabla a_k L$, forward propagation, backward propagation, $\sigma ' = \sigma (1 - \sigma ), \tanh ' = 1 - \tanh ^{2}$.