Feedforward Neural Networks — Layers, Notation and Parameters

Start here A feedforward neural network is just many sigmoid-type neurons arranged in layers, where information flows one way: from the input, through hidden layers, to the output. This page fixes the notation $(a_{i}, h_{i}, W_{i}, b_{i})$ that the whole of backpropagation uses — learn it well and the rest becomes much easier.

• Input: an n-dimensional vector $x = (x_{1}, \ldots , x_{n})$. Call it layer 0$(h_{0} = x).$
• Hidden layers: $L - 1$ of them (2 in the picture), each with n neurons.
• Output layer: k neurons (for example, one per class). Call it layer L (here $L = 3$).
• Between layer $i - 1$ and layer i: a weight matrix $W_{i}$ and a bias vector $b_{i}$.

(One later slide writes $\nabla W_L \in \mathbb{R}^{n\times k}$; the correct shape, matching $W_L$, is $k \times n$.)

Everyday example A factory line: raw material (input) passes through station 1, station 2 (hidden layers), and comes out as the finished product (output). Nothing goes backwards while making the product.

Pre-activation (the weighted sum):

$a_{i} = b_{i} + W_{i} h_{i-1}$

Activation (apply the activation function g):

$h_{i} = g(a_{i})$ $g =$ logistic (sigmoid), tanh, linear, …

Output layer uses an output activation function O:

$\hat{y} = f(x) = h_L = O(a_L)$ $O = \operatorname{softmax}$, linear, …

$a_{i}$ and $h_{i}$ are vectors (one number per neuron in the layer). We write $a_{i}$ instead of $a_{i}(x)$ to keep things short.

Lecture example — sizes with numbers Say x has $n = 3$ values, each hidden layer has 3 neurons, and there are $k = 2$ outputs. $W_{1}: 3 \times 3, b_{1}: 3 \to a_{1} = b_{1} + W_{1}x$ is a 3-vector, $h_{1} = g(a_{1})$ is a 3-vector. $W_{2}: 3 \times 3, b_{2}: 3 \to a_{2}, h_{2}$ are 3-vectors. $W_{3}: 2 \times 3, b_{3}: 2 \to a_{3}$ is a 2-vector, $\hat{y} = O(a_{3})$ is a 2-vector. Total parameters $= (9 + 3) + (9 + 3) + (6 + 2) =$ 32.

$For L = 3$:

$\hat{y}_{i} = f(x_{i}) = O(W_{3} g(W_{2} g(W_{1}x_{i} + b_{1}) + b_{2}) + b_{3})$

Read it inside-out: multiply by $W_{1}, add b_{1}$, apply g → multiply by $W_{2}, add b_{2}$, apply g → multiply by $W_{3}, add b_{3}$, apply O.

Gradient descent is unchanged; only $\theta$ is bigger:

t ← 0;  max_iterations ← 1000
initialise θ₀ = [W₁, …, W_L, b₁, …, b_L]
while t++ < max_iterations:
    θ_{t+1} ← θ_t − η ∇θ_t

Except now $\nabla \theta$ is huge: it contains $\partial L/\partial$(every single weight and bias) — $\partial L/\partial W_{111}, \partial L/\partial W_{112}, \ldots , \partial L/\partial W_L,nk, \partial L/\partial b_{11}, \ldots$. It is made of:

• $\nabla W_{1}, \ldots , \nabla W_{L-1} \in \mathbb{R}^{n\times n}, \nabla W_L \in \mathbb{R}^{k\times n}$
• $\nabla b_{1}, \ldots , \nabla b_{L-1} \in \mathbb{R}^{n}, \nabla b_L \in \mathbb{R}^{k}$

So we must answer two questions:

1. How do we choose the loss function $L(\theta )$? → next page.
2. How do we compute $\nabla \theta$ efficiently? → backpropagation (pages 3 and 4).

In one line: A feedforward network computes $a_{i} = b_{i} + W_{i}h_{i-1}$ and $h_{i} = g(a_{i})$ layer by layer, ending with $\hat{y} = O(a_L)$; its parameters are all the $W_{i}$ and $b_{i}$, learned by gradient descent.

Exam words: feedforward neural network, input layer (layer 0), hidden layer, output layer (layer L), pre-activation $a_{i}$, activation $h_{i}$, activation function g, output function O, weight matrix $W_{i}$, bias $b_{i}, \theta$.