Feedforward networks and backpropagation.
Learn through illustrated explanations, worked examples, practice questions and quick revision.
Before you start
Review matrix multiplication, transposes, vector shapes and the chain rule.
By the end: Trace a forward pass and compute gradients back through a small network.
Check the notation
a: pre-activation; h: activation; W: weight matrix; b: bias vector; L: loss. Check each layer’s shapes before multiplying.
Study sections
Follow the order or choose a sectionFeedforward Neural Networks — Layers, Notation and Parameters
A feedforward neural network is just many sigmoid-type neurons arranged in layers, where information flows one way : from the input, th…
02Output Functions and Loss Functions — Linear, Softmax, Squared Error, Cross-Entropy
What the network should output depends on the question. Predicting a rating (a real number) needs a different last layer and a differen…
03Backpropagation — the Idea, and Gradients at the Output Layer
The network made a mistake. Who is to blame? Backpropagation answers this by passing the blame backwards, layer by layer, using the cha…
04Backpropagation — Hidden Units, Weights, Biases and the Full Algorithm
We know the gradient at the output: ŷ − e(ℓ). Now we push it down the network. Three short formulas do all the work: one to pass blame…