Output Functions and Loss Functions — Linear, Softmax, Squared Error, Cross-Entropy

Start here What the network should output depends on the question. Predicting a rating (a real number) needs a different last layer and a different loss than picking one fruit out of four. This page gives the two standard pairs you will use again and again: linear + squared error and softmax + cross-entropy.

Lecture example — movie ratings Input: movie features (isActorDamon, isDirectorNolan, …). Output: three ratings, $y = [7.5, 8.2, 7.7]$ (IMDb rating, critics’ rating, RT rating). $So y \in \mathbb{R}^{3}$.

Loss: the loss should measure how $far \hat{y}$ is from y. For real-valued outputs, squared error does this:

$L(\theta ) = (1/N) \sum _{i=1}^{N} \sum _{j=1}^{k} (\hat{y}_{ij} - y_{ij})^{2}$

Output function: can O be the logistic (sigmoid)? No. Sigmoid squashes everything into (0, 1), but a rating like 8.2 is outside that range. So use a linear output:

$\hat{y} = O(a_L) = W_O a_L + b_O$

$Now \hat{y}$ is not stuck between 0 and 1.

Lecture example — fruit image Classify an image as one of 4 classes: Apple, Mango, Orange, Banana. If it is an apple, the true label is $y = [1, 0, 0, 0]$.

We could use squared error, but there is a better choice.

Notice: y is a probability distribution (all values $\ge 0$, total $= 1$). $So \hat{y}$ should also be a probability distribution.

$\hat{y}_{j} = O(a_L)_{j} = e^{a_L,j} / \sum _{i=1}^{k} e^{a_L,i}$

• $e^{anything}$ is positive → every $\hat{y}_{j} > 0$.
• Dividing by the total → they add up to 1.

Worked numbers: $a_L = [2.0, 1.0, 0.1]$ for [apple, mango, orange].

1. $e^{2} = 7.389, e^{1} = 2.718, e^0.1 = 1.105$
2. $Sum = 7.389 + 2.718 + 1.105 = 11.212$
3. $\hat{y} = [7.389/11.212, 2.718/11.212, 1.105/11.212] =$[0.659, 0.242, 0.099]
4. Check: $0.659 + 0.242 + 0.099 = 1.000$ ✔

$L(\theta ) = - \sum _{c=1}^{k} y_c \log \hat{y}_c$

Since $y_c = 1$ only for the true class ℓ and 0 for all others, every term except one vanishes:

$L(\theta ) = - \log \hat{y}_\ell$

So for classification we:

minimise $L(\theta ) = - \log \hat{y}$_ℓ or equivalently maximise $\log \hat{y}$_ℓ

(log here is the natural log, ln.)

$Is \hat{y}$_ℓ a function of $\theta$? Yes:

$\hat{y}_\ell = [O(W_{3} g(W_{2} g(W_{1}x + b_{1}) + b_{2}) + b_{3})]_\ell$

It is the probability that x belongs to class ℓ. We want to push it close to 1. $\log \hat{y}$_ℓ is called the log-likelihood of the data.

Other loss functions exist for other problems, but these two are by far the most common. For the rest of the backpropagation derivation we use softmax output + cross-entropy loss.

(The lecture also links a separate video on Information and Entropy for the background of cross-entropy.)

In one line: Use a linear output with squared error for real-valued targets, and a softmax output with cross-entropy (−log of the true class’s probability) for classification.

Exam words: linear output, squared error loss, softmax, probability distribution, one-hot label, cross-entropy, $-\log \hat{y}$_ℓ, log-likelihood.