The Supervised Learning Setup, Loss and Guess Work

Start here Before learning any weights we need to say clearly what we are learning and how we measure mistakes. Every machine learning problem in this course has the same five parts. Then we try the simplest possible way to learn — guessing — and see why it does not scale.

A single perceptron cannot perfectly classify data that is not linearly separable. “Cannot deal with” means: whatever line it draws, some points end up on the wrong side.

From now on we accept that driving the error to 0 is hard in most cases. Instead we aim for the minimum possible error.

The learning algorithm should aim to minimise the loss function.

Lecture example — movie setup Data: {$x_{i} =$ movie, $y_{i} =$ like/dislike}. Model: $\hat{y} = \frac{1}{1 + e^{-w^{T}x}} =$ probability of liking. Parameters: w. Algorithm: gradient descent. Loss: $L(w) = \sum _{i=1}^{n} (\hat{y}_{i} - y_{i})^{2}$ (squared error). Find the w that makes this smallest.

To keep things simple:

• only one input x,
• $w_{0}$ is renamed b (bias), as most books do,
• task: predict criticsRating (y) from imdbRating (x).

Model: $f(x) = \frac{1}{1 + e^{-(wx + b)}}$

Training data: N pairs (x, y). Objective:

minimise $L(w, b) = (1/N) \sum _{i=1}^{N} (y_{i} - f(x_{i}))^{2}$

Lecture example — two points Train with $(x, y) =$ (0.5, 0.2) and (2.5, 0.9). We want to find w*, b* such that f(0.5) → 0.2 and f(2.5) → 0.9. In other words: find a sigmoid curve that passes through both points.

Start with a random guess: $w = 3, b = -1$. How bad is it? (The lecture uses ½ in front, since $N = 2$.)

$L(w, b) = \frac{1}{2} \times [(0.9 - f(2.5))^{2} + (0.2 - f(0.5))^{2}]$

Step by step:

1. $f(2.5) = \sigma (3 \times 2.5 - 1) = \sigma (6.5) = \frac{1}{1 + e^{-6.5}} = \frac{1}{1 + 0.0015} =$0.9985
2. $f(0.5) = \sigma (3 \times 0.5 - 1) = \sigma (0.5) = \frac{1}{1 + 0.6065} =$0.6225
3. $(0.9 - 0.9985)^{2} = (-0.0985)^{2} = 0.0097$
4. $(0.2 - 0.6225)^{2} = (-0.4225)^{2} = 0.1785$
5. $L = \frac{1}{2} \times (0.0097 + 0.1785) = \frac{1}{2} \times 0.1882 =$0.0941

(The slide rounds f(2.5) to 0.99 and f(0.5) to 0.62.) We want L as close to 0 as possible. So we keep changing w and b by hand:

(The slides show 0.14, 0.0214 and 0.0028 for three of these rows — small rounding differences from the values above.)

The exact best answer can be worked out because $\sigma (wx + b) = y$ means $wx + b = \log (y/(1 - y))$:

• $0.5w + b = \log (0.2/0.8) = -1.386$
• $2.5w + b = \log (0.9/0.1) = 2.197$
• Subtract: $2w = 3.583 \to$ $w = 1.792$, then $b = -1.386 - 0.896 = -2.282$.

So the guesses were walking towards (1.79, −2.28).

With only 2 parameters we can draw the loss for every (w, b) — this is the error surface. Then just pick the lowest point.

Problems with guessing / plotting everything:

• With many data points and many parameters (real networks have millions), plotting every combination is impossible (intractable).
• Even here, the plot covers only w, b in (−6, +6), not (−∞, +∞).

Geometrically, the guess-work algorithm is just hopping around on this surface. We need a principled way to move downhill to the minimum — that is gradient descent (next page).

In one line: Supervised learning $=$ data + model + parameters + learning algorithm + loss; training means finding parameters with the smallest loss, and guessing does not scale.

Exam words: supervised learning, data, model, parameters, learning algorithm, objective/loss/error function, squared error loss, training, error surface, intractable.