The Perceptron, Weights, Bias and Errors

Start here The MP neuron treats every input the same and needs a human to choose $\theta$. In real life some reasons matter more than others. The perceptron gives each input its own weight, accepts real numbers, and — most importantly — can learn its weights from data.

• Frank Rosenblatt, an American psychologist, proposed the perceptron in 1958.
• Minsky and Papert (1969) refined and carefully analysed it. Their version is the “perceptron” used in this course.

Two main differences from the MP neuron:

1. Numerical weights $w_{1}, w_{2}, \ldots , w_{n}$ for the inputs, and a way to learn them.
2. Inputs are no longer limited to 0/1 — they can be real numbers.

$y = 1$ if $w_{1}x_{1} + w_{2}x_{2} + \ldots + w_{n}x_{n} \ge \theta y = 0$ otherwise

Now move $\theta$ to the left side:

$y = 1$ if $w_{1}x_{1} + \ldots + w_{n}x_{n} - \theta \ge 0$

The neat trick: add a fake input $x_{0} = 1$ with weight $w_{0} = -\theta$. Then $-\theta = w_{0}x_{0}$, and the rule becomes one clean sum starting at $i = 0$:

• $y = 1$ if $\sum _{i=0}^{n} w_{i}x_{i} \ge 0$
• $y = 0$ if $\sum _{i=0}^{n} w_{i}x_{i} < 0$

Lecture example — will I like this movie? Inputs (0 or $1$): x_{1} = isActorDamon, $x_{2} =$ isGenreThriller, $x_{3} =$ isDirectorNolan. From past viewing we know Nolan movies are almost always liked. So we give $w_{3} a big$ weight. Even if the actor is not Matt Damon and it is not a thriller, $w_{3}$ alone can push the sum over the threshold.

Weights $=$ importance. A bigger weight means that input matters more.

$w_{0}$ is called the bias because it represents a prior (prejudice) — how easily you say “yes” before looking at anything.

Both the weights and the bias are learned from data (here, the viewer’s history).

Why do we keep studying Boolean functions? Because they are the simplest test case. If a model can’t even do AND/OR/XOR, it can’t do real tasks. And “like/dislike a movie” is itself a yes/no (Boolean) decision.

Very important: a perceptron also splits the input space into two halves with a line (plane). So it also can only do linearly separable functions. The difference is that its line can be learned and placed anywhere.

For the perceptron to do OR, each row of the truth table gives one condition:

One solution: $w_{0} = -1, w_{1} = 1.1, w_{2} = 1.1.$ Check every row:

• $(0,0): -1 + 0 + 0 = -1 < 0 \to 0 ✔$
• $(0,1): -1 + 0 + 1.1 = 0.1 \ge 0 \to 1 ✔$
• $(1,0): -1 + 1.1 + 0 = 0.1 \ge 0 \to 1 ✔$
• $(1,1): -1 + 1.1 + 1.1 = 1.2 \ge 0 \to 1 ✔$

Many other solutions also work (for example $w_{0} = -1, w_{1} = 1, w_{2} = 1$).

$Fix w_{0} = -1$ and try different $(w_{1}, w_{2})$. Count how many of the 4 OR rows come out wrong.

Note on the slide: the lecture table lists 1 error for $w_{1} = -1, w_{2} = -1$. Working it out (above) gives 3 errors: (1,0), (0,1) and (1,1) all give a negative sum, so the output is 0 when OR says 1. Only (0,0) is correct.

If we work out the error count for every possible $(w_{1}, w_{2})$ and plot it, we get the error surface. Each point on it is one choice of weights; we want to reach a point with 0 errors.

For OR, an error happens if:

• $(x_{1}, x_{2}) = (0,0)$ but $w_{0} + w_{1}x_{1} + w_{2}x_{2} \ge 0$, or
• $(x_{1}, x_{2}) \ne (0,0)$ but $w_{0} + w_{1}x_{1} + w_{2}x_{2} < 0$.

Trying values by hand is slow. We need an algorithm that finds weights with zero error — that is the Perceptron Learning Algorithm.

In one line: A perceptron is an MP neuron with learnable weights and bias; it still draws only one line, but it can learn where to draw it.

Exam words: perceptron, Rosenblatt (1958), Minsky–Papert (1969), weight, bias $w_{0} = -\theta , x_{0} = 1$, prior/prejudice, error surface, linearly separable.