Representation Power of a Network of Sigmoid Neurons

Start here In Week 1, a network of perceptrons could represent any Boolean function exactly. Now the big claim: a network of sigmoid neurons with just one hidden layer can approximate any continuous function as closely as we like. The idea is like building a curve out of Lego bricks — and each brick is made from two sigmoids.

In other words: for any function $f(x): \mathbb{R}^{n} \to \mathbb{R}^{m}$, we can always find a neural network (1 hidden layer with enough neurons) whose output g(x) satisfies

$|g(x) - f(x)| < \epsilon$ for any small ε we choose.

This is from Cybenko (1989) and Hornik (1991). The lecture gives an illustrative proof, based on the online book Neural Networks and Deep Learning (Michael Nielsen, chapter 4).

Take any wiggly function. Cover it with many thin rectangular towers (like a bar chart).

• Each tower is shifted to its position and scaled to the function’s height there.
• Adding them up gives an approximation of the function.
• More towers (thinner) → better approximation.

So imagine a black box of “tower makers”: each one takes x and produces one tower. A simple network then adds the towers. Our job: find out what is inside a tower maker.

Step 1 — a very steep sigmoid is a step. $In \sigma (wx + b)$, if w is very large (say 50), the S-curve becomes almost vertical: a step. The step happens where $wx + b = 0, i$.e. at $x = -b/w$. So b controls where the step is.

Step 2 — subtract two steps.

• $h_{11} = \sigma (50(x - 1)) \to$ steps up at $x = 1 (w = 50, b = -50)$
• $h_{12} = \sigma (50(x - 3)) \to$ steps up at $x = 3 (w = 50, b = -150)$
• $h_{21} = h_{11} - h_{12}$ → 0 before 1, 1 between 1 and 3, 0 after 3. A tower!

As a network: input x → two sigmoid hidden neurons ($h_{11}$ with $w_{1}, b_{1}; h_{12}$ with $w_{2}, b_{2}$) → one output neuron $h_{21}$ with weights +1 and −1.

What activation should $h_{21} use$? It just needs to $add +1\cdot h_{11} - 1\cdot h_{12}$, so a linear (identity) output works.

Check at a few points $(x \to h_{11}, h_{12}, h_{21})$:

Lecture example — finding oil Decide whether there is oil at a location on the ocean bed (yes/no), using two factors: salinity $(x_{1})$ and pressure $(x_{2})$. The data shows that y ∈ {oil, no oil} is a complex function of $x_{1}$ and $x_{2}$. We want a network to approximate it — so we need towers in 2-D.

The neuron is now $f(x_{1}, x_{2}) = \frac{1}{1 + \exp (-(w_{1}x_{1} + w_{2}x_{2} + b))}$.

• Vary $w_{1}$ (with $w_{2} = 0, b = 0$): the S-curve rises along the $x_{1}$ direction; large $w_{1} \to a$ step “wall” along $x_{1}$.
• Vary $w_{2}$ (with $w_{1} = 0, b = 0$): same along $x_{2}$.
• Vary b (with $w_{1} = k, w_{2} = 0$): the wall slides to a different position.

Building the 2-D tower:

1. Two steep sigmoids in the $x_{1}$ direction (different b), subtracted → a wall open on two sides (a long ridge, not a tower).
2. Two more in the $x_{2}$ direction, subtracted → another ridge, crossing the first.
3. Add the two ridges → height 2 where they cross (the centre), height 1 on the arms, 0 elsewhere.
4. Pass this sum through one more steep sigmoid with a threshold between 1 and 2 (the lecture uses $w = 50, b = -100, i$.e. fires when the sum is about $\ge 2$) → only the centre survives: a closed tower.

Lecture example — the 2-D tower network Four sigmoid hidden units. $h_{11}$ and $h_{12} put a big$ weight (200) on $x_{1}$ and a small weight (6) on $x_{2}; h_{13}$ and $h_{14} put$ the big weight (±200) on $x_{2}$ and 6 on $x_{1}$. A linear unit computes $h_{11} - h_{12}$ (the $x_{1}$ ridge), another computes $h_{13} - h_{14}$ (the $x_{2}$ ridge). These two are added with weights +1, +1. The final sigmoid has $w = 50, b = -100$, so it outputs $\approx 1$ only where the sum is about 2 or more — the centre where both ridges overlap.

(The lecture leaves “n dimensions” as a “think” question; following the same pattern — one pair of opposite steps for each input direction — gives 2n sigmoid neurons per tower, plus the final thresholding unit.)

The catch: to approximate a complex function well you need many towers, so the hidden layer can become very large. The theorem says such a network exists — not that it is small, or easy to learn.

In one line: Steep sigmoids make steps, two steps make a tower, and enough towers added together approximate any continuous function — so one hidden layer of sigmoid neurons is a universal approximator.

Exam words: representation power, universal approximation theorem, Cybenko 1989, Hornik $1991, |g(x) - f(x)| < \epsilon$, tower function, step function from large $w, x = -b/w$, linear output neuron, 2-D tower.