Neural Networks

A hallway has a light switch at each end. Flip either one and the light changes. So with both switches down the light is off, with exactly one up it is on, and with both up it is off again. Simple to describe, and yet the weighted-sum classifier from Regression and Classification cannot learn it, whatever weights you give it.

Fixing it takes one new idea: stack simple units in layers, with a bend between them. Neural networks are built from that idea. Scaled up enormously, it is behind modern image recognition, speech recognition and language models.

#Why one weighted sum is not enough

The classifier in Regression and Classification computes a weighted sum of the features and compares it with a threshold. Geometrically, that draws one straight line through the space of inputs: on one side "yes", on the other "no".

Now suppose you first computed two new features: "how many switches are up" and "are both up". Then the answer is easy: count minus twice "both". With 1 switch up that gives 1. With 2 up it gives 2 − 2 = 0. A network learns intermediate features like these on its own.

#The building block: a neuron

A neuron in a network is three small steps:

1. Multiply each input by its weight and add them up.
2. Add a bias.
3. Pass the result through an activation function.

The first two steps are exactly the weighted sum you already know. The third adds the bend. The most common activation is ReLU: keep the number if it is positive, otherwise output 0. In code, max(0, z).

Neurons are arranged in layers. The inputs feed a hidden layer, the hidden layer's outputs feed the next layer, and the last layer gives the answer. "Deep learning" just means networks with many hidden layers.

#Trace the hallway network

Here is a network with two hidden neurons that solves the hallway problem. The weights are set by hand so you can follow every number; training can find other weights that also work.

• h1 = ReLU(s1 + s2): how many switches are up.
• h2 = ReLU(s1 + s2 − 1): 1 only when both are up, because the bias of −1 cancels a single switch.
• light = h1 − 2 × h2.

h1 = relu(1*s1 + 1*s2 + 0)h2 = relu(1*s1 + 1*s2 - 1)light = 1*h1 - 2*h2

Switch 2 alone works like switch 1 alone and gives 1. All four positions come out right. Running inputs through the network to get an output like this is called the forward pass.

function relu(z) {
  return Math.max(0, z);
}
 
// Weighted sum of the inputs, plus the bias.
function weightedSum(inputs, weights, bias) {
  let z = bias;
  for (let i = 0; i < inputs.length; i++) z += inputs[i] * weights[i];
  return z;
}
 
// Two switches in, light out (1 = on).
function hallwayLight(s1, s2) {
  const h1 = relu(weightedSum([s1, s2], [1, 1], 0));  // switches up
  const h2 = relu(weightedSum([s1, s2], [1, 1], -1)); // both up?
  return weightedSum([h1, h2], [1, -2], 0);
}
 
hallwayLight(0, 0); // 0
hallwayLight(1, 0); // 1
hallwayLight(1, 1); // 0

def relu(z):
    return max(0, z)
 
def weighted_sum(inputs, weights, bias):
    """Weighted sum of the inputs, plus the bias."""
    return bias + sum(x * w for x, w in zip(inputs, weights))
 
def hallway_light(s1, s2):
    """Two switches in, light out (1 = on)."""
    h1 = relu(weighted_sum([s1, s2], [1, 1], 0))   # switches up
    h2 = relu(weighted_sum([s1, s2], [1, 1], -1))  # both up?
    return weighted_sum([h1, h2], [1, -2], 0)
 
hallway_light(0, 0)  # 0
hallway_light(1, 0)  # 1
hallway_light(1, 1)  # 0

#Why the bend matters

Remove ReLU and see what happens with school algebra. The output becomes

(s1 + s2) − 2 × (s1 + s2 − 1) = 2 − s1 − s2

That is one weighted sum again, a straight line, and it gives 2 for both switches down. Stacking layers of plain weighted sums always collapses into a single weighted sum, however many layers you add. The activation function is what lets each layer build something new on top of the last.

#How a network is trained

Training uses the loop from How Models Learn: start with small random weights, run the forward pass, measure the loss, compute the gradient for every weight, and step downhill. The only hard part is the gradient, because a weight in the first layer affects the output through every layer after it.

The trick is to multiply rates of change along the path. Take the weight from s1 to h2, with both switches up. Nudge it up by 0.01. Then h2 rises by 0.01 × 1 = 0.01, because s1 is 1. ReLU passes that change through unchanged, because its input (1) is positive. The output changes by −2 times whatever h2 changes, so it falls by 0.02. The rate for that weight is 1 × 1 × (−2) = −2: the input, ReLU's rate and the next weight, multiplied together along the path. This is the chain rule. To get the gradient of the loss, multiply by one more rate at the end: how much the loss changes when the output changes.

Backpropagation does this for every weight at once, starting at the output and working backwards layer by layer, reusing each layer's rates for the layer before it. One backward sweep costs only a small multiple of one forward pass, which is why networks with millions or billions of weights can be trained at all.

#Common mistakes

#Try it yourself

#Check yourself

#What to practise next

The hidden values inside a network are lists of numbers that capture what the network has learned about its input. Embeddings takes that idea and turns it into one of the most useful tools in applied AI: representing words, documents and products as points where similar things sit close together.