Regression and Classification

Two questions land on your desk on the same day. The first: "How much rent should we ask for this 60 m² flat?" The second: "Is this incoming message spam?" Both are prediction problems, and both can be learned from past examples. But the answers have different shapes. The first is a number on a scale. The second is one of two boxes.

Predicting a number is called regression. Predicting a category is called classification. Most practical machine learning is one or the other, and they share almost all their machinery.

#Regression: fit a line

In How Models Learn the model was minutes = w × km. That line is forced through zero: 0 km means 0 minutes. Rent does not work like that. Here are three flats:

The best line through zero is about rent = 21.33 × size. It predicts 640 for the small flat and 1,493 for the large one, too low and then too high. The fix is a second parameter, the bias b, which lets the line start at any height:

rent = w × size + b

Gradient descent now tunes two dials instead of one. The gradient for w is the same as before: the average of 2 × error × input. For b, nudging it up by a tiny amount raises every prediction by that same amount, so its gradient is just the average of 2 × error. The line with the lowest loss is w = 15 and b = 350, which fits all three flats exactly: 15 × 30 + 350 = 800. The 60 m² flat gets 15 × 60 + 350 = 1,250.

Real models have one weight per feature: size, floor, distance to the station, and so on, plus one bias. Each weight says how much the prediction moves when that feature goes up by one.

#Classification: from a score to a probability

Now the spam question. Take one feature, the number of links in a message. You could compute a score the same way, score = w × links + b. But a score can be any number, like −2.3 or 4.4, and you want an answer between "surely not spam" and "surely spam".

So squash the score into the range 0 to 1 with the sigmoid function:

P(spam) = 1 / (1 + e−score)

Here e is a fixed number, about 2.718. You do not need its details, only the shape it produces. A score of 0 gives exactly 0.5. Large positive scores give values close to 1. Large negative scores give values close to 0. The output never quite reaches 0 or 1, so the model never claims total certainty.

Training this model on seven labelled messages, with the code further down, gives w ≈ 1.34 and b ≈ −2.34. The trace below uses these rounded values, so its probabilities can differ from the exact ones by 0.01. Step through what the model says about new messages.

score = 1.34 * links - 2.34p = 1 / (1 + e^(-score))if p >= threshold: spamelse: not spam

The model gives a probability; the threshold turns it into a decision. Those are two separate choices. The model learns the first from data. You choose the second from what each kind of mistake costs. Overfitting and Evaluation shows how to measure that trade-off.

#Training a classifier

Squared error works poorly with a sigmoid: when the model is confidently wrong, its gradient is tiny, so learning crawls. Classifiers use a different loss instead, called log loss or cross-entropy. It punishes confident mistakes hard. Saying 0.99 for a message that turns out to be real costs about five times as much as saying 0.6.

The calculus behind log loss is beyond this lesson, but the result is pleasantly simple. The gradient for each weight is the average of (p − label) × input, where the label is 1 for spam and 0 otherwise. That is the same shape you met before: error times input.

function sigmoid(z) {
  return 1 / (1 + Math.exp(-z));
}
 
// Learn w and b for: P(spam) = sigmoid(w * links + b)
function trainSpam(links, spam, rate, steps) {
  let w = 0;
  let b = 0;
  for (let s = 0; s < steps; s++) {
    let gradW = 0;
    let gradB = 0;
    for (let i = 0; i < links.length; i++) {
      const p = sigmoid(w * links[i] + b);
      const error = p - spam[i];      // spam[i] is 1 or 0
      gradW += error * links[i];
      gradB += error;
    }
    w -= rate * gradW / links.length;
    b -= rate * gradB / links.length;
  }
  return { w, b };
}
 
trainSpam([0, 1, 1, 2, 3, 4, 5], [0, 0, 1, 0, 1, 1, 1], 0.5, 2000);
// { w: 1.34..., b: -2.34... }

import math
 
def sigmoid(z):
    return 1 / (1 + math.exp(-z))
 
def train_spam(links, spam, rate, steps):
    """Learn w and b for: P(spam) = sigmoid(w * links + b)."""
    w, b = 0.0, 0.0
    for _ in range(steps):
        grad_w = grad_b = 0.0
        for x, y in zip(links, spam):
            p = sigmoid(w * x + b)
            error = p - y            # y is 1 or 0
            grad_w += error * x
            grad_b += error
        w -= rate * grad_w / len(links)
        b -= rate * grad_b / len(links)
    return w, b
 
train_spam([0, 1, 1, 2, 3, 4, 5], [0, 0, 1, 0, 1, 1, 1], 0.5, 2000)
# (1.34..., -2.34...)

Look at the training data: one 1-link message is spam, and one 2-link message is not. No threshold on links can get every example right, so the model settles on probabilities that reflect the mix. A probability tells you how sure the model is, which a bare yes or no cannot.

#More than two categories

To sort support tickets into "billing", "bug" or "account", the model computes one score per category, each with its own weights. A function called softmax turns those scores into probabilities that add up to 1, and the highest one wins. Language models use the same step to turn their scores for every possible next token into probabilities, as you will see in Sampling and Temperature.

#Common mistakes

#Try it yourself

#Check yourself

#What to practise next

To code one gradient step for a line with a bias, try Delivery Time Nudge. A model that fits its training examples well is not automatically good. Overfitting and Evaluation shows how to test a model honestly and which numbers to look at beyond "percent correct".