Overfitting and Evaluation

A student gets hold of last year's exam and memorizes every answer. On a practice run with those same questions, they score 100%. On the real exam, with new questions on the same topics, they fail. They learned the answers, not the subject.

Models do the same thing, and it is one of the most common ways machine learning goes wrong. A model that scores brilliantly on the examples it trained on can be useless on the next one. You need a way to catch that before your users do.

#Memorizing versus learning

Go back to the delivery data from What Is Machine Learning. The rule "late if at least 6 km" got 6 of 7 past deliveries right. The odd one out was a 10 km trip that arrived on time.

Now consider a fussier rule: "late if 6 to 9 km, or 11 km and over". It gets all 7 right. It is also nonsense. It says a 10 km delivery will be on time while 9 km and 11 km ones will be late, all because of one lucky day. The fussier rule fits the training data better and will predict new deliveries worse.

This is overfitting: the model has learned the noise in its examples along with the pattern. The opposite problem, underfitting, is a model too simple to capture the pattern at all, like predicting "on time" for every delivery.

#Test on data the model has never seen

The fix is simple and strict. Before training, split your labelled examples into separate piles:

• Training set (often around 70 to 80%). The model learns from these.
• Validation set. You use these to compare settings: which features, how flexible a model, which learning rate.
• Test set. Locked away until the very end. You score the final model on it once, and that score is your honest estimate of real-world performance.

Two piles are not enough. Every time you look at a score and change something, you are fitting a little to that data. Try twenty settings on the validation set and keep the best, and that best score is a bit flattering. The test set has not been part of any choice, so its score is not.

When data is scarce, cross-validation makes better use of it: split the training data into, say, 5 parts, train 5 times, each time holding out a different part for scoring, and average the 5 scores.

#Accuracy can lie

With a fair test set in hand, the obvious number to measure is accuracy: the share of predictions that were right. It hides a trap.

Picture a fraud detector checking 10 card payments, of which 3 are fraud. A lazy model that always says "fine" is right 7 times out of 10: 70% accuracy, while catching zero fraud. In real data, fraud is far rarer than 3 in 10, so the lazy model can score 99.9%.

To see what a classifier really does, sort every prediction into one of four boxes. This table is called a confusion matrix:

• True positive: fraud, and flagged. Good.
• False negative: fraud, but not flagged. A missed case.
• False positive: fine, but flagged. A false alarm.
• True negative: fine, and not flagged. Good.

#Precision and recall

Two questions matter for almost every classifier, and each has its own number.

• Precision: when the model raises a flag, how often is it right? It is true positives ÷ everything flagged. Low precision means many false alarms.
• Recall: of all the real cases, how many did the model catch? It is true positives ÷ all real positives. Low recall means many misses.

They pull against each other. In Regression and Classification you saw that raising the threshold flags fewer messages. That usually raises precision (the flags you do raise are surer) and lowers recall (you miss more). Which matters more depends on the cost of each mistake. A cancer screening wants high recall. A system that blocks customer accounts automatically wants high precision.

// actual and predicted are lists of 1 (fraud) and 0 (fine).
function evaluate(actual, predicted) {
  let tp = 0, fp = 0, fn = 0, tn = 0;
  for (let i = 0; i < actual.length; i++) {
    if (predicted[i] === 1 && actual[i] === 1) tp++;
    else if (predicted[i] === 1) fp++; // flagged a fine payment
    else if (actual[i] === 1) fn++;    // missed a fraud
    else tn++;
  }
  return {
    accuracy: (tp + tn) / actual.length,
    precision: tp + fp ? tp / (tp + fp) : 0, // of the flags, how many were right
    recall: tp + fn ? tp / (tp + fn) : 0,    // of the frauds, how many we caught
  };
}
 
evaluate([1, 0, 0, 1, 0, 0, 0, 1, 0, 0],
         [1, 0, 1, 0, 0, 0, 0, 1, 0, 0]);
// { accuracy: 0.8, precision: 0.666..., recall: 0.666... }

def evaluate(actual, predicted):
    """actual and predicted are lists of 1 (fraud) and 0 (fine)."""
    tp = fp = fn = tn = 0
    for a, p in zip(actual, predicted):
        if p == 1 and a == 1:
            tp += 1
        elif p == 1:
            fp += 1   # flagged a fine payment
        elif a == 1:
            fn += 1   # missed a fraud
        else:
            tn += 1
    return {
        "accuracy": (tp + tn) / len(actual),
        "precision": tp / (tp + fp) if tp + fp else 0,
        "recall": tp / (tp + fn) if tp + fn else 0,
    }
 
evaluate([1, 0, 0, 1, 0, 0, 0, 1, 0, 0],
         [1, 0, 1, 0, 0, 0, 0, 1, 0, 0])
# {'accuracy': 0.8, 'precision': 0.666..., 'recall': 0.666...}

For regression, the usual test number is the average error on the test set, for example the mean squared error from How Models Learn, or the average absolute error, which is easier to explain: "off by 40 a month on average".

#Common mistakes

#Try it yourself

#Check yourself

#What to practise next

To code the confusion matrix and these scores yourself, try Bruised Apple Detector. The models so far were weighted sums. Decision Trees and Ensembles learns a different kind of rule, a chain of yes-or-no questions, and shows how combining many trees fights overfitting.