Training and Fine-Tuning

Imagine a new hire who has read every document in the company archive: manuals, emails, old reports, the lot. They can tell you how almost anything works. But put them on the help desk, and when a customer asks a question they reply with three more questions. In the archive, a question was usually followed by more questions in a FAQ list, not by an answer, so that is what they produce.

That is roughly what a language model is like after its first and biggest stage of training. It has absorbed a vast amount of text, but it has not yet learned to act like an assistant. Getting from one to the other takes more training of a different kind.

#Stage 1: pretraining on raw text

The first lesson showed that a model outputs chances for the next token. Pretraining teaches it to give good chances. The recipe needs no human labels at all:

1. Take a piece of real text, for example "The cat slept on the mat".
2. Hide the next token, and ask the model for its chances given the text before it.
3. Measure how surprised the model was by the real next token.
4. Nudge the model's numbers so it would be a little less surprised next time.

Every position in every document is a free practice question. That is why pretraining can use so much text: books, websites, code, articles, in many languages.

Surprise has a precise form, called the loss: minus the natural log of the chance the model gave to the real token. If the model gave the right token 0.9, the loss is about 0.11, barely surprised. If it gave 0.01, the loss is about 4.6, very surprised. Training means making the average loss go down.

#Watch one example being learned

Below, a model must predict the last word of "The cat slept on the ___". It considers three words, and the real answer is "mat". Each word has a raw score, and softmax turns scores into chances, as in the attention lesson.

The nudge has a neat form: for every word, subtract rate × (chance − target) from its score, where rate is the learning rate and the target is 1 for the real word and 0 for the others. The real word's score goes up, the others go down, and the further off a chance is, the bigger its nudge. This is gradient descent, as in How Models Learn. Here we nudge the three scores directly; a real model nudges the billions of parameters that produced them.

chances = softmax(scores)loss = -ln(chances[real word])for each word i:  target = 1 if i is the real word else 0  scores[i] -= rate * (chances[i] - target)

That last caption is the important one. Training does not teach the model to recite one text. It pulls the chances toward what is common across millions of similar texts. Mostly, the model ends up with good guesses rather than stored documents. The exception is text repeated many times in training, like famous quotes or common licence text, which it can memorise word for word.

#The code

function softmax(scores) {
  const biggest = Math.max(...scores);
  const exps = scores.map((s) => Math.exp(s - biggest));
  const total = exps.reduce((a, b) => a + b, 0);
  return exps.map((e) => e / total);
}
 
// One training step on one example. Returns the new scores and the loss.
function trainStep(scores, correct, rate) {
  const chances = softmax(scores);
  const loss = -Math.log(chances[correct]); // surprise at the real word
  const newScores = scores.map((s, i) => {
    const target = i === correct ? 1 : 0;
    return s - rate * (chances[i] - target); // the nudge
  });
  return { newScores, loss };
}
 
let scores = [0, 1, -1]; // mat, sofa, moon
for (let step = 0; step < 4; step++) {
  const { newScores, loss } = trainStep(scores, 0, 1);
  console.log(loss.toFixed(2)); // 1.41, 0.60, 0.33, 0.22
  scores = newScores;
}

import math
 
def softmax(scores):
    biggest = max(scores)
    exps = [math.exp(s - biggest) for s in scores]
    total = sum(exps)
    return [e / total for e in exps]
 
# One training step on one example. Returns the new scores and the loss.
def train_step(scores, correct, rate):
    chances = softmax(scores)
    loss = -math.log(chances[correct])  # surprise at the real word
    new_scores = []
    for i, (s, c) in enumerate(zip(scores, chances)):
        target = 1 if i == correct else 0
        new_scores.append(s - rate * (c - target))  # the nudge
    return new_scores, loss
 
scores = [0, 1, -1]  # mat, sofa, moon
for step in range(4):
    scores, loss = train_step(scores, 0, 1)
    print(f"{loss:.2f}")  # 1.41, 0.60, 0.33, 0.22

Real pretraining runs this over trillions of tokens, in batches of many examples at a time, on large clusters of specialised chips for weeks or months. That cost is why only a few organisations pretrain large models from scratch, and why everyone else starts from an existing one.

#Stages 2 and 3: becoming an assistant

A pretrained, or base, model continues text. Ask it a question and it may continue with the kind of text that often follows a question on the web.

Two more stages fix this. Both are much smaller than pretraining.

Instruction tuning continues training with the same next-token method, but on a much smaller set of carefully written conversations: a request, then a good reply. The model learns the pattern "after a user turn comes a helpful assistant turn". Knowledge mostly comes from pretraining; this stage mainly teaches format and behaviour.

Preference tuning goes further. The model writes several answers to the same request, and they are compared: by people, by a separate model trained on people's judgements, or by automatic checks where the answer can be verified, like code that must pass tests or a maths answer that can be checked. Training then makes the preferred kinds of answers more likely. This is where much of the tone, caution and step-by-step reasoning of an assistant comes from.

You may also hear about fine-tuning as something a company does to an existing model: training it further on its own examples, for example to always produce a house format or to handle a narrow task well. It is the same idea, at a smaller scale.

#Common mistakes

#Try it yourself

#Check yourself

#What to practise next

Training gives the model good chances for the next token. Sampling and Temperature covers the last step: how one token is actually chosen from those chances.