Sampling and Temperature

Ask an assistant to suggest a name for your new bakery. Ask again in a fresh chat with the exact same words. You get a different name. Nothing broke, and the model did not change its mind. It rolled a weighted die, and the die landed somewhere else.

In What Is an LLM, the toy generator always took the most likely word. Most assistants do not. Here is how they pick instead, and what the settings called temperature and top-p do to the pick.

#Picking from the chances

Each model call gives every possible next token a raw score. Softmax turns the scores into chances that add up to 1, as in the attention lesson. Then one token is chosen.

Always taking the top token is called greedy decoding. It is predictable, but it tends to produce flat, repetitive text, and it can get stuck repeating the same phrase. So most systems sample: they pick at random, but weighted by the chances. A token with chance 0.46 is picked about 46% of the time.

The usual way to sample: draw a random number r between 0 and 1, then walk through the tokens adding up their chances. The first token whose running total passes r is the pick. Each token owns a slice of the line from 0 to 1, and the slice is as wide as its chance.

#Temperature: sharpen or flatten

Temperature is a single number that changes the chances before the pick. Divide every score by the temperature, then apply softmax as usual.

• Below 1, the gaps between scores get bigger, so the top token gets even more likely. The text is safer and more predictable.
• Exactly 1 leaves the model's chances as they are.
• Above 1, the gaps shrink, so unlikely tokens get a bigger share. The text is more varied, and more likely to go wrong.
• 0 is treated as greedy: always the top token. (You cannot divide by zero, so code handles it as a special case.)

Here is a model finishing "My favourite fruit is", with five candidate words.

scaled = each score / temperaturechances = softmax(scaled)r = random number in [0, 1)running = 0for each token: running += chance  if r < running: pick this token

#The code

The random number is passed in as r so you can test the function. In real use you would pass Math.random() or random.random().

function chancesAt(scores, temperature) {
  if (temperature === 0) { // greedy: all the chance on the top score
    const top = scores.indexOf(Math.max(...scores));
    return scores.map((_, i) => (i === top ? 1 : 0));
  }
  const scaled = scores.map((s) => s / temperature); // the only change
  const biggest = Math.max(...scaled);
  const exps = scaled.map((s) => Math.exp(s - biggest));
  const total = exps.reduce((a, b) => a + b, 0);
  return exps.map((e) => e / total);
}
 
// r is a random number in [0, 1).
function sample(chances, r) {
  let running = 0;
  for (let i = 0; i < chances.length; i++) {
    running += chances[i];
    if (r < running) return i;
  }
  return chances.length - 1; // guard against rounding
}
 
const words = ["apple", "mango", "banana", "kiwi", "durian"];
const chances = chancesAt([2, 1.5, 1, 0, -1], 1);
words[sample(chances, 0.62)]; // "mango"

import math
 
def chances_at(scores, temperature):
    if temperature == 0:  # greedy: all the chance on the top score
        top = scores.index(max(scores))
        return [1.0 if i == top else 0.0 for i in range(len(scores))]
    scaled = [s / temperature for s in scores]  # the only change
    biggest = max(scaled)
    exps = [math.exp(s - biggest) for s in scaled]
    total = sum(exps)
    return [e / total for e in exps]
 
# r is a random number in [0, 1).
def sample(chances, r):
    running = 0.0
    for i, c in enumerate(chances):
        running += c
        if r < running:
            return i
    return len(chances) - 1  # guard against rounding
 
words = ["apple", "mango", "banana", "kiwi", "durian"]
chances = chances_at([2, 1.5, 1, 0, -1], 1)
words[sample(chances, 0.62)]  # "mango"

The cost is tiny next to running the model: one pass over the vocabulary to scale and normalise, and one pass to sample, so O(V) for a vocabulary of V tokens. The top-p filter below adds a sort, O(V log V), which is still tiny.

#Top-p: cut off the long tail

A real vocabulary has tens of thousands of tokens or more. Most have tiny chances, but together that "long tail" adds up, and any one of them can be picked now and then, often derailing a sentence. Top-p sampling (also called nucleus sampling) cuts the tail off first.

Sort the tokens by chance and keep the most likely ones until their chances add up to at least p. Drop the rest, and rescale the kept ones so they sum to 1 again. With the fruit example at temperature 1 and p = 0.9: apple 0.46, plus mango 0.74, plus banana 0.91. That passes 0.9, so kiwi and durian are dropped, and the three survivors become about 0.51, 0.31 and 0.19.

function topP(chances, p) {
  const order = chances.map((_, i) => i).sort((a, b) => chances[b] - chances[a]);
  const kept = new Set();
  let running = 0;
  for (const i of order) { // keep the likeliest until they cover p
    kept.add(i);
    running += chances[i];
    if (running >= p) break;
  }
  return chances.map((c, i) => (kept.has(i) ? c / running : 0));
}

def top_p(chances, p):
    order = sorted(range(len(chances)), key=lambda i: -chances[i])
    kept, running = set(), 0.0
    for i in order:  # keep the likeliest until they cover p
        kept.add(i)
        running += chances[i]
        if running >= p:
            break
    return [c / running if i in kept else 0.0 for i, c in enumerate(chances)]

A simpler cousin, top-k, keeps a fixed number of tokens, say the top 40. Top-p adapts: when the model is sure, it may keep only one or two tokens; when many continuations are reasonable, it keeps more. Systems often combine temperature with one of these filters.

#Choosing settings

Low temperature suits tasks with one right answer: extracting fields from a form, classifying a ticket, writing code to a spec. Moderate temperature suits open-ended writing and brainstorming, where several different good answers are worth seeing. Very high temperatures rarely help: past a point, the extra variety is mostly mistakes. Many apps pick these values for you and do not expose them at all. Where you can set them, check the documentation: the allowed range and the default differ between providers, so the same number does not mean exactly the same thing everywhere.

#Common mistakes

#Try it yourself

#Check yourself

#What to practise next

Write the temperature step yourself in Storyteller Word Odds. Then continue with Why Models Hallucinate, which uses everything in this module to explain why a model can be fluent and wrong at the same time.