Tokens

Ask an AI assistant how many times the letter "r" appears in a long, unusual word, and it may get it wrong while explaining its answer in perfect English. Ask it to write a poem about the same word and it does fine. The model is not careless about letters. It mostly never sees them.

In What Is an LLM you saw that a model predicts the next piece of text, again and again. Those pieces are called tokens, and they are not letters. Understanding them explains a lot: why limits and prices are counted in tokens, why some languages cost more than others, and why counting letters is hard for a system that writes so well.

#Text becomes a list of numbers

A model works only with numbers. Before it reads anything, a program called a tokenizer cuts the text into pieces from a fixed list, the vocabulary, and replaces each piece with its position in that list. The model sees only those numbers, called token IDs.

Three things to notice. Common short words like "tired" are a single token. A long or rare word is split into chunks. And the space usually belongs to the token after it (shown as ␣), so " tired" and "tired" are different tokens with different IDs.

Single letters would make every sentence dozens of tokens long, and the model would need many more steps to read and write anything. Whole words would need a vocabulary with every word, name, typo and number ever written. Pieces in between avoid both problems: frequent text takes few tokens, and anything at all can still be spelled out from smaller pieces.

#Where the pieces come from

Nobody picks the vocabulary by hand. A common method, called byte-pair encoding, learns it from a sample of text with a very simple rule:

1. Start with every word split into single characters.
2. Count every pair of neighbouring pieces across all the text.
3. Glue the most frequent pair into one new piece, and add it to the vocabulary.
4. Repeat until the vocabulary is as big as you want.

Here it is on a tiny sample of six words: tea, tea, team, teams, steam, seat.

split every word into lettersrepeat k times:  count each pair of neighbouring pieces  best = the most frequent pair  glue best together everywhere  add best to the vocabulary

Real tokenizers run the same idea on billions of words, usually on raw bytes rather than letters so that any text in any script can be handled, and stop after tens of thousands of merges, often more. To tokenize new text later, they apply the learned merges in the same order.

#The code

function learnMerges(words, numMerges) {
  let pieces = words.map((w) => w.split("")); // start from single letters
  const merges = [];
  for (let m = 0; m < numMerges; m++) {
    const counts = new Map(); // count every pair of neighbouring pieces
    for (const word of pieces) {
      for (let i = 0; i + 1 < word.length; i++) {
        const pair = word[i] + "|" + word[i + 1];
        counts.set(pair, (counts.get(pair) || 0) + 1);
      }
    }
    let best = null; // most frequent pair; the first one seen wins a tie
    for (const [pair, c] of counts) if (best === null || c > counts.get(best)) best = pair;
    if (best === null) break; // every word is already one piece
    const [a, b] = best.split("|");
    merges.push(a + b);
    pieces = pieces.map((word) => {
      const out = [];
      for (let i = 0; i < word.length; i++) {
        if (word[i] === a && word[i + 1] === b) { out.push(a + b); i++; }
        else out.push(word[i]);
      }
      return out;
    });
  }
  return { merges, pieces };
}
 
learnMerges(["tea", "tea", "team", "teams", "steam", "seat"], 3).merges;
// ["ea", "tea", "team"]

def learn_merges(words, num_merges):
    pieces = [list(w) for w in words]  # start from single letters
    merges = []
    for _ in range(num_merges):
        counts = {}  # count every pair of neighbouring pieces
        for word in pieces:
            for a, b in zip(word, word[1:]):
                counts[(a, b)] = counts.get((a, b), 0) + 1
        if not counts:
            break  # every word is already one piece
        best = max(counts, key=counts.get)  # the first one seen wins a tie
        merges.append(best[0] + best[1])
        new_pieces = []
        for word in pieces:
            out, i = [], 0
            while i < len(word):
                if i + 1 < len(word) and (word[i], word[i + 1]) == best:
                    out.append(word[i] + word[i + 1])
                    i += 2
                else:
                    out.append(word[i])
                    i += 1
            new_pieces.append(out)
        pieces = new_pieces
    return merges, pieces
 
learn_merges(["tea", "tea", "team", "teams", "steam", "seat"], 3)[0]
# ['ea', 'tea', 'team']

Each merge scans all the text once, so learning k merges over n characters is O(k · n) in this simple version. Production tokenizers use smarter bookkeeping, but the result is the same kind of vocabulary.

#Why tokens matter to you

Limits and prices are in tokens. How much text a model can read at once, and what a request costs, are measured in tokens, not words or characters. For ordinary English prose, expect somewhat more tokens than words. The exact ratio depends on the tokenizer.

Not all text is equal. A tokenizer learns its merges from its training sample. Text that was common there, like English prose and popular programming languages, packs into few tokens. Languages that were rarer, unusual names, long numbers and random strings like IDs or hashes break into many small pieces. The same message can cost several times more tokens in one language than another.

Letters are hidden. The model receives the ID for "␣tired", not t-i-r-e-d. It can often still spell words, because it has seen many spellings in its training text, but counting or reversing letters asks it to recover something it was never directly shown. Numbers have a similar problem. Depending on the tokenizer, "2847" might be one token, two chunks like "284" and "7", or four single digits. When the chunks do not line up with place value, the model cannot line up units with units and tens with tens, which is part of why arithmetic on long numbers is unreliable without a calculator.

After this step, each token ID is turned into a list of numbers, a vector, that the model learns during training. That is the embedding idea from the previous module, applied to tokens.

#Common mistakes

#Try it yourself

#Check yourself

#What to practise next

Build a tokenizer yourself in Knitting Pattern Shorthand (learning merges) and Sign Shop Letter Blocks (cutting text into known pieces). Then continue with Attention and Transformers, which shows how the model combines the token vectors so that each one can use information from the others.