Embeddings

A customer types "warm jacket" into a clothing shop's search box. The shop sells an "insulated winter coat", which is exactly what they want. A keyword search misses the coat, because the two phrases do not share a single word. Meanwhile it returns a "jacket potato" recipe card from the café section.

The fix is to stop comparing letters and start comparing meanings. To do that, a computer needs meaning in a form it can calculate with: numbers. That is what an embedding is.

#Meaning as a position

An embedding is a list of numbers that represents a piece of content, chosen so that things with similar meanings get similar lists. A list of numbers like this is also called a vector. You can think of each one as a point in space. "Insulated winter coat" and "warm jacket" land near each other. "Linen shirt" lands somewhere else, and "jacket potato" far away.

Real embeddings have hundreds or thousands of numbers per item, not two. You cannot picture that many dimensions, but the arithmetic is the same as in two or three, so small examples are enough to learn the method.

#Where the numbers come from

Nobody types embeddings in by hand. They come out of a neural network trained on a task that forces similar things to look similar inside it. Two common tasks:

• Predict the context. Train a network to guess a word from the words around it. Words that appear in similar sentences, like "coat" and "jacket", must produce similar hidden values to make similar guesses.
• Match pairs. Show the network pairs that belong together, like a question and its answer, or a search and the product people clicked. Train it so matching pairs get close vectors and random pairs get distant ones.

After training, you keep the network and use one of its layers as the embedding: feed in any text and read off the list of numbers.

#Measuring closeness

To use embeddings, you need one number that says how similar two lists are. The standard choice is cosine similarity. It looks at the direction the two vectors point in, ignoring how long they are. It is 1 when they point the same way, 0 when they are at right angles (in embedding terms, unrelated), and −1 when they point in opposite directions.

It takes two ingredients, both school algebra:

• The dot product: multiply matching positions and add. For [4, 1, 1] and [5, 2, 2] it is 4×5 + 1×2 + 1×2 = 24.
• The length of a vector: Pythagoras in more dimensions. For [4, 1, 1] it is √(16 + 1 + 1) = √18 ≈ 4.24.

cosine(a, b) = dot(a, b) / (length(a) × length(b))

Dividing by the lengths is what makes it about direction only. A long document and a short one about the same thing still score as similar.

#Trace a search

To keep the numbers small, give each item a hand-made 3-number embedding: how warm it is, how dressy, how waterproof. The query "warm jacket" becomes [4, 1, 1]. Real embedding dimensions do not have tidy names like these, but the search works the same way.

qlen = length(query)for each item:  score = dot(query, item) / (qlen * length(item))sort items by score, highest first

function dot(a, b) {
  let total = 0;
  for (let i = 0; i < a.length; i++) total += a[i] * b[i];
  return total;
}
 
// Cosine similarity: 1 means same direction, 0 means unrelated.
function cosine(a, b) {
  return dot(a, b) / (Math.sqrt(dot(a, a)) * Math.sqrt(dot(b, b)));
}
 
// Item names with their scores, most similar first.
function search(query, items) {
  return Object.entries(items)
    .map(([name, vec]) => [name, cosine(query, vec)])
    .sort((x, y) => y[1] - x[1]);
}
 
search([4, 1, 1], {
  "insulated winter coat": [5, 2, 2],
  "wool overcoat": [4, 4, 0],
  "rain shell": [1, 1, 5],
  "linen shirt": [0, 3, 0],
}); // coat 0.985, overcoat 0.833, shell 0.454, shirt 0.236

import math
 
def dot(a, b):
    return sum(x * y for x, y in zip(a, b))
 
def cosine(a, b):
    """Cosine similarity: 1 means same direction, 0 means unrelated."""
    return dot(a, b) / (math.sqrt(dot(a, a)) * math.sqrt(dot(b, b)))
 
def search(query, items):
    """Item names with their scores, most similar first."""
    scored = [(name, cosine(query, vec)) for name, vec in items.items()]
    return sorted(scored, key=lambda pair: pair[1], reverse=True)
 
search([4, 1, 1], {
    "insulated winter coat": [5, 2, 2],
    "wool overcoat": [4, 4, 0],
    "rain shell": [1, 1, 5],
    "linen shirt": [0, 3, 0],
})  # coat 0.985, overcoat 0.833, shell 0.454, shirt 0.236

#What embeddings are used for

• Search by meaning. Embed every document once. At query time, embed the query and return the nearest documents, as in the trace.
• Recommendations. "People who liked this also liked" becomes "items near this one".
• Grouping and duplicates. Near-identical support tickets or product listings sit almost on top of each other.
• Feeding language models. Language models turn every token into an embedding before doing anything else. And retrieval systems use embedding search to find the passages a model should read before answering, covered in RAG Basics.

#Common mistakes

#Try it yourself

#Check yourself

#What to practise next

To code cosine similarity yourself, try Bookshop Blurb Match. Next, see how these pieces combine into a language model in What Is an LLM, or jump to RAG Basics to put embedding search to work.