Chunking and Retrieval

You cut a cookbook into index cards so you can find recipes quickly. To save time, you cut every card at exactly 100 words. One card ends with "Bake at". The next starts with "180°C for 25 minutes, then let it cool." Later you search your cards for "how long does the lemon cake bake?" and the card with the answer never mentions lemon cake. It does not come up.

This is one of the most common reasons RAG systems give bad answers. The model is fine and the search is fine. The pieces were cut badly, so the right piece either does not exist or does not look like what it contains.

#The size trade-off

Every chunk becomes one vector, and that one vector has to stand for the whole chunk. That pulls in two directions.

• Too big, and a chunk covers several topics. Its vector becomes an average of all of them and matches none of them strongly. It also costs a lot of context when retrieved, mostly for text the question did not need.
• Too small, and a chunk loses what it is about. "Bake for 25 minutes" is useless without "lemon cake". Answers that span two sentences get split across chunks.

There is no single right size. A paragraph or a short section, a few hundred tokens, is a common starting point for prose. The better rule: a chunk should make sense on its own. If you handed it to someone with no other context, could they tell what it is about?

#Three ways to cut

1. Fixed size with overlap. Take a window of N words, move it forward by fewer than N, so neighbouring chunks share a few words. Simple, works on any text, and a sentence cut at one edge appears whole in the next chunk.
2. Follow the structure. Split on headings, then paragraphs, and only fall back to fixed size when a section is too long. An FAQ splits into one chunk per question and answer. Code splits by function. Tables stay whole.
3. Add the context back. Prefix each chunk with where it came from: "Lemon cake > Baking: …". Store metadata too: document title, section, date, product, version. The prefix helps the vector match; the metadata lets you filter.

Here is the first method on the recipe sentence: 15 words, windows of 6, overlap 2, so each window starts 4 words after the last.

step = size - overlapfor start = 0, step, 2·step, ...:  chunk = words[start : start + size]  if start + size >= length: stop

function chunkWords(text, size, overlap) {
  const words = text.split(/\s+/);
  const step = size - overlap; // must be at least 1
  const chunks = [];
  for (let start = 0; start < words.length; start += step) {
    chunks.push(words.slice(start, start + size).join(" "));
    if (start + size >= words.length) break;
  }
  return chunks;
}
 
chunkWords("Bake the lemon cake at 180°C for 25 minutes then let it cool before icing", 6, 2);
// 4 chunks, the second is "at 180°C for 25 minutes then"

def chunk_words(text, size, overlap):
    words = text.split()
    step = size - overlap  # must be at least 1
    chunks = []
    for start in range(0, len(words), step):
        chunks.append(" ".join(words[start:start + size]))
        if start + size >= len(words):
            break
    return chunks
 
chunk_words("Bake the lemon cake at 180°C for 25 minutes then let it cool before icing", 6, 2)
# 4 chunks, the second is "at 180°C for 25 minutes then"

Chunking is one pass, O(n) for n words. Overlap has a storage cost: you get about n ÷ (size − overlap) chunks, so overlap 2 on size 6 makes about 1.5 times as many chunks as no overlap. Real systems count tokens instead of words, and use much larger windows.

#Retrieving well

Plain "top k by cosine similarity" is a fine start. Each of these additions fixes a common way it goes wrong.

• Metadata filters. If the user asks about the kettle, only search kettle chunks. Filters are exact and cheap, and they stop near-identical text about a different product from winning.
• A minimum score. Top k always returns k chunks, even when none are relevant. A threshold lets retrieval say "nothing found" so the model can say so too.
• Hybrid search. Embeddings are weak at exact strings like error codes, part numbers and names. Run a keyword search alongside the vector search and merge the results.
• Reranking. Retrieve a generous 20 or 50 candidates cheaply, then have a slower, more careful model score each one against the question and keep the best few.
• Query rewriting. In a chat, "and for the blue one?" means nothing on its own. Rewrite it into a full question using the conversation before searching.

Here are the first two in code, building on cosine from RAG Basics. Vectors use three toy dimensions: cleaning, indicator lights, filling.

const dot = (a, b) => a.reduce((s, x, i) => s + x * b[i], 0);
const cosine = (a, b) => dot(a, b) / Math.sqrt(dot(a, a) * dot(b, b));
 
function retrieve(queryVec, chunks, { product, k, minScore }) {
  return chunks
    .filter((c) => c.product === product)
    .map((c) => ({ text: c.text, score: cosine(queryVec, c.vec) }))
    .filter((c) => c.score >= minScore)
    .sort((a, b) => b.score - a.score)
    .slice(0, k);
}
 
const manual = [
  { product: "kettle", text: "Kettle > Care: descale monthly with vinegar.", vec: [5, 0, 1] },
  { product: "kettle", text: "Kettle > Lights: red means boiling.", vec: [1, 4, 0] },
  { product: "toaster", text: "Toaster > Care: empty the crumb tray weekly.", vec: [4, 1, 1] },
  { product: "kettle", text: "Kettle > Filling: never fill above MAX.", vec: [1, 1, 4] },
];
// "How do I clean the kettle?" embeds to [5, 1, 0]
retrieve([5, 1, 0], manual, { product: "kettle", k: 2, minScore: 0.5 });
// [{ text: "Kettle > Care: ...", score: 0.96 }]

import math
 
def dot(a, b):
    return sum(x * y for x, y in zip(a, b))
 
def cosine(a, b):
    return dot(a, b) / math.sqrt(dot(a, a) * dot(b, b))
 
def retrieve(query_vec, chunks, product, k, min_score):
    scored = [
        {"text": c["text"], "score": cosine(query_vec, c["vec"])}
        for c in chunks if c["product"] == product
    ]
    scored = [c for c in scored if c["score"] >= min_score]
    scored.sort(key=lambda c: c["score"], reverse=True)
    return scored[:k]
 
manual = [
    {"product": "kettle", "text": "Kettle > Care: descale monthly with vinegar.", "vec": [5, 0, 1]},
    {"product": "kettle", "text": "Kettle > Lights: red means boiling.", "vec": [1, 4, 0]},
    {"product": "toaster", "text": "Toaster > Care: empty the crumb tray weekly.", "vec": [4, 1, 1]},
    {"product": "kettle", "text": "Kettle > Filling: never fill above MAX.", "vec": [1, 1, 4]},
]
# "How do I clean the kettle?" embeds to [5, 1, 0]
retrieve([5, 1, 0], manual, "kettle", k=2, min_score=0.5)
# [{'text': 'Kettle > Care: ...', 'score': 0.96}]

Work through it. Without the filter, the toaster's care chunk would win with 0.97, because cleaning is cleaning. The filter removes it. Of the kettle chunks, care scores 0.96, lights 0.43 and filling 0.28. The threshold removes the last two, so you get one chunk even though k is 2. Sending the lights chunk just to fill a slot would only add noise.

#Measure it

You cannot tune chunking by reading a few answers. Write 20 to 50 real questions, and for each one note which chunk or section holds the answer. Then run retrieval alone and count how often the right chunk lands in the top k. When you change chunk size, overlap or filters, run the list again. A number that goes up is far more trustworthy than a feeling that answers seem better. Evals builds this into a habit.

#Common mistakes

#Try it yourself

#Check yourself

#What to practise next

Retrieval brings in knowledge from documents. Memory applies the same ideas to what the model learns about a user or project over many conversations.