The Context Window

A meeting room has one whiteboard. Through the morning people fill it with notes, diagrams and decisions. By noon it is full, and the next person who needs space picks up the eraser and wipes the oldest corner. Nobody checks whether that corner held the one decision everybody agreed on at 9 a.m.

A model's context has the same problem. In What Is Context you saw that the app resends the whole conversation on every call. That cannot go on forever, because the model can only read so much text at once. That limit is the context window.

#A fixed budget, counted in tokens

The window is a maximum number of tokens per call. Three things about it trip people up.

• Everything counts. The instructions, the tool descriptions, pasted documents, every earlier message and your new one all come out of the same budget. Users only see their own messages, so they underestimate how full it is.
• The reply counts too. The model writes its answer token by token into the same window. If the input fills it, there is no room left to answer, and the reply gets cut off.
• The size depends on the model. Windows differ between models and have grown over time. Check the documentation for the one you use rather than trusting a number you remember.

#What happens when it fills up

When a conversation outgrows the window, the app has to do something. There are three common choices.

1. Refuse. Return an error and ask you to start a new chat or shorten your input. Honest, but annoying.
2. Drop the oldest messages. Keep the instructions, keep the newest turns, and quietly leave out whatever does not fit. This is simple and widely used.
3. Summarize. Replace old turns with a short summary. Better, but a summary can lose details. Memory covers this.

The second choice is the dangerous one, because it is silent. Watch it on a cooking chat. To keep the numbers small, imagine a toy window of 120 tokens with 20 reserved for the reply, so the input may use 100.

used = system + new messagefor each old message, newest first:  if used + its tokens > budget: stop  keep it, used += its tokenssend system + kept + new message

The most important fact in the chat was said first, and "oldest first out" threw it away first. Long-lived facts like allergies, names and hard rules should not live only in the scrolling history. Put them in the instructions or a pinned note that is sent every time.

#The trimming in code

// history is oldest first. Each item has a token count.
function fitToWindow(system, history, newMessage, budget) {
  let used = system.tokens + newMessage.tokens;
  const kept = [];
  for (let i = history.length - 1; i >= 0; i--) { // newest first
    if (used + history[i].tokens > budget) break;
    kept.push(history[i]);
    used += history[i].tokens;
  }
  kept.reverse(); // back to oldest first
  return { request: [system, ...kept, newMessage], used };
}
 
const history = [15, 10, 10, 35, 10, 25].map((tokens, i) => ({ id: i, tokens }));
const result = fitToWindow({ tokens: 20 }, history, { tokens: 10 }, 100);
console.log(result.used, result.request.length); // 100 5

# history is oldest first. Each item has a token count.
def fit_to_window(system, history, new_message, budget):
    used = system["tokens"] + new_message["tokens"]
    kept = []
    for msg in reversed(history):  # newest first
        if used + msg["tokens"] > budget:
            break
        kept.append(msg)
        used += msg["tokens"]
    kept.reverse()  # back to oldest first
    return {"request": [system, *kept, new_message], "used": used}
 
history = [{"id": i, "tokens": t} for i, t in enumerate([15, 10, 10, 35, 10, 25])]
result = fit_to_window({"tokens": 20}, history, {"tokens": 10}, 100)
print(result["used"], len(result["request"]))  # 100 5

The loop uses break, not "skip and keep looking". Skipping a big message and keeping an older small one would leave a hole in the middle of the conversation, and the model would read replies to questions it never sees.

#Fitting is not the same as reading well

A bigger window means more fits. It does not mean the model uses all of it equally well. Three effects show up as the context grows.

• Cost and speed. The whole context is sent and read on every call. A longer context usually costs more and takes longer, turn after turn.
• Things in the middle get less attention. Models tend to use information near the start and the end of a long context more reliably than information buried in the middle. A key line on page 40 of 80 can be easier to miss than the same line at the top.
• Noise distracts. Irrelevant text is not neutral. A model asked about one product, with ten similar product sheets in its context, can mix up details between them.

#Common mistakes

#Try it yourself

#Check yourself

#What to practise next

You now know the budget. Context Engineering is about spending it well: choosing, shrinking and ordering what goes in. Long chats that slowly get worse are covered in Context Rot.