Design a RAG System

An engineer asks the company's internal assistant how to rotate the staging database password. Three seconds later they get four clear steps, each with a link to the runbook page it came from. Behind that answer, a system found 8 relevant passages among 50 million, checked the engineer was allowed to read every one, and gave them to a language model with instructions to answer only from them.

This is retrieval-augmented generation, covered as an idea in RAG basics and chunking and retrieval. Interviewers ask it as a system design question because the model call is the easy part. The hard parts are keeping millions of documents indexed and fresh, finding the right passages fast, never leaking a document to someone who cannot open it, and knowing whether the answers are any good. The steps follow A Framework for Any Design.

#Clarify the problem

• Functional: ask a question in plain language and get an answer with citations; answer only from documents the asker may read; say when the documents do not contain the answer; ingest from the wiki, tickets and drives as they change; collect feedback on answers.
• Non-functional: zero permission leaks; first words within 2 seconds, full answer within about 10; edits searchable within 10 minutes, and removed access effective within 5; cost per question measured and bounded; available during working hours in every region.
• Out of scope: taking actions, training or fine-tuning models, public customers.

#Estimate the scale

• Chunks: 10 million documents cut into about 5 passages each is 50 million chunks.
• Questions: 20,000 people × 10 questions a day = 200,000 a day. Spread over 8 working hours, 200,000 ÷ 28,800 s ≈ 7 a second; with a 3x peak, about 20 a second. A small rate, but each request is expensive.
• Model input: 8 chunks of about 500 tokens is 4,000 tokens of context; with instructions and a 300-token answer, about 4,500 tokens per question. 200,000 × 4,500 = 900 million tokens a day. This number, not the request rate, drives cost.
• Changes: if 1% of documents change each day, that is 100,000 documents and about 500,000 chunks to re-embed, about 6 a second on average.

#Define the API

POST /ask                       Authorization: the employee's identity
  body: { "question": "How do I rotate the staging DB password?", "conversation_id": "c_19"? }
  200  text/event-stream
       event: token     data: "1. Open the "
       ...
       event: done      data: { "answer_id": "a_77", "citations": [
                          { "doc_id": "wiki-4410", "title": "Staging runbook", "url": "..." } ] }
  429  too many questions from this user
  503  model unavailable; body holds the top documents as plain search results
 
POST /answers/{id}/feedback     body: { "helpful": false, "comment": "outdated" }
 
POST /ingest/events             internal, from connectors
  body: { "source": "wiki", "doc_id": "wiki-4410", "change": "updated" | "deleted" | "acl" }
  202 Accepted

The decision to argue is streaming. A full answer takes several seconds to generate, but the first words can arrive in about one. Streaming over server-sent events makes the 2-second target reachable. Citations come at the end because the model decides which sources it used as it writes. The 503 shape matters too: when the model is down, ranked documents are still a useful answer.

#Model the data

documents                         -- source of truth for what is indexed
  doc_id          VARCHAR(64) PRIMARY KEY     -- "wiki-4410"
  source          VARCHAR(16)
  title, url      TEXT
  allowed_groups  TEXT[]          -- copied from the source system
  content_hash    CHAR(64)
  updated_at      TIMESTAMP
 
chunks                            -- lives in the search index
  chunk_id        VARCHAR(80)     -- "wiki-4410#3"
  doc_id          VARCHAR(64)
  text            TEXT            -- about 500 tokens, about 2 KB
  embedding       VECTOR(1024)
  allowed_groups  TEXT[]          -- filter field
  text_hash       CHAR(64)        -- skip re-embedding unchanged chunks
  index_version   INT             -- which chunker and embedding model made it

The hot query is "nearest chunks to this vector, and best keyword matches for these words, among chunks this person may read". That needs a search index that supports vector search, keyword search and filters together: a search engine with vector support, or a vector database with keyword support. At smaller scale, a relational database with a vector extension is a fine start. The document table is ordinary relational data that tells ingestion what is indexed. Vector search here is approximate: it trades a little recall for speed, so measure recall rather than assume it.

#High-level design

Both paths use the same embedding model, because vectors from different models cannot be compared. Step through one question, then an edit and a permission change.

#Deep dives

#The ingestion pipeline

Connectors subscribe to each source's change events and also crawl on a schedule, so a missed event is caught by the next crawl. Each change becomes a message on a queue, and workers process documents independently:

1. Fetch and extract the text, including from PDFs and slides. Extraction errors are the most common silent failure, so count them per source.
2. Chunk along the document's own structure, headings first, at about 500 tokens with a little overlap, and put the title and section heading at the top of each chunk so it makes sense alone.
3. Hash each chunk and embed only the ones whose hash changed, in batches. Most edits touch a few chunks.
4. Upsert and delete in the index: write new and changed chunks, and remove chunks that no longer exist, by doc_id.

Changing the embedding model or the chunker is different: old and new vectors cannot be compared. Build a new index alongside the old one, with a new index_version, and switch queries when it is complete. If the embedding service handles 1,000 chunks a second (an assumption; measure yours), backfilling 50 million chunks takes 50,000 seconds, about 14 hours. Meanwhile, keep writing live changes to both indexes.

#Retrieval quality versus latency

Vector search finds passages with the same meaning in different words. Keyword search finds exact terms: error codes, product names, "staging". Questions inside a company are full of exact terms, so use both. A common way to combine the lists is reciprocal rank fusion: each list gives a chunk 1 ÷ (60 + rank) points, and the totals are sorted.

Chunks found by both methods rise to the top, even when neither ranked them first. Then a reranker, a model that reads the question and one chunk together, scores the top candidates more accurately than either search can, at the cost of one model pass per candidate.

I would pick hybrid plus reranker: 200 ms is small next to a second of generation, and better chunks let you send 8 instead of 20, which cuts tokens. What would change my mind: a strict latency target with no generation step, where hybrid alone is enough.

#Permissions

The model uses whatever text it is given, and a prompt instruction is not a security boundary. So restricted text must never reach the prompt. Filter in the retrieval step, using permissions copied from the source onto every chunk.

• Filter inside the search, not after it. Taking the top 50 and then dropping what the user cannot read can leave nothing. Someone in a small team may find 48 of the top 50 hidden. Search engines with filtered vector search apply the filter while searching.
• Sync permission changes fast and separately. An ACL change only updates the allowed_groups field, so it is cheap. Run it more often than content sync: removed access within 5 minutes.
• Read the asker's groups fresh. Access is also removed when a person leaves a group. Look groups up in the directory, cached for a minute, not in a login token that lives for hours.
• Check again at answer time for sensitive sources. For HR or legal spaces, ask the source system whether the user can still open each cited document before returning it. It costs a call per document and closes the window between a permission change and the sync.

Retrieved documents are also untrusted input: a page can contain text written to steer the model. Treat chunk text as data in the prompt, and see prompt injection for defences.

#Caching and cost

Cost is the 900 million tokens a day. The levers, in order:

• Send fewer, better chunks. A reranker that lets you send 6 chunks instead of 8 cuts the context from 4,000 to 3,000 tokens per question.
• Cache answers to repeated questions, keyed by the normalized question and the user's group set. Leaving groups out of the key leaks answers between users with different access. Store which documents each answer cites, and delete those entries when any of them gets a content or ACL event; otherwise a cached answer keeps quoting a page after it was edited or restricted. Free-form questions repeat less than you hope, so measure the hit rate before counting on it.
• Keep the prompt's fixed part at the start. Some model servers can reuse work for a repeated prompt prefix; stable instructions first make that possible.
• Route easy questions to a smaller, cheaper model, checked with the evaluation set below.

Be careful with a "semantic" cache that returns an answer for a similar question. "Parental leave in France" and "parental leave in Germany" are similar and have different answers.

#Evaluation

Without measurement, every change to chunking, models or prompts is a guess. Build a golden set of a few hundred real questions, each labelled with the documents that answer it, and score every change against it before shipping (see evals):

• Retrieval recall at 8: how often is a right chunk among the 8 sent? If it is not there, the model cannot answer well, whatever the prompt says.
• Faithfulness: is every claim in the answer supported by a cited chunk? A model can grade this at scale; check its grades against people on a sample.
• Correct "I don't know": include questions the documents cannot answer, and score the refusals.

In production, track thumbs down, citation clicks, the "I don't know" rate and answers per source, and add failed questions to the golden set.

#Bottlenecks and failures

At 10x the questions, about 200 a second at peak, the model and the reranker run out first; the index barely notices. Add model capacity, cache more, and cut tokens. At 10x the documents, 2 TB of vectors at 4 bytes per number, shard the index by chunk and query all shards in parallel, and store numbers in 1 byte each.

#What a strong answer sounds like

#Follow-up questions

#How do you handle follow-up questions in a conversation?

Rewrite the follow-up into a standalone question using the recent turns ("And for production?" becomes "How do I rotate the production database password?"), then retrieve with that. Retrieving on the raw follow-up finds nothing useful.

#The answer is wrong but the right document exists. How do you debug it?

Check retrieval first: was the right chunk in the 50 candidates, and in the final 8? If not, fix chunking, search or ranking. If it was, look at the prompt and the model's use of it.

#Why not put all the documents in the context window?

Ten million documents fit in no context window, and large windows cost tokens on every question. Models also use long contexts unevenly. Retrieval sends a few thousand relevant tokens instead.

#How do you handle tables and spreadsheets?

Chunk them by rows with the header repeated in each chunk. For numeric questions such as "sum this column", give the model a tool that queries the data instead.

#How would you deploy this in several regions?

Replicate the index to each region and run answer services and models close to users. Ingestion can stay central. If documents must stay in a region by law, keep separate indexes and route by the user's region.