Design an LLM Chat Service

Maya types "Plan three days in Lisbon" and the answer starts appearing word by word less than a second later. She closes the app halfway through, opens it on the train, and the reply is there, finished. A consumer AI chat assistant is the kind of product this question means.

It looks like a chat app with a bot on the other end, but the hard parts differ: the "other person" is a pool of GPUs that is expensive, slow to grow and easy to overload, each reply takes seconds, and each request carries the conversation so far. The steps follow A Framework for Any Design. Try each "Your turn" box before you read on.

#Clarify the problem

• Functional: start a conversation, send a message, stream the reply as it is generated, stop a reply, list and reopen past conversations, edit a message or regenerate a reply.
• Non-functional: first words within 1 second for most requests; replies stream at least as fast as people read; a reply survives a reconnect; no user message is ever lost; free users are throttled before paid users; every request passes safety checks.
• Out of scope: training the model, file uploads, tool use and web search, voice.

#Estimate the scale

Count in tokens, not requests: a 5-token and a 2,000-token reply are one request each but cost 400 times apart. These figures are this scenario's assumptions; measure your own model and hardware.

• Messages: 10 million users × 10 = 100 million a day ÷ 86,400 ≈ 1,200 a second, about 3,500 at a peak of 3x.
• Tokens per request: about 2,000 in (instructions plus history) and 500 out.
• Output rate: 3,500 × 500 = 1.75 million tokens a second at peak, and 3,500 × 2,000 = 7 million input tokens a second to read.
• Open streams: at 50 tokens a second per stream, a 500-token reply takes 10 seconds. Requests in flight = arrival rate × time each takes = 3,500 × 10 = 35,000 open streams at peak.
• Storage: a user message of about 400 bytes, a reply of about 2,000 bytes and 600 bytes of metadata make 3 KB per exchange: 100 million × 3 KB = 300 GB a day, about 110 TB a year.

#Define the API

POST /conversations                          201 { "conversation_id": "c42" }
 
POST /conversations/{id}/messages            Accept: text/event-stream
  body: { "content": "Plan three days in Lisbon",
          "client_msg_id": "m-7f3a",          retry-safe
          "parent_id": "a19" }                which reply this follows
  200 event stream:
    event: start  data: { "user_msg_id": "u20", "reply_id": "a21" }
    id: 1   event: delta  data: { "text": "Day one:" }
    id: 2   event: delta  data: { "text": " start in Alfama" }
    ...
    event: done   data: { "finish": "complete", "tokens_out": 520 }
  400 message too long
  429 Too Many Requests  Retry-After: 30  { "reason": "budget" | "busy" }
 
GET  /replies/{reply_id}/stream              Last-Event-ID: 180
  200 event stream from chunk 181, or the whole reply if it has finished
POST /replies/{reply_id}/stop                stops generation, keeps the text so far
GET  /conversations?cursor=...               newest first
GET  /conversations/{id}/messages?before=...

The decision worth arguing is server-sent events over WebSockets. Traffic is one-way while a reply streams; the user's message is an ordinary POST. Server-sent events are plain HTTP, pass through existing proxies and load balancers, and define resume: a reconnecting client sends Last-Event-ID. A WebSocket buys two-way traffic you do not need. Real-time communication compares the options.

#Model the data

conversations              -- partition key: user_id
  user_id, conversation_id       PRIMARY KEY
  title          TEXT
  updated_at     TIMESTAMP       -- list newest first
  summary        TEXT NULL       -- of older turns, see deep dives
  summary_upto   message_id NULL
  current_leaf   message_id      -- the path being shown
 
messages                   -- partition key: conversation_id
  conversation_id, message_id    PRIMARY KEY   -- time-ordered ids
  parent_id      message_id NULL
  role           VARCHAR(10)     -- user | assistant
  content        TEXT
  tokens         INT
  status         VARCHAR(10)     -- streaming | complete | stopped | blocked | failed
 
usage                      -- one row per user per day
  user_id, day, tokens_in, tokens_out

Access patterns: append a message, read a conversation in order, list a user's conversations by recency. No joins, append-only writes, and 110 TB a year needs partitioning: a wide-column store (such as Cassandra) keyed by conversation fits, the same choice as in Design a Chat App. The stream buffer is separate: an in-memory log per reply (a stream in a store such as Redis), with a 10-minute expiry after the reply finishes.

#High-level design

The chat service does the product work and owns each reply. The inference router keeps the GPUs full without letting the queue grow past what users will wait for. Model servers write tokens into a buffer; the chat service reads, checks and forwards them.

#Deep dives

Interviewers push hardest on GPU capacity, then streaming, context and safety.

#Streaming that survives reconnects

The key move is to separate generating a reply from delivering it. If the reply lived only in the HTTP connection, every drop would lose a half-finished answer and the GPU time spent on it.

• Generation writes numbered chunks into the stream buffer for that reply, whoever is listening.
• Delivery is any chat server reading the buffer from the sequence number the client last saw.
• Retries: a POST retried with the same client_msg_id attaches to the existing reply's stream, so one message never produces two replies (see idempotency).
• Stop: an explicit stop frees the batch slot at once. A disconnect does not: the reply is capped at 1,000 tokens, and users expect it when they come back.

If the owning chat server crashes mid-reply, the buffer still has every chunk; another server sees the reply stuck in streaming with no heartbeat and finishes it as a continuation, like a lost replica.

#GPU capacity: batching, admission and tiers

You cannot autoscale out of a peak: new GPU capacity takes minutes to start and may not exist. So the design manages a fixed pool with four mechanisms.

Continuous batching. Each step, a replica produces one token for every stream in its batch, so 40 streams cost little more per step than one. A fixed batch waits for its longest reply, leaving finished slots idle; continuous batching adds requests between steps and frees a slot the moment a reply ends. The limit is memory: each stream keeps per-token state (the attention key/value cache), so an 8,000-token conversation takes about four times the room of a 2,000-token one. Long contexts mean fewer streams per replica. A long prompt is read in pieces between generation steps so it does not stall the batch.

Admission control. The router keeps a queue per tier and estimates the wait as queued tokens ÷ pool throughput. When the free tier's estimate passes 10 seconds, the gateway rejects new free requests with 429. A fast "busy, try in 30 seconds" beats a spinner that times out after the GPU time was spent.

Rate limits counted in tokens. A per-user token bucket filled with tokens, not requests, is fair: one 8,000-token request costs what four 2,000-token ones do. The output length is unknown at the start, so reserve input plus the output cap, then refund the difference, as in the tracer.

Priority tiers. The router fills free slots from the paid queue first, with a fixed share kept for free requests so they are slowed but not starved. Under overload the free tier is shed first.

#Building the context

The model remembers nothing between requests: each one carries the instructions and the conversation so far, all inside a limited context window. Even when the window is large, longer prompts cost more, delay the first word, and can make answers worse (context rot). So set a budget per request, here 8,000 tokens: system instructions 1,000, the reply cap 1,000, a summary up to 500, and recent turns in the remaining 5,500.

A 40-turn conversation at about 600 tokens a turn is 24,000 tokens, so most of it cannot be sent. Older turns are covered by a rolling summary. When the recent turns pass about 4,500 tokens, well before the 5,500 limit, a background job folds the oldest 2,000 tokens of them into the summary with a small model and stores it with the id of the last message covered. Retrieval over old turns, as in Design a RAG System, can recover a detail the summary dropped; context engineering covers the prompt side.

Order matters for cost. A model server can reuse work done on a recent prompt prefix, but only if it matches token for token from the start. Put fixed instructions first, then the summary, then the turns. Between summary refreshes the cut point stays fixed and turns are only appended, so consecutive requests share a long prefix. The router sends a conversation back to the replica that served its last turn (consistent hashing on the conversation id, with a fallback when it is full) so that prefix is not read again. Caching whole replies rarely pays: two contexts almost never match after the first turn.

#Safety in the path

There are two checks, placed for cost and latency (see guardrails):

• Input, before the queue. A small classifier checks the message; a blocked request costs no GPU time. Pasted text can carry instructions aimed at the model, so treat it as data (see prompt injection).
• Output, while streaming. Checking only the finished reply means showing unchecked text for 10 seconds; checking every token is too slow. Hold each sentence until it passes, a fraction of a second of delay. Mark passed chunks in the buffer, so a resumed stream replays only checked text. If a later sentence fails, stop generation and send a retract event so the client replaces what it showed.

If the safety service is down, fail closed: pause new generations rather than ship unchecked output. The cost: a classifier outage becomes a product outage, so it needs the same redundancy as the model servers.

#Bottlenecks and failures

At 10x, 35,000 messages a second at peak, the web tier and store scale out as usual. The GPU pool needs about 9,000 replicas, which you may not be able to buy or power. The controls above decide who waits; beyond them, the levers are fewer tokens per reply, smaller models for more requests, and sending load to regions with spare capacity at night.

#What a strong answer sounds like

#Follow-up questions

#How do you measure whether you are meeting the latency goal?

Track time to first token and tokens per second per stream, split into queue wait and model time, per tier. Rising queue wait with flat model time means you need capacity or shedding, not faster code.

#How does "regenerate" work?

It creates a sibling of the old reply with the same parent and runs the same context again. The conversation's current leaf points to the new one; the old one stays reachable. Sampling makes the new reply differ (see sampling and temperature).

#How would you add memory across conversations?

Keep a small set of facts per user, extracted after conversations, and add relevant ones within the context budget. Users must be able to see and delete them (see memory).

#A user deletes a conversation. What has to happen?

Delete it from the store, expire its buffer, and drop any summary or memory built from it. Logs and backups follow a retention policy you should state.

#How do you roll out a new model version?

Run it on a slice of replicas, send it a small share of traffic, and compare evals, safety rates and latency before shifting more.