Design Post Search

Something happens in your city and you type "ferry strike" into the search box of a social app. You expect posts from the last few minutes, newest first, or the most-liked posts about it if you flip the sort. The kind of product this question means is Twitter search: keyword search over a huge, fast-moving stream of short posts.

The hard part is that the index is written far more often than it is searched, and one of the two sort orders depends on a number (likes) that changes 116,000 times a second. Search Systems already covers the inverted index, analysers and feeding an index from the database; this lesson builds on it. The steps follow A Framework for Any Design.

#Clarify the problem

• Functional: search public posts by keywords (all must match); sort by newest or by most liked, optionally within a time window; page through results.
• Non-functional: new posts searchable within 10 seconds; 500 ms or better at the 99th percentile; like-based order may lag about a minute; deleted posts disappear from results within seconds; search stays up if parts of the index are down, possibly with fewer old results.
• Out of scope: typo tolerance, phrase search, personalised ranking, private posts, autocomplete (see Design Typeahead Search).

#Estimate the scale

A day is 86,400 seconds; assume peaks of 3x.

• New posts: 500 million ÷ 86,400 ≈ 5,800 a second, about 17,000 at peak.
• Likes: 10 billion ÷ 86,400 ≈ 116,000 a second, 20 per post on average.
• Searches: 200 million ÷ 86,400 ≈ 2,300 a second, about 7,000 at peak.
• Writes against reads: (5,800 + 116,000) ÷ 2,300 ≈ 53, so about 50 writes for every search.

#Define the API

GET /v1/search?q=ferry+strike&sort=recent&limit=20&cursor=...
GET /v1/search?q=ferry+strike&sort=likes&since=2026-10-15&limit=20
  200 { "results": [ { "post_id": "1849...", "author": "...", "text": "...",
                       "likes": 1532, "created_at": "..." } ],
        "next_cursor": "...", "partial": false }
  400 Bad Request     empty query, or more than 10 words
  410 Gone            cursor snapshot expired; start again

The decision worth arguing is pagination. For sort=recent, post ids are time-ordered (see Unique IDs), so the cursor is just "ids smaller than the last one shown": new posts never shift a page. For sort=likes, a post can gain 500 likes between page 1 and page 2, so the server ranks once, keeps the ordered ids for 10 minutes and the cursor names that snapshot. The partial flag tells the app that some shards timed out and the list may miss a few old posts: a degraded answer beats an error.

#Model the data

posts DB (source of truth, partitioned by post_id)
  post_id     64-bit, time-ordered     text        up to 1 KB
  author_id   64-bit                   status      live | deleted
 
index document (one per post, never stores the text)
  post_id     64-bit                   tokens      ["ferry", "strike", ...]
  like_tier   0..6                     -- 0: 0-9 likes, 1: 10-99, 2: 100-999 ...
 
index layout
  one index per day  ->  split into N shards by hash(post_id)
 
like_counts (key-value, partitioned by post_id)
  post_id -> count, last_offset

Three stores, three access patterns. The posts database is read by key to fetch the text of 20 results; a wide-column or key-value store fits. The search index answers "which ids contain all these words", and needs a search engine such as Elasticsearch or a purpose-built inverted index. The like count store answers "how many likes do these 2,000 ids have" in one batch; an in-memory key-value store such as Redis, backed by a durable table, fits.

#High-level design

#Deep dives

Interviewers push on four things: indexing at 5,800 posts a second, sorting by likes, sharding, and keeping popular queries cheap.

#Indexing a firehose

An inverted index is cheap to read and expensive to change in place, because each postings list is sorted and compressed. So the indexer never edits lists. It buffers new posts in memory and writes them out as a small, immutable segment every second; a search reads all segments of a shard. A background job merges small segments into bigger ones, the same idea as the log-structured storage in a key-value store. Deletes are recorded as a list of dead ids and removed for real during merges.

Time-ordered post ids pay off twice here. New posts always have the largest ids, so they append to the end of every list. And a newest-first query can walk lists backwards and stop early. Step through it on one shard.

Within a second of a post arriving, it is in a segment; the 10-second freshness goal leaves room for the stream's own delay. At 17,000 posts a second at peak and 30 tokens each, the fleet adds about 510,000 postings a second, spread over the shards of today's index.

#Sorting by likes without rewriting the index

Rewriting a document per like means 116,000 index updates a second, 20 times the post rate. The fix is to separate "roughly how popular" from "exactly how many".

With tiers, a post with 1 million likes has changed tier 6 times (at 10, 100, 1,000, 10,000, 100,000 and 1 million), not a million times. Most posts cross none or one boundary; even if every post crossed two, that is 2 × 5,800 = 11,600 updates a second, a tenth of the like rate.

A likes query searches the highest tier first, within the time window. Suppose "ferry strike" matches 4 posts in tier 4 and 150 in tier 3. Those 154 are the only possible top 20, since every lower-tier post has fewer likes. The service fetches their 154 exact counts in one batch, sorts, and returns 20. Lower tiers are never read. The order is exact except for a post whose tier update is a few seconds behind, which the requirements allow. A broad word over 30 days can put 50,000 matches in one tier; then each shard sends only the newest few hundred from that tier, which bounds the work and makes the order approximate. Finer tiers (half powers of ten) shrink that case for twice the tier updates.

The counts themselves are batched as in Design Photo Sharing: the aggregator sums likes per post each second and stores the last applied stream offset with the count, so a replay cannot add twice.

#Sharding the index

Split by time first, then by document. Each day gets its own index, split across shards by hash of post id, as Search Systems recommends: every shard holds a fair slice, and a post touches one shard. Time first has three benefits. Writes all go to today's index, which can sit on more machines. Newest-first queries read only the latest days unless they need more. And an old day's index never receives new posts, so it can be merged into one large segment and moved to cheaper machines.

The cost is fan-out. A query that needs a year reads 365 day indexes times their shards. So older days are rolled into week and month indexes, and likes queries default to a window of 30 days; "all time" is allowed but slower.

With 20 shards per query, the slowest shard sets the latency. If one shard is slow 1% of the time, a query that waits for all 20 is slow 1 − 0.99²⁰ ≈ 18% of the time. That is why the tracer sent a hedged request to another replica after 50 ms.

#Hot and cold data, and caching

The recent 7 days, 525 GB of index, live in memory on fast machines with 3 replicas, because nearly every newest-first query ends there. Older data, about 27 TB a year, lives on SSDs with fewer replicas. If the cold tier is down, the service returns recent results with partial: true.

Queries are skewed: during a news event, thousands of people type the same words. Cache the ranked list of ids, keyed by normalised query, sort and window, for 15 seconds, short enough that new posts appear. Also cache per-shard results for days that are over: an old day gains no new posts, so its answer for "ferry strike" changes only through deletes and tier moves, and can be kept for an hour. A deep page then costs one live query on today's shards and cache reads for the rest. Deleted posts are filtered when the text is fetched, so a cached list can never show one.

#Bottlenecks and failures

At 10x (5 billion posts a day), today's index takes 170,000 posts a second at peak, so it needs about 10 times the shards, and the recent 7 days grow to 5 TB of memory. The like stream grows to 1.2 million events a second; it is partitioned by post id so aggregators scale out. The first thing to hurt is usually fan-out latency, which you fix with fewer, larger rolled-up indexes for old data and narrower default windows.

#What a strong answer sounds like

#Follow-up questions

#How do you add relevance ranking?

Fetch a few hundred candidates per shard by a cheap score (text match plus tier plus recency), then re-rank the merged list with a richer model in the search service. The two-stage shape keeps expensive scoring to a few hundred posts.

#A post is deleted. How fast does it disappear?

Immediately for users: the service filters ids whose status is deleted when it fetches text. The index entry is marked dead within seconds and physically removed at the next merge.

#How would you search hashtags or a single user's posts?

A hashtag is just another token. "From this user" adds an author_id field to the document and a filter in the query.

#How do you change the analyser, say to add stemming?

Build new indexes alongside the old ones, newest days first, by replaying from the posts database, then switch reads one day at a time. Old and new can run side by side because each day is a separate index.

#Why not use the database's full-text search?

At 75 GB of new index a day and 5,800 inserts a second, a single database's text index cannot keep up or be split by time cleanly. For a few million posts it would be the right first answer.