Caching
Keep hot data in memory so most reads skip the database: where caches sit, how to keep them fresh, what to evict, and how to talk about it in an interview.
On this page
An online shop's product page reads one row from the database per view. On a normal day that is 400 reads a second. During a sale it is 4,000 a second, each read takes about 5 ms, and the database starts timing out, even though nearly every read asks for one of the same few hundred products.
A cache fixes this by remembering answers: the first request for product 42 goes to the database, and the next few thousand get a copy from memory in about half a millisecond.
The idea
A cache is a small, fast store of copies of data that is expensive to fetch or compute. Check it first. If the value is there (a hit), return it. If not (a miss), read the source and keep a copy for next time.
It works because real traffic is lopsided: a small share of keys gets most of the reads. If 90% of reads hit the cache, the database sees 4,000 × 0.1 = 400 reads a second during the sale, the same load as a quiet day. Average latency drops too: 0.9 × 0.5 ms + 0.1 × (0.5 + 5) ms = 1 ms, against 5 ms with no cache.
The price is that you now have two copies of the data, and they can disagree. Most of caching is deciding how much disagreement you can live with.
Hit, miss and invalidation
The most common pattern is cache-aside: the application talks to both the cache and the database. Step through two reads and a write, and watch what the write must do to the cached copy.
- cache hits
- 0
- db reads
- 0
1A shopper asks for product 42.
- cache hits
- 1
- db reads
- 0
2It asks the cache for key
p:42. The key is there: a hit, in about 0.5 ms.- cache hits
- 1
- db reads
- 0
3The page goes back to the shopper. The database was never touched.
- cache hits
- 1
- db reads
- 0
4Product 77 is not in the cache: a miss.
- cache hits
- 1
- db reads
- 1
5The app reads product 77 from the database, about 5 ms.
- cache hits
- 1
- db reads
- 1
6It stores the row in the cache with a 10-minute time to live (TTL), so the next reads of 77 are hits.
- cache hits
- 1
- db reads
- 1
7An editor changes the price of product 42. The app writes the database first: it is the source of truth.
- cache hits
- 1
- db reads
- 1
8Then it deletes
p:42from the cache. Skip this and shoppers see the old price until the TTL runs out.- cache hits
- 1
- db reads
- 2
9The next read of 42 misses, loads $18 and caches it again.
- cache hits
- 1
- db reads
- 4,000/s
10Now the cache node dies. All 4,000 reads a second land on a database sized for 400. Plan for this with cache replicas and a limit on the traffic that falls through.
Four ways to wire a cache
The options differ in who loads the cache and when writes reach the database.
| Option | Good at | Bad at | Pick it when |
|---|---|---|---|
| Cache-aside Usual default | Simple. Only data that is read gets cached. If the cache dies, the app still works, slower. | A miss costs three trips. Every write path must remember to invalidate. | Read-heavy data. The default answer in an interview. |
| Read-through | The cache loads misses itself; the app reads from one place. | The cache must know how to query your database. | Your cache library supports it and you want simpler app code. |
| Write-through | Writes update cache and database together, so reads after a write are fresh. | Slower writes. Data never read still fills the cache. | Data is read right after it is written (a user's own profile). |
| Write-behind | Writes return once the cache has them; the database is updated later in batches. | If the cache dies before flushing, acknowledged writes are lost. | Huge write rates where losing seconds of data is fine (view counters). |
What to drop when memory is full
A full cache must evict something. LRU (least recently used) drops the key untouched for longest; it is the usual default because recent use predicts the next use. LFU (least frequently used) keeps long-term favourites, but last week's hot key can squat in memory. A TTL is a different tool: it removes entries too old to trust, even if there is room. Use LRU for space and a TTL as a safety net for any invalidation you miss.
The stampede
A popular key expires. Every request that arrives before someone refills it is a miss, and each one runs the same database query. If rebuilding the home page feed takes 200 ms and 4,000 requests arrive a second, about 4,000 × 0.2 = 800 identical queries pile onto the database at once. That is a cache stampede, and it can take down the database the cache was protecting.
The fixes make one request rebuild the value while the rest wait or use the old one: a short lock per key in the cache, or request coalescing in each server. Also refresh hot keys a little before they expire, and add random jitter to TTLs so keys written together do not expire together.
Numbers worth remembering
- Reading memory on the same machine: about 100 ns. A round trip to another machine in the same data center: about 0.5 ms. A networked cache read is mostly that round trip.
- An indexed database read: about 1 to 10 ms, more under load.
- One in-memory cache node: on the order of 100,000 simple reads a second.
- Sizing: hot items × bytes per item. 10 million hot products at 2 KB each is 107 × 2 × 103 = 20 GB, which fits in one large node or a small cluster.
With the cache in place, how many reads per second still reach the database at peak?
- 4,000 product page views a second at peak
- Each view reads one product
- The cache hit rate is 95%
Show the working
- Misses are 100% − 95% = 5% of reads
- 4,000 × 0.05 = 200 reads a second reach the database
Estimate 200/s (anything from 100/s to 400/s is a good answer)
Going from 90% to 95% hits halves the database load.
In the interview
Bring up a cache when reads dominate (10 per write or more) and some keys are much hotter than others, or when a result is expensive to compute and often requested. Give the numbers and say what staleness you accept. Adding a cache to every box by reflex is a red flag.
Product pages get slow during sales. What would you change?
Reads outnumber writes about 100 to 1 here, and a few hundred products get most of the views, so I'd put a cache-aside layer in front of the product table. Key by product id, 10-minute TTL, and the write path deletes the key when a price changes.
A customer sees a stale price at checkout. Is that acceptable?
On the listing page, a few seconds of staleness is fine. At checkout, no: checkout reads the price from the database, so the cache can never charge someone the wrong amount.
What happens if the cache cluster goes down?
Reads fall through at ten times the normal database load. I'd run cache replicas, rate limit the fall-through so the database degrades instead of dying, and warm the cache before it takes full traffic again.
Follow-ups you should expect, with short answers:
- "How do you keep the cache consistent?" Delete on write, plus a TTL as a safety net. For data that must never be stale, read the database.
- "What if one key is extremely hot?" Keep a copy in each app server's memory for a few seconds.
- "Why delete instead of updating the cached value on write?" Two writers updating the cache can finish in the wrong order and leave the older value cached. A delete cannot.
Common mistakes
Reads hit a cache 80% of the time. Each cache read costs 1 ms and each database read 10 ms. What is the average read latency with cache-aside?
A popular key expires and 500 requests miss at once, each running the same 300 ms query. Which fix targets this directly?
View counters are updated 50,000 times a second and losing a few seconds of counts is fine. Which write strategy fits?
What to practise next
Caches spread across machines need a way to decide which node holds which key: that is consistent hashing. To see a cache inside a full design, work through Design a URL Shortener.