Reliability and Failure
Availability as arithmetic, redundancy and failover, circuit breakers, bulkheads, graceful degradation, load shedding and multi-region basics.
On this page
A checkout request calls five services one after another: cart, pricing, inventory, payment, orders. Each is up 99.9% of the time, which sounds excellent. But checkout needs all five, so it works only 0.9995 ≈ 99.5% of the time. Over a 30-day month that is about 216 minutes of failed checkouts instead of 43.
Every dependency you add in a chain multiplies in its failures. Reliability is the craft of making the whole stay up when parts do not.
The idea
Availability is the share of time (or of requests) that the system does its job. Two rules of arithmetic drive every design choice:
- In series, multiply availabilities. Five parts at 99.9% each: 99.5% together. More hops, lower availability.
- In parallel, multiply the failure chances. Two copies that each fail 1% of the time are both down 0.01 × 0.01 = 0.01% of the time, so 99.99% together. This assumes they fail independently, which is a simplification: a shared power feed, a bad deploy or the same bug takes both down at once.
So: put redundant copies of everything critical, keep the critical path short, and make sure a failure in one part cannot drag the rest down with it.
| Availability | Downtime per 30-day month | Downtime per year |
|---|---|---|
| 99% ("two nines") | 7.2 hours | 3.65 days |
| 99.9% | 43 minutes | 8.8 hours |
| 99.99% | 4.3 minutes | 53 minutes |
| 99.999% | 26 seconds | 5.3 minutes |
Each extra nine cuts downtime tenfold and usually costs much more effort than the last one. At four nines, a human cannot even be paged and log in before the month's budget is gone: recovery must be automatic.
Redundancy and failover
Stateless servers are easy: run several behind a load balancer, and health checks remove a dead one in seconds. State is harder. A database leader with a standby needs failover: notice the leader is gone, promote the standby, point traffic at it. Downtime is detection plus promotion: with a health check every 5 seconds and 3 misses required, detection alone takes 15 seconds. Detect too eagerly and a slow leader gets replaced while still alive, which is how split brain starts (see consensus and leader election).
Stopping one failure from spreading
The worst outages are cascades: one slow dependency ties up its callers, which then fail their own callers. A product page that calls a recommendations service shows how the patterns fit together.
- views/s
- 1000
- recs latency
- 50 ms
1Normal day: 1,000 page views a second. Each calls catalog (20 ms) and recommendations (50 ms) in parallel.
- views/s
- 1000
- recs latency
- 5 s
2Recommendations slows to 5 seconds. Calls in flight = rate × wait = 1,000 × 5 = 5,000. The page fleet has 2,000 threads, so every thread is soon stuck waiting.
- views/s
- 1000
- recs latency
- 5 s
3Now the whole page fails, even though catalog is healthy. A minor feature has taken down a major one: a cascade.
- views/s
- 1000
- in flight to recs
- 300
4Fix 1, a timeout: give up on recommendations after 300 ms. In flight drops to 1,000 × 0.3 = 300 calls.
- views/s
- 1000
- in flight to recs
- 300
5Fix 2, a bulkhead: recommendations calls get their own pool of 400 threads. If it fills, those calls fail at once; the other 1,600 threads stay free for catalog.
- views/s
- 1000
- in flight to recs
- 0
6Fix 3, a circuit breaker: more than half the calls in the last 10 seconds failed, so the breaker opens. The page stops calling recommendations at all.
- views/s
- 1000
- in flight to recs
- 0
7Graceful degradation: the page shows a cached "popular items" list instead. Shoppers still see products and can buy. Recommendations gets no load and can recover.
- views/s
- 1000
- recs latency
- 50 ms
8After 30 seconds the breaker goes half-open and lets a few trial calls through. Recommendations has recovered, the trials succeed in 50 ms, and the breaker closes.
Two more patterns protect a service from its own callers. Load shedding: when a server is past what it can handle, it rejects extra requests at once with 503, instead of accepting everything and timing out on all of it. Shed the least important work first: browsing before checkout, background refreshes before user clicks. Rate limits do the same per client (see rate limiting). And retries must be limited, or they multiply load on a service that is already struggling (see idempotency and retries).
Multi-region basics
Within one region, spread copies across several zones (separate buildings and power), which covers most hardware failures. Going multi-region guards against losing a whole region, and it is expensive. Two numbers frame the choice: RPO, how much recent data you can lose, and RTO, how long recovery may take.
| Option | Good at | Bad at | Pick it when |
|---|---|---|---|
| One region, several zones Usual default | Survives a building or rack failure. Synchronous replication is cheap at zone distances. | A region-wide outage takes you down. | Most systems, and the starting point in an interview. |
| Active-passive regions | A second region holds an async copy and can take over. Simple data model: still one writer. | Failover takes minutes; RPO is the replication lag; the standby must be tested or it will not work. | A region outage must cost minutes, not hours, and some seconds of data loss is acceptable. |
| Active-active regions | Users are served nearby, and every region already takes live traffic, so there is no untested standby. | Writes in two regions can conflict, so each record needs a home region or conflict resolution. Users homed in a lost region can still lose recent unreplicated writes. | Global latency matters and the data can be partitioned by user or region. |
I start with one region across three zones. What changes my mind: a stated requirement to survive a region outage, which moves me to active-passive, or users on several continents who need low latency, which moves me to active-active with each user's data homed in one region.
A request passes through three services in series, then a database pair where either copy can serve it. About how many minutes a month is the path down?
- Each service: 99.95% available
- Each database copy: 99.9% available, failing independently
- A 30-day month is 43,200 minutes
Show the working
- Services in series: 0.99953 ≈ 0.9985
- Database pair: 1 − 0.001 × 0.001 = 0.999999
- Path: 0.9985 × 0.999999 ≈ 0.9985, so down 0.15% of the time
- 0.0015 × 43,200 ≈ 65 minutes
Estimate 65 minutes (anything from 25 minutes to 200 minutes is a good answer)
The redundant database barely matters; the three single services in a row dominate. Fix the chain first.
In the interview
Reliability questions come after the main design: "What happens if this dies?" Answer per component with what the user sees and what you do. Say the availability target with its arithmetic: "99.9% is 43 minutes a month, so failover must be automatic and under a minute."
Your feed calls an ads service. The ads service starts taking 10 seconds. What happens?
Feed requests call ads with a 200 ms timeout from a separate small connection pool, so slow ads cannot use up the threads the feed needs. If more than half of ad calls fail over 10 seconds, a breaker opens and the feed is served without ads. Users lose ads for a while, not the feed.
And if the feed itself is overloaded?
It sheds load: past its concurrency limit it returns 503 at once for background refreshes first, so foreground requests keep getting answers.
Follow-ups you should expect:
- "How do you test this?" Kill instances and inject latency in a test environment, then carefully in production. A failover never exercised will not work when needed.
- "Why not five nines?" 26 seconds a month means no human in the loop, and every dependency must be better still. Most products do not need it.
- "What is your RPO for payments?" Zero: replicate payment writes synchronously to another zone before confirming.
Common mistakes
Two independent servers are each up 99%. Requests succeed if either is up. What is the availability?
A circuit breaker is open. What does the caller do with a new request?
A service receives 2,000 requests a second and each call to a dependency waits 2 seconds. How many calls are in flight?
What to practise next
You cannot react to a failure you have not noticed: observability is next. To work through failures component by component, try Design a Key-Value Store.