Replication

A shop runs on one database server. At 14:00 its disk fails. The last backup is from 02:00, so 12 hours of orders are gone. Restoring the 2 TB backup at 200 MB/s takes 2 × 10¹² ÷ 2 × 10⁸ = 10,000 seconds, nearly 3 hours, and the site is down the whole time.

With a second server receiving every change as it happens, the same failure usually costs under a second of writes and about a minute of downtime.

#The idea

Replication keeps copies of the same data on several machines, called replicas. Every change made on one is shipped, through a log of changes, to the others. You get three things: durability (a dead disk loses nothing), availability (another replica takes over), and read capacity (reads spread across copies).

The cost: copies are not updated at the same instant, and between a write and its arrival the replicas disagree. Most replication decisions are about how long that gap is and who can see it.

#Leader and followers

The common setup: one replica, the leader, accepts all writes and streams its log of changes to followers, which apply them in the same order. Reads can go to any of them.

#Synchronous or asynchronous

With synchronous replication the leader waits for a follower to confirm before telling the client "saved". No acknowledged write is lost on failover, but every write pays a round trip, and if that follower is down, writes stall. With asynchronous replication writes are fast and never wait, but a failover loses whatever had not shipped, as in the tracer.

The usual middle ground is one synchronous follower and the rest asynchronous: every acknowledged write is on two machines, and if the synchronous follower fails, another takes its role.

#Three ways to arrange replicas

#Quorums: why R + W > N

In a leaderless store each key lives on N replicas. A write succeeds once W of them confirm; a read asks R of them and keeps the value with the highest version. If R + W > N, every set of R replicas shares at least one replica with every set of W, so a read always reaches at least one copy of the latest successful write. This needs a strict quorum: if stand-in nodes may accept writes while the owners are down (a "sloppy" quorum), reads can miss the write until it is handed back.

Be precise about what a quorum does not give you. Two clients writing the same key at once both succeed, and the store must merge them or pick one; last-writer-wins silently drops a write. A write that reached fewer than W replicas reports failure but may still be read later. And mid-write, one reader can see the new value and a later reader the old one. R + W > N guarantees overlap, not the behaviour of a single up-to-date copy (linearizability), which needs extra machinery.

#Numbers worth remembering

• Replication lag inside one region: usually well under a second when healthy; seconds to minutes when a follower is overloaded or catching up.
• Synchronous replication adds one round trip per write: about 0.5 ms in a data center, 50 to 150 ms between continents.
• Async failover loses roughly write rate × lag. Detection takes seconds: a shorter timeout replaces leaders that were only slow.

#In the interview

Bring up replication when the requirements say "must not lose data" or "highly available", or reads swamp one server. Say which topology, sync or async, and how reads right after a write behave.

Follow-ups you should expect:

• "How do you avoid two leaders after a failover?" Elect the leader through a consensus service and give each leader a term number; storage rejects writes carrying an old term (fencing). See consensus and leader election.
• "Isn't replication a backup?" No. A bad DELETE replicates in milliseconds. Keep backups with point-in-time recovery too.

#Common mistakes

#What to practise next

Replication copies all the data to every replica; when the data itself is too big for one machine, you split it with sharding. The guarantees here are made precise in CAP and consistency, and a quorum store is built end to end in Design a Key-Value Store.