CAP and Consistency

A profile service keeps its database leader in an East data center and a read replica in the West. A user in the West changes her bio, refreshes 200 ms later, and sees the old bio: the replica is one second behind. To her, the save button looks broken.

Then the link between the data centers drops for 90 seconds. Writes from the West cannot reach the leader. Either the West refuses them until the link returns, or it accepts them and risks two versions of the same row. That choice is what CAP is about.

#The idea

Once data lives on several machines, copies can disagree. A consistency model is the promise about what a read can return after a write. Stronger promises are easier to build on but cost latency or availability. Weaker ones are fast, but the app must cope with stale data.

CAP says: when the network splits the replicas into groups that cannot talk (a partition), each group must either refuse some requests or answer them without the latest writes. You cannot have both. In CAP, C means linearizable: every read sees the latest completed write, as if there were one copy. A means every request to a working node gets an answer.

So CAP is a choice made only during a partition; with a healthy network you can have both.

PACELC adds the part CAP leaves out. If there is a Partition, choose Availability or Consistency; Else, even when all is well, choose Latency or Consistency. A strongly consistent write waits for other replicas to confirm: across continents, around 100 ms on every write. This everyday cost usually matters more than the rare partition.

#Lag, then a partition

Step through the profile service: a stale read, the fix, then the split.

One system made both choices: West reads chose availability (stale but answered), writes chose consistency (refused). CAP is decided per operation, not per database.

#Four promises, as users see them

• Strong (linearizable). Once a write returns, every read anywhere sees it. Add a conditional write ("book 14C only if it is free") and two people cannot both get the seat.
• Eventual. If writes stop, all copies converge. Until then a like count may read 42, then 41 from another replica.
• Causal. If one write depends on another, everyone sees them in that order. A reply never appears before the comment it answers.
• Read-your-writes. A user always sees their own changes, even if others see them later. Ana's bio after she saves it.

Start from eventual consistency plus read-your-writes for the author. Spend strong consistency only where a stale read breaks a rule across users: only one winner, never below zero.

#What a quorum does and does not give you

Leaderless stores write to W of N replicas and read from R. If R + W > N, every read set overlaps every write set, so a read contacts at least one replica that has the latest acknowledged write. With N = 3, W = 2, R = 2 that holds: 2 + 2 = 4 > 3.

That overlap is not linearizability. Concurrent writes can land in different orders on different replicas, a half-failed write may be visible on one replica, and some stores fall back to "any W nodes" during failures. Say "quorum reads see the latest acknowledged write in the normal case", not "quorums make it strongly consistent".

#Numbers worth remembering

• Replica lag in one region: usually milliseconds, but seconds or minutes under load. Design for "unbounded but usually small".
• Round trip within a data center: about 0.5 ms. Between regions on one continent: tens of ms. Between continents: around 100 ms.
• N replicas with majority quorums tolerate (N − 1) ÷ 2 failures, rounded down: 3 tolerate 1, 5 tolerate 2.

#In the interview

Bring consistency up when you add replicas, caches or a second region, and whenever two users can race for the same thing. Do not announce "this is an AP system"; state the need per feature, with the reason: "Seat booking must be strongly consistent, because two people paying for one seat is the failure we cannot have. The seat map can lag by a few seconds."

Follow-ups you should expect:

• "Is your database CP or AP?" Usually configurable per operation. Say which operations refuse during a partition and which serve stale data.
• "Why not promote the West replica when the link drops?" From the West, a cut link looks like a dead leader. Promoting would create two leaders: split brain. Promotion needs a majority vote, so put a tiebreaker in a third site (see consensus).
• "How do you get read-your-writes with replicas?" Route the writer's reads to the leader for a short window, or remember the log position of their last write and read only from a replica that has reached it.

#Common mistakes

#What to practise next

Consistency across replicas is half the story; consistency between concurrent writers on one database is transactions and isolation. To see consistency chosen per feature in a full design, work through Design Ticket Booking.