Design a Payment System

A customer presses "Pay $40" on an online shop. A second later they see "Thank you". In between, the shop's server asked a payment platform to charge the card, the platform asked a card processor, and the processor asked the card network and the bank. Any of those calls can time out after the money has moved. The system you design has to know, every time, whether this customer was charged, exactly once, and who is owed what.

Interviewers ask this question to see whether you change your instincts when money is involved. Most designs trade a little correctness for availability. This one does the opposite, and the deep parts are about uncertainty: retries, lost responses, late notifications and books that must balance. The steps follow A Framework for Any Design.

#Clarify the problem

• Functional: create a payment for an order and charge the card through the processor; report its status; refund all or part of it; keep a ledger of what each shop is owed; reconcile with the processor daily.
• Non-functional: a payment request retried any number of times charges at most once; no acknowledged payment is ever lost (zero data loss on a database failure); every movement of money is recorded and auditable for years; the API usually answers in about 2 seconds, mostly the processor's time, and gives up on the processor after 5; availability is high, but correctness wins any conflict.
• Out of scope: fraud detection, currency conversion, sending payouts to bank accounts, storing raw card numbers (the processor holds them and gives us a token).

#Estimate the scale

A day has 86,400 seconds; call it about 100,000 for quick sums, but use the real number when it matters.

• Payments: 5 million ÷ 86,400 ≈ 58 a second on average. Sales days run hotter; plan for 10x, about 600 a second.
• Money: 5 million × $40 = $200 million a day. A bug that loses 0.1% is $200,000 a day, which is why this design spends its effort on correctness, not throughput.
• Payment records: 5 million × 365 ≈ 1.8 billion a year; at 1 KB each, about 1.8 TB a year.
• Uncertain outcomes: if 1 call in 1,000 to the processor times out, that is 5,000 payments a day whose result you do not know at first. They are a normal, daily case, not an edge case.

At 600 payments a second, throughput is easy for one well-sized relational database. The hard part is that each of those writes has to be exactly right.

#Define the API

POST /v1/payments          Idempotency-Key: ord-5521-try
  body: { "amount": 4000, "currency": "USD", "card_token": "tok_...", "order_ref": "5521" }
  201 { "id": "pay_88", "status": "succeeded" }
  201 { "id": "pay_88", "status": "processing" }      outcome not known yet
  402 { "id": "pay_88", "status": "failed", "reason": "card_declined" }
  409 Conflict      a request with this key is still running
  422 Unprocessable the same key was used with a different body
 
GET  /v1/payments/{id}
  200 { "id": "pay_88", "status": "succeeded", "refunded": 0 }
 
POST /v1/payments/{id}/refunds   Idempotency-Key: ref-5521-1
  body: { "amount": 1500 }
  201 { "id": "re_9", "status": "processing" }
 
Webhooks we send to shops:  payment.succeeded, payment.failed, refund.succeeded

The decision worth arguing is the processing answer. When the processor does not reply in time, the honest response is "we don't know yet", not "failed". If the API said failed, the shop would ask the customer to pay again, and the first charge might still go through. So the API can return processing, and the shop learns the outcome from a webhook or by polling.

#Model the data

payments
  payment_id      VARCHAR(32) PRIMARY KEY
  merchant_id     BIGINT
  amount_cents    BIGINT
  status          VARCHAR(16)   -- created | processing | succeeded | failed | refunded
  processor_ref   VARCHAR(64) NULL
  idem_key        VARCHAR(64)
  request_hash    CHAR(64)      -- detects a reused key with a new body
  UNIQUE (merchant_id, idem_key)
 
payment_events                  -- append-only history
  payment_id, seq, from_status, to_status, source, created_at
 
ledger_entries                  -- append-only, never updated
  entry_id        BIGINT PRIMARY KEY
  txn_id          BIGINT        -- entries of one txn sum to zero
  account         VARCHAR(64)   -- "processor_receivable", "merchant:42"
  amount_cents    BIGINT        -- debit positive, credit negative
  created_at      TIMESTAMP

Money is stored as whole cents in integers, never floating point, which cannot represent most decimal fractions exactly. Access patterns are small, transactional and mixed: insert a payment and its key together, move it to a new status only from the right old status, and write ledger lines that must all commit or none. That is a relational database with ACID transactions, run as a single leader with a synchronous replica. Key-value stores are great at scale, but here the one-transaction guarantees matter more than horizontal growth, and 600 payments a second does not need that growth.

#High-level design

Not drawn: our own webhooks to shops. Write each one to an outbox table in the same transaction as the status change, and let a sender deliver it with retries. A crash between "succeeded" and "tell the shop" then delays the notice instead of losing it.

Step through a payment whose first reply is lost, which is the case the design exists for.

#Deep dives

#Idempotency keys, end to end

The idempotency lesson covers the basic pattern. In a payment system, three details decide whether it actually works.

• The key is written in the same transaction as the payment. If the key lives in a cache and the payment in the database, a crash between the two writes lets a retry slip through.
• The key is scoped and fingerprinted. Unique per shop, not globally, and stored with a hash of the request. The same key with a different amount is a client bug: return 422 rather than guess.
• The key travels downstream. Our retries to the processor reuse pay_88 as its idempotency key, so a retry after a lost reply returns the first charge instead of making a second. Every hop that can be retried needs its own key.

This is what "exactly once" means here, and it is worth saying precisely: requests can arrive any number of times, and the charge takes effect once. No network can deliver a message exactly once; the receiver makes repeats harmless.

#The payment state machine

Every payment is in exactly one state, and only certain moves are allowed. Writing the rules down turns a pile of race conditions into a table you can check.

Each move is a conditional update: SET status = 'succeeded' WHERE payment_id = 'pay_88' AND status = 'processing'. If the reply and the webhook race, the first applies and the second changes zero rows, which is correct. A late "processing" event can never drag a succeeded payment backwards, because there is no such arrow.

Payments stuck in processing are expected. A sweeper finds ones older than a minute and asks the processor for the result by idempotency key. Only when the processor says it has no charge with that key may the payment move to failed, and even that answer can race with the original request still in flight. So either wait past the processor's own request timeout before trusting "no such charge", or resend the charge with the same key: the processor then returns the first result or makes the charge once, and the outcome becomes definite.

#A double-entry ledger

A single balance column per shop is easy to update and impossible to audit: when it is wrong, nothing tells you why. A ledger records every movement as entries that sum to zero. For the $40 payment, with a 3% platform fee of $1.20:

+4000 − 3880 − 120 = 0. (Real processors also charge their own fee; leaving it out keeps the example small.) Three rules make the ledger trustworthy:

• Every transaction sums to zero. Check it before commit. Money cannot appear or vanish, only move between accounts.
• Append only. A mistake is fixed with a new, reversing transaction, never by editing a row. The history is the audit trail.
• Balances are derived. A shop's balance is the sum of its entries. You may cache it in a balance row updated in the same transaction, but the entries are the truth, and you can rebuild the cache from them.

#Webhooks, retries and reconciliation

The processor sends webhooks at least once and not always in order. The handler verifies the signature, then in one transaction inserts the event id into a table with a unique key and applies the event through the state machine. Only after that commits does it return 200. A repeat hits the unique key and is acknowledged without changing anything; a crash before the commit sends no 200, so the processor delivers the event again. Recording the id first and applying the change later would turn a crash in between into a lost event. Out-of-order events are harmless because illegal moves change nothing. If an event looks wrong, fetch the payment's current state from the processor's API instead of trusting the event.

Retries go out with exponential backoff and jitter, and always with the same key. One rule interviewers probe: never retry a payment on a second processor while the first one's outcome is unknown. The keys are not shared between processors, so both could charge.

Reconciliation is the safety net under all of it. Every day the processor publishes a settlement file listing what it actually charged and refunded. The reconciler matches each line to a payment and its ledger lines, by processor reference and amount.

#Bottlenecks and failures

At 10x, 6,000 payments a second at peak and 18,000 ledger lines a second, one database leader starts to strain. Partition by merchant_id: a payment and its ledger lines for one shop stay on one partition, so each transaction is still local. Platform-wide accounts like platform_fees and processor_receivable become hot rows, and every payment touches them; give each partition its own copy of those accounts and sum them for reports. The processor's rate limits also become real, which is a conversation with the provider, not a design trick.

Correctness beats availability in every row below. When in doubt, the system stops taking payments rather than guessing; a shop losing sales for two minutes is bad, a customer charged twice or a ledger that does not balance is worse.

#What a strong answer sounds like

#Follow-up questions

#How do refunds work?

A refund is its own record with its own key and states, linked to the payment. When it succeeds, it writes the reverse ledger lines; partial refunds are allowed while their total stays at or below the original amount.

#Authorize now, capture later?

For goods that ship later, the processor can authorize (reserve the funds) and capture when the order ships. Add authorized and captured states; authorizations expire after some days, so a job captures or releases them in time.

#How do you pay shops?

Once a day, sum each shop's ledger balance, create a payout record with its own idempotency key, and write ledger lines moving the amount from the shop's account to a payouts account. The bank transfer is another external call with the same unknown-outcome care.

#How do you store card numbers?

You don't. The customer's browser sends the card to the processor, which returns a token. Keeping raw card numbers out of your systems shrinks the compliance burden enormously.

#Could you use an eventually consistent database?

For the payment records and ledger, no: you need the key insert and the status change to be atomic, and a lost acknowledged write is lost money. For read-only views such as a shop's dashboard, a replica a few seconds behind is fine.