Message Queues and Streams

A signup endpoint creates the account, then calls an email provider to send a welcome message before it responds. One afternoon the provider slows from 100 ms to 3 seconds. At 50 signups a second, 50 × 3 = 150 requests are now stuck waiting at any moment, the server's worker pool fills up, and signups fail even though the account database is fine.

The welcome email did not need to happen before the response. Put it on a queue, answer the user in milliseconds, and let a separate worker send it whenever the provider allows.

#The idea

A producer writes a message to a durable broker and moves on. A consumer reads it later and does the work. The two no longer share a speed or a fate: a burst of messages waits in the broker instead of crushing the consumer, and a dead consumer delays work instead of failing requests.

The costs: the work happens later (seconds, sometimes minutes), there is one more system to run, and messages can arrive twice.

#Queue or log

Brokers come in two shapes. A queue (such as RabbitMQ or SQS) hands each message to one consumer and deletes it once acknowledged. A log (such as Kafka) appends messages and keeps them for days; each consumer group remembers its own position (an offset) and can rewind.

One job type and one kind of worker: use a queue. Once a second team wants the same events, or you need replay, use a log.

#Delivery guarantees

A consumer does two things: the work, and the acknowledgement. A crash can land between them, and the order you choose decides what a crash costs.

• At-most-once: ack, then work. A crash after the ack loses the message. Fine for metrics you can afford to drop.
• At-least-once: work, then ack. A crash before the ack means the message comes back and the work runs again. The usual choice.
• "Exactly-once": no broker can stop a consumer from doing an external action twice. What you can build is exactly-once effect: at-least-once delivery plus processing that is idempotent, so the second run changes nothing. Some brokers offer exactly-once within their own reads and writes; it does not cover your email provider.

#Ordering, partitions and consumer groups

A log is split into partitions. Order is guaranteed only inside one partition. The producer picks the partition by hashing a key, so choose the key whose events must stay in order: with order_id as the key, "created", "paid" and "shipped" for one order arrive in sequence, while different orders spread across partitions.

A consumer group is one logical subscriber. Inside a group, each partition is read by exactly one consumer at a time, so a topic with 12 partitions supports at most 12 busy consumers in a group. Different groups (email, analytics) each read every message independently. Pick the partition count for the parallelism you will need later; adding partitions afterwards remaps keys and breaks per-key order during the change.

A hot key, one huge customer, lands entirely on one partition. If that is a risk, add a suffix to spread it, and give up ordering for that key.

#Backpressure

A queue absorbs bursts; it does not fix a consumer that is slower than its producers on average. If producers write 1,200 messages a second and consumers handle 1,000, the backlog grows by 200 a second, 720,000 an hour, and each message waits longer. Watch consumer lag (messages, or seconds, behind) and act on it:

• Scale consumers, up to the partition count.
• Bound the queue and push back on producers: return 429 or 503 and let clients retry later.
• Shed or sample low-value messages (debug events) before high-value ones (payments).

#Numbers worth remembering

• Publishing to a broker in the same data center: a few ms, including replication for durability.
• One partition handles on the order of thousands to tens of thousands of small messages a second; total throughput scales with partitions.
• Little's law: messages waiting = arrival rate × wait time. 50 a second × 3 s = 150, the hook's stuck requests.

#In the interview

Bring in a queue when work is slow, flaky or not needed for the response (emails, thumbnails, fan-out), when traffic is bursty, or when several systems react to one event. Then say the guarantee and how you handle its cost: "The click events go to a log partitioned by link id. Delivery is at-least-once, so the counter worker dedupes by event id."

Follow-ups you should expect:

• "How do you keep events for one user in order?" Key by user id so they share a partition, and process each partition with one consumer.
• "What if consumers fall behind for an hour?" Alert on lag, scale consumers up to the partition count, and work out drain time from the spare capacity.
• "The API crashes after creating the account but before publishing. Now what?" No welcome email, ever. Write the message to an outbox table in the same transaction as the account (see transactions).
• "What if the provider has no idempotency key?" Record sent job ids in your own table. A crash between sending and recording can still send twice, so make that window short.

#Common mistakes

#What to practise next

Every consumer here relies on idempotency and retries, so read that next. To see a queue carry a whole design, work through Design a Notification System.