Design a Notification System

Your bank texts you a login code, a shop emails a receipt, and your phone buzzes when a parcel ships. Behind each of these, some product service said "tell user 42 about this", and a shared notification service decided how, when and through which company's network the message actually left.

Interviewers ask this question because the hard parts are not the happy path. They are what happens when a provider times out and you do not know whether the text went out, when a marketing campaign for 50 million users sits in front of a login code, and when a retry sends someone the same alert three times. The steps follow A Framework for Any Design.

#Clarify the problem

• Functional: other services send a notification to a user by template, on one or more channels (push, SMS, email); campaigns to large segments; users choose which categories they get on which channel; callers can ask for the status of a notification.
• Non-functional: high-priority messages (login codes, security alerts) reach the provider within 5 seconds, even during a campaign; an accepted notification is never silently dropped; duplicates are rare; accepting requests is 99.99% available, while sending may lag when providers struggle.
• Out of scope: the in-app notification inbox, campaign audience tools, A/B tests of message text.

#Estimate the scale

• Average rates: 500 million push ÷ 86,400 s ≈ 5,800 a second; 100 million emails ≈ 1,160 a second; 5 million texts ≈ 58 a second. About 7,000 a second in total.
• Campaign peak: the average hides the real peak. A campaign to 50 million users that should finish within 30 minutes needs 50,000,000 ÷ 1,800 s ≈ 28,000 sends a second, four times the daily average, on one channel.
• Preferences and devices: 100 million users × 200 bytes of preferences = 20 GB; at 2 devices each and 200 bytes per device token, 40 GB. Both fit in a cache, which matters because every send reads them.
• Provider limits: assume our SMS provider account allows 500 texts a second. That is a hard ceiling we must stay under, not a target.

#Define the API

POST /v1/notifications
  Idempotency-Key: order-881-shipped
  body: { "user_id": 42, "template": "order_shipped", "params": { "order": "881" },
          "category": "order_updates", "priority": "high" | "normal" | "bulk",
          "channels": ["push", "email"]?, "expires_in": "24h"? }
  202 Accepted   { "notification_id": "n_501" }
  400 Bad Request         unknown template or missing parameter
  404 Not Found           unknown user
  409 Conflict            same Idempotency-Key with a different body
  429 Too Many Requests   this caller is over its quota
 
GET  /v1/notifications/n_501
  200 { "push": "sent", "email": "retrying", "sms": "skipped: user opted out" }
 
PUT  /v1/users/42/preferences   { "order_updates": ["push", "email"], "marketing": [] }
POST /v1/campaigns              { "segment": "...", "template": "...", "send_at": "..." }   202
POST /webhooks/{provider}       delivery receipts and bounces from providers

The decision worth arguing is 202, not 200. Sending depends on providers that can be slow or down for minutes, and the caller (an order service, a login flow) should never wait on them. Accept the request, store it, return an id. The cost is that the caller cannot learn about failures in the response; it gets a status endpoint instead. The Idempotency-Key makes the caller's own retries safe, as in Idempotency and Retries.

#Model the data

notifications                     -- about 500 bytes, kept 90 days
  notification_id   CHAR(26)      PRIMARY KEY
  idempotency_key   VARCHAR(100)  UNIQUE (per caller), expires after 24 h
  user_id           BIGINT
  template, params, category, priority, created_at, expires_at
 
deliveries                        -- one row per (notification, channel, device)
  notification_id, channel, target    -- target: device token, phone or email
  status            -- queued | sent | delivered | failed | skipped
  attempts, provider, provider_msg_id, last_error, updated_at
 
preferences   PRIMARY KEY (user_id)   -- category -> allowed channels, quiet hours, time zone
devices       PRIMARY KEY (user_id, device_token)   -- platform, last_seen

The log is write-heavy (thousands of inserts and status updates a second), read by id, and expires after 90 days: a key-value or wide-column store with a TTL suits it (such as DynamoDB or Cassandra). Preferences and devices are small and change rarely, but every send reads them. Keep them in a relational database, which suits the occasional update and the support tools that query it, and put a read-through cache in front. The idempotency key needs an atomic "insert if absent", which both kinds of store offer for a single key.

#High-level design

Split the work at the two places where speed changes. The API must answer callers in milliseconds, so it only checks, records and enqueues. The providers are slow and unreliable, so each channel gets its own queue and sender pool, and a slow email provider never holds up a push.

Follow one "order shipped" notification from the order service to a phone and an inbox, including a retry from the caller and a timeout at the email provider.

#Deep dives

#The pipeline: queues per channel and per priority

Channels differ in speed, cost and failure. Push is fast and cheap; email providers can be slow; SMS is capped at 500 a second. If they share a queue, the slowest channel sets everyone's pace. The 30-minute estimate shows that priority needs the same treatment.

The three API priorities map to two queues: high and normal share the fast one, bulk gets its own. Give the fast queue a guaranteed share of senders and of each provider's rate limit, say 20%, and let bulk use the rest. Run campaigns at a set pace below the provider ceiling: 28,000 pushes a second for 30 minutes is fine. Retries go to a separate delay queue, so a failing message never blocks the head of the main queue.

#Retries and duplicates

Every queue here delivers at least once (see Message Queues and Streams), so a sender can crash after the provider accepted a message but before it marked the job done. The job runs again, and the user gets two texts. You cannot make this exactly once: the provider call and your own status write are two systems with no shared transaction. You can make duplicates rare:

• Before sending, check the deliveries row; skip it if it already says sent. This closes most of the window, not all of it.
• Send a stable id with each request (n_501:email). Some providers drop repeats of an id they have seen; push platforms can collapse notifications that share a collapse key on the device.
• Retry only errors that can succeed later: timeouts, 5xx and 429. An invalid device token or a hard email bounce will never succeed; remove the token or suppress the address instead.
• Back off exponentially with random jitter (1 s, 2 s, 4 s and so on, capped at 10 minutes) until the notification's deadline, then dead-letter it.

The same gap exists at intake. The API writes the log, then enqueues. If it crashes in between, the notification is recorded but never queued, and the caller's retry finds the key and stops there. So a sweeper re-enqueues anything still marked queued after a minute, and routers skip ids they have already routed.

The deadline is a product decision per category, and it is where judgement shows. A login code is useless after 5 minutes, so drop it then. A shipping update is still useful a day later. For login codes a duplicate is far better than a missing code, so retry aggressively; for marketing, a duplicate annoys people and costs money, so retry less and accept the odd loss.

#Preferences and rate limits

Check preferences in the router, as late as possible. A campaign that starts at 09:00 reaches some users at 09:29; one who opted out at 09:15 must be skipped. That works because the campaign job feeds work in at the send pace, so little waits in the bulk queue after routing. Some categories cannot be turned off (security alerts), and the preference model must say so. Quiet hours need the user's time zone: a normal message that falls in the night is parked in a scheduled store and released in the morning; a high-priority one goes through.

Three kinds of limit protect different things:

• Per user, so people are not spammed: for example at most 3 marketing pushes a day, counted in the cache with a key like 42:marketing:2026-11-14.
• Per caller, so a buggy service in a loop cannot flood users or the queues: a quota per API key, answered with 429.
• Per provider, so we stay under the 500 texts a second our account allows: a token bucket shared by all SMS senders. See Rate Limiting.

#When a provider fails

Providers fail in three ways: errors, slowness and silent loss. Errors are easy to see. Slowness is worse, because senders sit waiting and the queue grows. Put a short timeout on every call and a circuit breaker per provider: after, say, half the calls in 30 seconds fail, stop calling it for a while and send elsewhere or hold the work in the queue.

Silent loss is the hardest: the provider says "accepted" and nothing arrives. Watch delivery receipts from the provider webhooks, and alert when the delivered rate for one provider drops well below its usual level. For login codes, users notice before you do; give them a "resend code" button that goes through the high-priority queue.

#Bottlenecks and failures

At 10x, campaigns need about 280,000 sends a second and the log takes about 270 TB over 90 days. Senders and routers are stateless and scale out. The first real wall is the providers: their rate limits do not grow because yours did. Spread load across several provider accounts or providers, and stretch campaigns over longer windows. The preference cache comes next, read on every send: replicate it and keep a few seconds of copies in each router's memory.

#What a strong answer sounds like

#Follow-up questions

#How do you send a campaign to 50 million users?

Do not make 50 million API calls. A campaign job reads the segment in pages, creates notifications in batches, and feeds the bulk queues at a set pace, 28,000 a second for 30 minutes. It records progress so it can resume after a crash without starting over.

#How do you handle device tokens that stop working?

Platform push services report invalid or expired tokens. Delete those tokens at once; sending to dead tokens wastes capacity and can count against you with the platform.

#A user gets the same alert from three services. How do you stop it?

Add a short dedupe window per user and topic in the router: if an alert with the same group key went out in the last 10 minutes, merge or skip it. Better, give each event one owner that calls the notification service.

#How do you send at 9 a.m. in each user's time zone?

Store scheduled notifications by send time, rounded to the minute. A scheduler reads the current minute's bucket and enqueues it. Users spread across time zones spread the load naturally.

#How do you know the system is healthy?

Track queue age per channel and priority (how old is the oldest job), send and delivery rates per provider, and retry and dead-letter counts. Alert on high-priority queue age above a few seconds; see Observability.