Realtime Communication

A chat app has 1 million people with the app open. To see new messages, each phone asks the server "anything new?" every 5 seconds. That is 1,000,000 ÷ 5 = 200,000 requests a second. About 2,000 new messages arrive a second in total, so at most 2,000 of those 200,000 answers carry anything: 99% of the work returns "no". And a message still waits 2.5 seconds on average before anyone sees it.

The fix is to turn the direction around: keep a connection open and let the server speak first. That is cheap for the network and expensive for your architecture, because now your servers remember who is connected to them.

#The idea

HTTP is built for the client to ask and the server to answer. Realtime features need the server to send when something happens. There are four ways to get there, from "ask often" to "hold a two-way pipe open":

• Short polling: the client asks on a timer.
• Long polling: the client asks, and the server holds the request open until there is news or about 30 seconds pass. Then the client asks again at once. Messages that arrive between two polls must be fetched with a cursor ("everything after message 9001"), or they are missed.
• Server-sent events (SSE): one HTTP response that never ends; the server writes events into it as they happen. One direction only, server to client, text only. Browsers reconnect automatically and send the last event id they saw, so a server that tags events with ids can resume.
• WebSocket: starts as an HTTP request asking to upgrade, then becomes a two-way channel of messages over the same TCP connection. Nothing is built in for reconnecting or replaying missed messages; you write that.

For chat, pick WebSockets: both sides talk often. Pick SSE when the client only listens. Keep short polling for things that change every few minutes; a million open connections for one update an hour is waste too.

#How open connections change the design

With request and response, any server can answer any request. With a WebSocket, Ben's connection lives on one particular server for hours. When Ana sends Ben a message, it lands on Ana's server, and something must get it to Ben's. That one fact reshapes the system:

• Connection servers become their own tier. They hold sockets and pass messages through, nothing else, so business logic can be deployed without dropping a million connections.
• Pub/sub routes messages between servers. When Ben connects, his server subscribes to a channel user:ben. Anyone with a message for Ben publishes to that channel. The alternative is a registry ("Ben is on server 7") in a shared store. Pub/sub keeps that registry for you: a dead server's subscriptions vanish with it.
• The database stays the source of truth. Basic pub/sub delivers only to whoever is subscribed at that moment, at most once. Store the message first, so a client that was offline can catch up.

The random wait in step 7 matters. Server 2 may have held 50,000 sockets. If all reconnect in the same second, they hit the rest of the fleet like a small attack. Spread over 30 seconds, it is 50,000 ÷ 30 ≈ 1,700 a second.

Long-lived connections bring two chores. Heartbeats: send a small ping every 30 seconds or so, because a phone that loses signal does not close its socket, and idle connections get cut by proxies. Deploys: drain a server by telling clients to reconnect elsewhere, a slice at a time, instead of killing it with everyone attached.

#Numbers worth remembering

• Polling load = users ÷ interval in seconds.
• An idle connection costs memory, not CPU. Assume a scenario number such as 50,000 per server and say you would load test it.
• Heartbeat load = connections ÷ heartbeat interval: 2 million connections pinging every 30 seconds is about 67,000 small messages a second.

#In the interview

Bring this up as soon as a requirement says "instantly", "live" or "as it happens": chat, notifications, a driver's position, a shared document. Pick the transport, then spend your time on the hard part, which is routing and recovery. Phrases that show judgement: "Users both send and receive, so WebSockets." "Connection servers are stateful, so I keep them thin." "Pub/sub is fire-and-forget, so the store is the truth and clients resync by last message id."

Follow-ups to expect:

• "A user has a phone and a laptop." Two connections, possibly on two servers, both subscribed to the same user channel, so both get the message.
• "How do you balance WebSocket traffic?" Least connections, long idle timeouts on the balancer, and draining on deploy.
• "Are messages in order?" Within one conversation, yes: the store gives each message the next id in that conversation and clients sort by it. Global order is not needed.
• "What if a publish is lost while Ben is online?" His app sees a gap, 9003 after 9001, and fetches 9002 from the store. A lost last message leaves no gap, so apps also sync when opened.

#Common mistakes

#What to practise next

Open connections invite abuse and floods, which is where Rate Limiting comes in. To put all of this into a full design, work through Design a Chat App.