Load Balancing

Launch day: the API expects 3,000 requests a second at peak, and one app server handles about 1,000 before its latency climbs. So you need at least three servers. But clients know one address, and when one of those servers crashes mid-afternoon, somebody has to stop sending it traffic.

#The idea

A load balancer sits in front of many identical servers and owns the public address; it picks a server for each request or connection. That gives you two things at once: capacity, because you scale by adding servers behind it, and availability, because a dead server is taken out of the pool.

It costs an extra hop (under a millisecond inside a data center), one more component that must not fail, and a rule: any server must be able to handle any request. That rule is what "stateless" means.

#Spreading requests and surviving a crash

Run four servers, not three, so losing one still leaves enough. Each then gets 3,000 ÷ 4 = 750 requests a second. Step through round robin, then a crash.

Health checks come in two kinds. Active checks call an endpoint on a timer, as above. Passive checks watch real traffic and eject a server that returns errors or times out, which catches problems faster.

#Layer 4 or layer 7

#Choosing a server

Hashing is where consistent hashing comes in: with a plain hash % n, going from 4 to 5 servers moves about 4 keys in 5 to a different server.

#Sticky sessions and stateless servers

If server A keeps your shopping cart in its own memory, every request from you must reach A. Balancers can do that with sticky sessions, usually a cookie naming the server. But load gets uneven, and when A dies the carts on it are gone.

The better answer is stateless servers: keep session data in a shared store (an in-memory store such as Redis) or in a signed token the client sends. Then any server can handle any request, a crash loses nothing, and you add or remove servers freely.

#When the balancer is the weak point

One balancer in front of everything is a single point of failure. The usual fix is the pair in the first diagram: an active and a standby that share a virtual IP, with the standby taking the IP when heartbeats stop. At larger scale, run several active balancers and spread clients over them with DNS returning several addresses, or with one address announced from many places (anycast). Managed cloud balancers already do this; say so rather than drawing it.

Reverse proxies and API gateways sit in the same spot. A reverse proxy receives requests on behalf of servers: it terminates TLS, compresses and caches. A layer 7 load balancer is a reverse proxy that spreads load. An API gateway is a reverse proxy with API concerns added: authentication, rate limiting, and routing /orders and /users to different services. In a diagram, one box labelled "API gateway / load balancer" is fine.

#Numbers worth remembering

• Worst-case detection time is check interval × failures needed: 5 s × 2 = about 10 s.
• Servers needed: peak requests ÷ per-server capacity, rounded up, plus at least one spare.

#In the interview

Draw a load balancer as soon as you have more than one server, then name the choices it implies: "Layer 7, round robin, health checks every few seconds, and the app servers are stateless, with sessions in a shared store, so I can scale them freely."

Follow-ups to expect:

• "What if the load balancer dies?" Active-standby pair sharing a virtual IP, or several active ones behind DNS or anycast.
• "How do you deploy without dropping requests?" Take a server out of the pool, let in-flight requests finish (draining), update it, put it back.
• "WebSockets behind a balancer?" Least connections, since each connection lasts hours, and plan for clients reconnecting when a server goes away.

#Common mistakes

#What to practise next

Once requests reach your servers, the next win is not doing the same work twice: Caching. To see a balancer in front of separate read and write paths, work through Design a URL Shortener.