Load Balancer Design: L4/L7 Routing, Health Checks, and Failover

Every scalable system has a load balancer — but most candidates treat it as a box to draw and move on. Interviewers use load balancer questions to probe whether you understand traffic distribution, failure isolation, and the difference between routing policy and health policy.

A load balancer solves three distinct problems:

1. Horizontal scaling: distribute incoming requests across N backend instances so no single instance is overloaded.
2. Failure isolation: detect unhealthy backends and stop sending traffic to them before clients notice.
3. Policy enforcement: apply routing decisions based on request content (L7), connection metadata (L4), or geographic proximity — not just availability.

The fundamental tradeoff that every interview question probes: more intelligent routing requires more context, which requires more CPU and state. An L4 load balancer makes a routing decision in microseconds with zero knowledge of the application. An L7 load balancer can route based on URL path, HTTP headers, and JWT claims — but it must terminate TLS, parse the HTTP request, and maintain connection state. That overhead is worth it for API routing, session affinity, and canary deployments — and it is not worth it for raw TCP throughput at 10M+ connections/sec.

What most candidates miss: the load balancer itself is a single point of failure unless you design for it. Active-active pairs with BGP Anycast or ECMP eliminate this; single-LB deployments move the SPOF from the backend to the LB tier.

L4 (Transport Layer) load balancing operates at the TCP/UDP level. It sees source IP, destination IP, and port — nothing about the application payload. Routing decisions are made by rewriting the destination IP in the TCP SYN packet (DSR mode) or establishing a new TCP connection to the backend (proxy mode). Because it never reads the HTTP headers, TLS session, or URL path, it is extremely fast: HAProxy in L4 mode handles 10M+ concurrent connections on commodity hardware. AWS NLB operates at L4.

L7 (Application Layer) load balancing terminates the TLS connection, reads the full HTTP request (headers, path, body if needed), applies routing rules, and establishes a new connection to the chosen backend. This requires significantly more CPU (TLS termination is expensive) but enables: path-based routing (/api → API cluster, /static → CDN origin), header-based routing (canary deployments via X-Canary: true), WebSocket upgrade handling, gRPC routing, and request-level observability. AWS ALB, Nginx, HAProxy in HTTP mode, and Envoy operate at L7.

The interview answer: use L4 for raw throughput (gaming, streaming, large-file transfer), use L7 for anything HTTP/gRPC where you need routing flexibility or observability. In practice, most internet-facing APIs use L7 with TLS termination at the LB; internal service-to-service traffic uses L4 or service mesh (Envoy/Linkerd) for lower overhead.

Round Robin assigns requests sequentially: request 1 → backend A, request 2 → backend B, request 3 → backend C, request 4 → backend A, etc. Simple and zero-state. Works well when all backends are identical and requests have similar processing time. Fails when: backends have different capacity (CPU/memory), requests have highly variable cost (mix of fast and slow queries), or any backend is slow (round-robin happily sends traffic to a 2000ms-latency backend at the same rate as a 5ms backend).

Weighted Round Robin adds a weight to each backend (backend A weight=2, backend B weight=1), so A gets 2x the traffic. Use for heterogeneous hardware, canary deployments (new version at weight=5%, old at weight=95%), and blue-green rollouts.

Least Connections routes each new request to the backend with the fewest active connections. Better than round-robin for variable-length requests (database queries, file uploads). Requires state: each LB instance must track connection counts per backend. Degrades when backends have different connection-handling throughput.

Power of Two Choices (P2C): randomly sample 2 backends from the pool, route to the one with fewer active connections. Statistically, P2C achieves near-optimal load distribution (within a log-log factor of optimal) with O(1) overhead and avoids the thundering herd problem where all LB instances simultaneously flood the "least loaded" backend. This is the algorithm used in Nginx's least_conn with random sampling, and it is the best general-purpose algorithm for stateless APIs.

Consistent Hashing: hash the request key (client IP, session ID, user ID) to a ring; always route the same key to the same backend. Required for: WebSocket servers (connection must stay on one backend), stateful game servers, and cache-warming scenarios where routing the same user to the same pod avoids cache misses. The downside: when backends are added or removed, ~1/N of keys remap (with virtual nodes to reduce hotspots). Not suitable for stateless APIs.

Health check design determines your failure detection window. A typical health check config: interval=10s, unhealthy_threshold=3, healthy_threshold=2, timeout=5s. This means: a backend that fails needs 3 consecutive failures × 10s interval = 30 seconds before it is removed from the pool. During those 30 seconds, the LB continues sending traffic to the failing backend. Design your retry budget accordingly: if health check detection is 30s, your clients must retry for at least 30s, or your SLO takes the hit.

Health check types by depth:

• TCP connect (L4): just verifies the port is accepting connections. Does not detect application-level failures (worker pool exhausted, DB connection pool full). Use as a baseline, not a sole health signal.
• HTTP 200 check (L7 shallow): GET /health returns 200. Fast but still surface-level — a 200 can be returned while the handler is stuck behind a slow mutex.
• Readiness probe (L7 deep): /ready performs a lightweight DB ping, cache ping, and dependency check. Returns 200 only when the backend can actually serve traffic. Use for Kubernetes readiness gates; use shallow health for liveness.

Connection draining (AWS: "deregistration delay") is critical for zero-downtime deploys. When a backend is removed from the pool (deploy, scale-down, or health failure), the LB: (1) immediately stops sending new requests to that backend, (2) waits up to drain_timeout (typically 30–60s) for in-flight requests to complete, then (3) forcefully closes remaining connections. Without connection draining, a deploy that removes a pod mid-request causes in-flight requests to fail with TCP RST. This is the most common source of deploy-time 502/503 errors in production.

Sticky sessions (session affinity) routes requests from the same client to the same backend for the duration of a session. Implemented via: a cookie inserted by the LB (AWSALB in AWS ALB, SERVERID in HAProxy), or consistent hashing on client IP (less reliable due to NAT/proxies). Sticky sessions create uneven load distribution and complicate deploys (draining a sticky backend strands all its sessions). Prefer stateless backends with session state in Redis; use sticky only when the application cannot be made stateless (legacy apps, WebSocket).

Global load balancing directs users to the nearest or healthiest data center before traffic hits your infrastructure.

GeoDNS returns different DNS A records based on the client's resolver location. A user in Europe resolves api.example.com to the EU region's IP; a user in Asia resolves to the APAC region's IP. Pros: simple to implement, works with any CDN. Cons: DNS TTL means failover takes 60–300 seconds (users cached the old IP); clients using non-local DNS resolvers (Google 8.8.8.8) are misrouted based on resolver location, not client location.

Anycast advertises the same IP block from multiple PoPs via BGP. Routers automatically send traffic to the topologically nearest PoP that announces the route. Failover is near-instant (BGP reconvergence in seconds, not DNS TTL minutes). Anycast is how Cloudflare, Google Cloud Load Balancing, and AWS Global Accelerator operate. It requires BGP infrastructure (AS number, IP blocks) — not available to most startups but standard at hyperscale.

Active-Active vs Active-Passive regions: active-active serves traffic from all regions simultaneously — lower latency but requires data replication to be read-correct in all regions. Active-passive keeps one region on standby; failover requires promoting the passive region, which adds latency to failover and requires careful data sync management. Prefer active-active for read-heavy workloads (most traffic is reads); active-passive for write-heavy workloads where conflict resolution is expensive.