Load Balancing & Proxies

A single instance of your app can only handle so much traffic, and if it dies, you’re down. The answer is to run many copies — but now a request arriving at app.example.com has to be sent to one of them, and that one had better be alive. A load balancer is the traffic cop that makes many servers look like one, and a proxy is the more general pattern it’s built on: a middleman that stands between client and server. This is the layer that turns “a server” into “a service.”

The problem: one name, many servers

DNS points app.example.com at a single address. But you want ten identical app instances behind it, with traffic shared among them and dead ones skipped. So you point the name at a load balancer, and it fans requests out:

                                   +--> app instance 1  (healthy)
   client --> [ Load Balancer ] ---+--> app instance 2  (healthy)
                  picks one         +--> app instance 3  (DEAD — skipped)
                                   +--> app instance 4  (healthy)

This buys two things at once: scale (more instances = more capacity) and availability (one instance can die without taking the service down). Both depend on the balancer knowing which instances are healthy — which we’ll get to.

L4 vs L7: how deep does it look?

Load balancers come in two flavours, named for the stack layer they operate at:

	L4 (transport)	L7 (application)
Sees	IP + port, raw TCP/UDP	full HTTP — paths, headers, cookies
Decides by	connection-level info	request content
Speed	very fast, low overhead	slightly slower, far smarter
Can do	spread connections	route /api vs /img, rewrite headers, terminate TLS

An L4 balancer just forwards TCP connections — fast and protocol-agnostic, but blind to what’s inside. An L7 balancer speaks HTTP, so it can route /api/* to one pool and /static/* to another, do sticky sessions by cookie, and terminate TLS. Most web traffic uses L7 because the smarts are worth the small cost; L4 shines for raw throughput and non-HTTP protocols.

Algorithms: which backend gets the request

Once it’s decided to forward, the balancer picks a backend by some rule:

   Round robin        1, 2, 3, 4, 1, 2, ...   (simple, even rotation)
   Least connections  send to whoever's least busy right now
   IP hash            same client -> same backend (sticky sessions)
   Weighted           bigger servers get a larger share

Round robin is the sensible default. Least connections handles uneven request durations better. IP hash gives stickiness when a backend holds per-user state (though stateless apps, which avoid this need, are preferred — a theme that returns in containers).

Health checks: the automation that matters most

Here is the heart of the page. A balancer must not send traffic to a dead instance — so it constantly health-checks each backend and removes failing ones from rotation automatically:

   every few seconds:
     LB ->  GET /healthz  -> instance      200 OK  -> keep in rotation
     LB ->  GET /healthz  -> instance      timeout -> REMOVE from rotation
                                                       (re-add when it recovers)

Think about what this replaces. Without it, when an instance crashes at 3 a.m., a human gets paged, SSHes in, and manually pulls the box from rotation while users hit errors. With it, the balancer notices in seconds and reroutes — no page, no errors, no human. This is the recurring thread in its purest form: a manual, error-prone, slow step replaced by a tight automated loop, and production gets dramatically safer for it. Your job becomes writing a good /healthz endpoint — one that actually checks dependencies, not just “the process is up.”

Forward vs reverse proxy

A proxy is any middleman between client and server. The direction it faces is what splits the two kinds, and the names confuse everyone exactly once:

   FORWARD PROXY  (sits in front of CLIENTS)
   your laptops --> [ proxy ] --> internet
   hides/controls who is going OUT  (corporate egress filter, VPN)

   REVERSE PROXY  (sits in front of SERVERS)
   internet --> [ proxy ] --> your app servers
   hides/protects what's behind it  (load balancer, TLS terminator, cache)

A forward proxy represents the client (it’s the gateway your office traffic goes out through). A reverse proxy represents the server (clients think they’re talking to your app; they’re really talking to the proxy). A load balancer is a reverse proxy. So is a CDN edge node. As a DevOps engineer, the reverse proxy is your bread and butter.

TLS termination

A favourite reverse-proxy job: TLS termination. The proxy holds the certificate, does the TLS handshake with the client, and forwards plain HTTP to the backends over the trusted internal network.

   client  --HTTPS-->  [ reverse proxy ]  --HTTP-->  app instances
                        (holds the cert,            (no cert needed,
                         decrypts here)              don't even know about TLS)

The payoff: certificates live and rotate in one place instead of on every instance — fewer things to get wrong, one place to automate renewal. (In higher-security setups you re-encrypt to the backends too, which we’ll meet as mTLS in service meshes.)

nginx: the canonical example

nginx is the reverse proxy you’ll see everywhere — as a load balancer, TLS terminator, and static file server. A minimal config shows every concept above in one place:

upstream app {
    least_conn;                  # algorithm
    server 10.0.1.10:8080;       # backend instances
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
}

server {
    listen 443 ssl;                          # terminate TLS here
    ssl_certificate     /etc/ssl/app.crt;
    ssl_certificate_key /etc/ssl/app.key;

    location / {
        proxy_pass http://app;               # reverse-proxy to the pool
        proxy_set_header Host $host;         # preserve the original Host header
    }
}

That’s a reverse proxy, an L7 load balancer with an algorithm, and TLS termination — in fifteen lines. The same patterns power cloud load balancers (AWS ALB/NLB), Kubernetes Services and Ingress, and Envoy. Learn them here and they recur everywhere.

The thread

The manual step a load balancer removes is the human in the failover loop. Before health-checked balancing, taking a sick instance out of rotation meant a pager, an SSH session, and downtime while someone reacted. The balancer does it continuously, in seconds, with no one awake. That is what makes running many cheap, fallible instances safer than running one precious server — failure stops being an event and becomes routine.

By the numbers — how fast a health check ejects a dead backend

The page’s claim is that automation beats a human in the failover loop. Quantify it. A balancer’s health check has two knobs: how often it probes, and how many failures in a row mark a backend “down.” With typical production values:

   probe interval        = 2 s
   unhealthy threshold   = 3 consecutive failures
   time to eject         = 2 s x 3  =  ~6 s before traffic stops flowing to the dead node

Six seconds, untouched by human hands. Compare the manual path: an alert fires, a pager wakes someone, they find a laptop, VPN in, SSH to the balancer, and pull the node — minutes at best, and every one of those seconds is users hitting errors. The automated loop turns a multi-minute, error-prone scramble into a single-digit-second non-event. (The knobs trade off: too aggressive and a brief blip ejects a healthy node; too slow and a dead node serves errors longer.)

The architect’s lens

Step back from the nginx config and answer the five questions that decide whether to put a balancer in front of your service:

• Why does it exist? Because one name (app.example.com) must map to many identical instances, and a request had better land on a live one — the balancer is the reverse proxy that makes many servers look like one, turning “a server” into “a service.”
• What problem does it solve? Two at once: scale (more instances = more capacity) and availability (one instance can die without taking the service down). The load-bearing piece is the health check — a GET /healthz every few seconds that ejects a dead backend in ~6 s (2 s interval × 3 failures), replacing the human-in-the-failover-loop with an automated loop.
• What are the trade-offs? L4 vs L7 is the central one: L4 sees only IP+port and is very fast but blind; L7 parses full HTTP so it can route /api vs /static and terminate TLS, at slightly higher cost. Health-check knobs trade too — too aggressive ejects healthy nodes on a blip, too slow serves errors longer.
• When should I avoid it? When you don’t need it: a single instance with no HA requirement adds a hop and a component to operate for nothing. Prefer stateless apps so you can use plain round-robin and skip IP-hash stickiness, which couples a user to one backend.
• What breaks if I remove it? Failover goes back to a pager, an SSH session, and downtime while someone manually pulls a sick box; certificates sprawl back onto every instance instead of rotating in one place via TLS termination; and “many cheap fallible instances” stops being safer than one precious server.

Check your understanding

1. What two distinct benefits do you get from putting many instances behind a load balancer, and how does each one depend on health checks?
2. Contrast L4 and L7 load balancers. Give one thing an L7 balancer can do that an L4 one fundamentally cannot, and why.
3. Explain forward vs reverse proxy by who each one represents. Which kind is a load balancer?
4. Walk through what a health check replaces. What manual, error-prone step disappears, and what does your job become instead?
5. What is TLS termination, and why does doing it at the reverse proxy make certificate management safer?

Show answers

1. Scale (more instances means more capacity) and availability (one instance can die without taking the service down). Both depend on health checks: scale is only usable if traffic actually spreads to live instances, and availability only holds if the balancer detects and skips the dead ones.
2. An L4 balancer sees only IP + port and forwards raw TCP/UDP; an L7 balancer parses full HTTP. Only L7 can route by request content — e.g. send /api/* to one pool and /static/* to another, or do cookie-based stickiness — because L4 can’t see inside the connection to read the path or headers.
3. A forward proxy represents the client (your laptops go out through it — corporate egress, VPN); a reverse proxy represents the server (clients think they’re talking to your app but reach the proxy). A load balancer is a reverse proxy.
4. It replaces the human in the failover loop: without it, a crashed instance means a pager, an SSH session, and downtime while someone manually pulls the box from rotation. With it, the balancer detects the failure and reroutes in seconds. Your job shifts to writing a good /healthz endpoint that truly checks dependencies, not just “the process is up.”
5. TLS termination is the reverse proxy holding the certificate and doing the TLS handshake with the client, then forwarding plain HTTP to backends over the trusted internal network. It’s safer because the certificate lives and rotates in one place instead of on every instance — one thing to automate, one place to get right.