Traffic Cop: The Algorithms Deciding Which Node Gets Your Request

Sep 3

Every request that reaches a distributed system passes through a decision point most users never think about: which of the N available backend nodes should actually handle this? Get that decision wrong at scale and you get a cluster where one node is melting down while three others sit idle. Load balancing is the discipline of making that decision correctly, millions of times a second.

The Two Layers of Load Balancing

Before the algorithm question, there's a placement question: where does the load balancer actually sit?

Layer 4 (Transport): Operates on IP and TCP/UDP information alone — source/destination address and port. Fast, low-overhead, but blind to the actual content of requests. It can't tell a lightweight health-check ping from a heavyweight video upload; it just routes packets to a backend.
Layer 7 (Application): Operates on the actual HTTP request — path, headers, cookies, even payload. This lets it make smarter routing decisions (send /api/inference to your GPU pool, send /static/* to a cheap cache tier), at the cost of having to actually parse and terminate the connection.

L4 Load Balancer:
[Client] --TCP-------------> [LB: routes by IP/port] --> [Backend]
(doesn't see HTTP path or headers)

L7 Load Balancer:
[Client] --HTTP GET /api/inference--> [LB: parses request] --> [GPU Pool]
[Client] --HTTP GET /static/logo.png-> [LB: parses request] --> [Cache Tier]

Most production stacks run both — an L4 balancer (often hardware or cloud-provided) in front, distributing raw connections across a fleet of L7 balancers (like nginx or Envoy) that then make the smarter, content-aware routing call.

Load Balancing Algorithms

Once you know where the decision happens, the next question is how it's made. A handful of algorithms cover most real deployments:

Algorithm

How It Decides

Best Fit

Weak Point

Round Robin

Cycles through backends in fixed order

Uniform backends, uniform request cost

Ignores actual backend load

Weighted Round Robin

Round robin, but bigger/faster nodes get proportionally more requests

Heterogeneous hardware (mixed GPU tiers)

Weights are static, not adaptive

Least Connections

Sends the request to whichever backend has the fewest active connections

Long-lived or variable-duration requests

Requires the LB to track live connection state

Least Response Time

Combines active connections with recent latency

Latency-sensitive workloads

More overhead to compute per request

Consistent Hashing

Routes based on a hash of the request (e.g., session ID, user ID)

Sticky sessions, cache-affinity workloads

Uneven load if hash keys aren't well distributed

Random

Picks a backend uniformly at random

Surprisingly competitive at scale with "power of two choices"

Pure random alone can create hot spots

The Power of Two Choices

One of the more counterintuitive results in load balancing research: you don't need perfect global knowledge of every backend's load to get near-optimal distribution. Picking two random backends and sending the request to whichever has less load performs almost as well as tracking full global state, at a fraction of the coordination cost. This result (Mitzenmacher, 1996) is why so many production load balancers — including Envoy's default policy — use "P2C" rather than exhaustive least-connections scanning across a large fleet.

import random
from dataclasses import dataclass

@dataclass
class Backend:
node_id: str
active_connections: int = 0

def power_of_two_choices(backends: list[Backend]) -> Backend:
"""Pick two random backends, route to whichever is less loaded."""
a, b = random.sample(backends, 2)
return a if a.active_connections <= b.active_connections else b

# Simulate routing 1000 requests across a 10-node fleet
backends = [Backend(node_id=f"node-{i}") for i in range(10)]

for _ in range(1000):
chosen = power_of_two_choices(backends)
chosen.active_connections += 1
# In production, connections would later decrement as requests complete

loads = sorted(b.active_connections for b in backends)
print(loads) # Load spreads far more evenly than pure random assignment

Health Checks: Load Balancing's Prerequisite

None of these algorithms matter if the load balancer is routing traffic to a dead node. Health checking is the unglamorous mechanism that makes every routing decision above actually safe:

Active Health Checks: The load balancer periodically pings each backend (GET /healthz) on its own schedule, independent of real traffic, and removes non-responsive nodes from rotation.
Passive Health Checks: The load balancer watches real traffic for failures — timeouts, 5xx responses — and ejects a backend after a threshold of consecutive failures, without needing a dedicated probe.

Most production systems run both: active checks catch a node that's fully down before it receives live traffic, and passive checks catch degraded-but-technically-responding nodes that active checks might miss.

Load Balancing Meets Distributed State

Load balancing gets genuinely hard the moment backends aren't stateless. If a user's session, an in-progress model context window, or cached conversation state lives on a specific node, routing that user's next request to a different node means that state has to be fetched, replicated, or reconstructed — the same reconciliation problem CRDTs and consensus protocols solve at the data layer.

This is where sticky sessions (routing based on a hash of session ID, cookie, or client IP) trade load-balancing efficiency for state locality: a node might get more than its fair share of traffic, but it's traffic for sessions it already has warm context for. For local inference clusters serving long-running agent sessions or multi-turn conversations, this trade-off often matters more than perfectly even CPU utilization.

Choosing an Approach

There's no universally correct algorithm — only a correct algorithm for your traffic shape:

Uniform, short-lived requests, homogeneous hardware: Round robin or plain random is often good enough, and simpler to reason about than anything smarter.
Heterogeneous hardware (mixed GPU generations, mixed instance sizes): Weighted round robin or least-connections accounts for the fact that not every node can handle the same load.
Stateful sessions or cache-affinity workloads: Consistent hashing keeps a given user or session pinned to the node that already holds their context.
High-throughput, latency-sensitive services at scale: Power of two choices gives near-optimal distribution without the coordination overhead of tracking full cluster state.

The load balancer is the first architectural decision most requests encounter and the last one most engineers think about — which is exactly why misconfigured load balancing is such a common source of "mysteriously uneven" cluster performance.

http://24-7reporters.com/news/cipher-grid-catalog-spotlight-six-titles-one-fall-reading-list/0561816/

http://5starsdiscovery.com/news/cipher-grid-catalog-spotlight-six-titles-one-fall-reading-list/0561816/

http://actionswift.com/news/cipher-grid-catalog-spotlight-six-titles-one-fall-reading-list/0561816/

http://akhbarharian.com/news/cipher-grid-catalog-spotlight-six-titles-one-fall-reading-list/0561816/

http://allsportstoday.com/news/cipher-grid-catalog-spotlight-six-titles-one-fall-reading-list/0561816/

http://aseancoverage.com/news/cipher-grid-catalog-spotlight-six-titles-one-fall-reading-list/0561816/

http://aseanscoop.com/news/cipher-grid-catalog-spotlight-six-titles-one-fall-reading-list/0561816/

http://asialogue.com/news/cipher-grid-catalog-spotlight-six-titles-one-fall-reading-list/0561816/

http://asiashift.com/news/cipher-grid-catalog-spotlight-six-titles-one-fall-reading-list/0561816/

http://asiaviralnews.com/news/cipher-grid-catalog-spotlight-six-titles-one-fall-reading-list/0561816/

http://bajetharian.com/news/cipher-grid-catalog-spotlight-six-titles-one-fall-reading-list/0561816/

http://bankingreporter.com/news/cipher-grid-catalog-spotlight-six-titles-one-fall-reading-list/0561816/