How Load Balancers Distribute Traffic Across Servers

One server handling all your traffic works fine – until it does not. Traffic spikes, the server gets overwhelmed, response times climb, and users start seeing errors. Adding more servers solves the capacity problem, but then you have a new one: how does incoming traffic know which server to go to?

A load balancer fixes that. It takes every incoming request before it reaches your servers and picks which one should handle it. Your servers never deal with users directly – the load balancer is the only thing users ever actually connect to.

This article walks through how that process works, what algorithms load balancers use to make routing decisions, and a few things worth getting right before you deploy one.

Prerequisites

• You already run at least one server with a web app or API on it.
• NNetworking basics like IP addresses and HTTP are not new to you.
• You have thought about running multiple servers but are not sure how traffic should reach them.

What a Load Balancer Actually Does

A load balancer does three things, really:

1. Takes the incoming request before any server sees it.
2. Picks a backend server based on whatever algorithm you have configured.
3. Passes the request along and returns the response to the user.

Nothing complicated from the user’s side – they hit your domain, they get a response. They have no idea how many servers are behind it or which one just handled them.

From the user’s perspective, nothing unusual happens. They hit your domain, they get a response. They have no idea whether that response came from server one, server two, or server ten.

Beyond routing, most load balancers also run health checks on backend servers. If a server stops responding, the load balancer pulls it out of rotation and stops sending it traffic — automatically, without anyone doing anything manually.

How Traffic Gets Distributed

The method a load balancer uses to pick a server is called a load balancing algorithm. Different algorithms suit different workloads. Here are the main ones.

Round Robin

The simplest approach. Requests go to each server in order – first request to server one, second to server two, third to server three, then back to server one again.

It works well when your servers are identical and your requests are roughly the same size. The problem shows up when requests are not equal. A request that kicks off a heavy database query takes far longer than one serving a cached response. Round robin keeps rotating regardless – the slower server keeps getting new requests while it is still grinding through the old ones.

Weighted Round Robin

Round robin with a twist – you tell the load balancer that some servers can handle more than others. A server with weight three gets three requests before the next server gets one.

This comes in handy when your servers are not the same spec. Maybe you upgraded two of them but kept an older one running. Rather than sending equal traffic to unequal machines, you dial in weights that reflect what each one can actually handle.

Least Connections

Here the load balancer checks how many open connections each server is currently holding and sends the next request to the one with the least. No fixed rotation, just a live look at who is least occupied right now.

When requests finish at different speeds, this naturally evens things out. If some requests take much longer to process than others, least connections naturally compensates – slower servers accumulate fewer new connections because they are already handling longer ones.

IP Hash

The load balancer hashes the user’s IP address and always sends that IP to the same backend server. Same user, same server, every time.

This matters when your application stores session data locally on the server rather than in a shared database or cache. Without IP hash, a user might hit server one for their first request, then server two for the next – and server two has no idea who they are.

Least Response Time

The load balancer watches actual response speeds and factors in active connections together. New requests go to whichever server is performing best at that moment.

It takes more work to run than the simpler algorithms, but in setups where servers behave differently under heavy load it gets closer to genuinely optimal distribution.

Layer 4 vs Layer 7 Load Balancing

Load balancers work at different depths, and that changes what they can actually do with a request.

• Layer 4 operates at the transport level – it sees IP addresses and ports, nothing else. It routes fast and asks no questions about what is inside the request. Simple, low overhead, but limited.
• Layer 7 reads the full HTTP request – the URL, headers, cookies, hostname. That opens up much smarter routing. Send /api traffic to your API servers and /uploads to a different pool entirely. Route based on which subdomain the request came in on. None of that is possible at Layer 4 because it never looks that deep.

For most web apps, Layer 7 is the right choice. Layer 4 makes sense when you are moving enormous amounts of traffic and every millisecond of processing overhead matters.

Health Checks

A load balancer is only as good as its awareness of what is actually running. Health checks are how it keeps track.

At a basic level, a health check pings each server on a regular interval. If the server does not respond, it gets marked as unhealthy and removed from the pool. When it recovers, it gets added back.

A better health check does more than ping – it hits a real endpoint on your application and checks the response. A server can be reachable but still broken if the application process crashed. Checking an actual HTTP endpoint catches that.

Key settings to configure:

• Check interval – how often to run the check. Every 10 to 30 seconds is typical.
• Failure threshold – how many consecutive failures before marking a server unhealthy. Usually two or three, to avoid pulling a server for a single blip.
• Recovery threshold – how many consecutive successes before bringing it back. Prevents a flapping server from rejoining too quickly.

Sticky Sessions

Some applications expect the same user to always land on the same server – because session data, uploaded files, or cached state lives locally on that server rather than somewhere shared.

Sticky sessions solve this by tagging a user’s requests so they always go to the same backend. The load balancer sets a cookie on the first request, and from then on it uses that cookie to route the user to the same server.

It works, but it creates an uneven distribution problem. If ten users all get sticky to server one and it gets busy, they all suffer – the load balancer cannot move them somewhere healthier without breaking their session.

The cleaner long-term solution is to move session state out of individual servers and into a shared store like Redis. Then any server can handle any request, sticky sessions become unnecessary, and load balancing works the way it is supposed to.

A Practical Setup

A common production setup on VPS looks like this:

Users
  |
  ▼
Load Balancer (HAProxy or Nginx)   ← single entry point
  |           |           |
  ▼           ▼           ▼
App Server 1  App Server 2  App Server 3
  |           |           |
  └───────────┴───────────┘
              |
              ▼
        Database Server

The load balancer runs on its own VPS instance and handles all incoming traffic. The app servers sit behind it and never receive direct public traffic. The database server is only reachable from the app servers – no public exposure at all.

HAProxy is a solid choice for this setup. Here is a minimal HAProxy configuration that uses least connections across three backend servers:

frontend web_frontend
    bind *:80
    default_backend app_servers
backend app_servers
    balance leastconn
    option httpchk GET /health
    server app1 192.168.1.10:80 check
    server app2 192.168.1.11:80 check
    server app3 192.168.1.12:80 check

The option httpchk line tells HAProxy to hit /health on each server to confirm it is actually responding, not just reachable. If a server fails that check, HAProxy stops sending it traffic until it recovers.

Conclusion

Load balancers do more than split traffic – they make your infrastructure resilient. A server can go down and users never notice because the load balancer simply stops sending it requests. You can add capacity by adding servers without changing anything users interact with. You can drain a server for maintenance without downtime.

The algorithm you choose matters less than most people think when you are starting out. Round robin or least connections handles the majority of real workloads well. What matters more is getting health checks right, deciding early whether session state lives on the server or somewhere shared, and making sure your load balancer itself is not a single point of failure – because a load balancer that goes down takes everything with it.