I Killed a Live Server. The Site Survived.

We recorded this live: two tiny web servers, nginx in front as a load balancer, a request loop printing every answer — then we killed server 1 with kill -9. Every response after that came from server 2. The failed-request counter stayed at zero. When we restarted server 1, it rejoined the rotation within ~2 seconds.

That is what a load balancer does: it spreads traffic across servers and routes around dead ones. This guide builds the exact same setup on your laptop in about 10 minutes. You need Python 3 and Docker — nothing else.

The three files

Make a folder and create these three files.

server.py — a 10-line web server that answers with its own name (you’ll run it twice):

import sys
from http.server import HTTPServer, BaseHTTPRequestHandler

NAME, PORT = sys.argv[1], int(sys.argv[2])

class H(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.end_headers()
        self.wfile.write(f"{NAME}\n".encode())
    def log_message(self, *args):
        pass

print(f"{NAME} listening on :{PORT}")
HTTPServer(("0.0.0.0", PORT), H).serve_forever()

nginx.conf — the load balancer. The upstream block lists both servers; with no method specified, nginx uses round robin by default:

worker_processes 1;
events { worker_connections 128; }
http {
    upstream app {
        server host.docker.internal:8001 max_fails=1 fail_timeout=2s;
        server host.docker.internal:8002 max_fails=1 fail_timeout=2s;
    }
    server {
        listen 80;
        location / {
            proxy_pass http://app;
            proxy_connect_timeout 1s;
            proxy_read_timeout 2s;
            proxy_next_upstream error timeout;
        }
    }
}

loop.sh — the heartbeat that proves every claim (colored output + a live failed counter):

#!/bin/bash
TOTAL=0; FAIL=0
while true; do
  R=$(curl -s --max-time 2 http://localhost:8080/)
  TOTAL=$((TOTAL+1))
  case "$R" in
    "SERVER 1") printf "\033[1;32m  SERVER 1\033[0m   total:%-4d failed:%d\n" "$TOTAL" "$FAIL" ;;
    "SERVER 2") printf "\033[1;36m  SERVER 2\033[0m   total:%-4d failed:%d\n" "$TOTAL" "$FAIL" ;;
    *) FAIL=$((FAIL+1)); printf "\033[1;41m  REQUEST FAILED \033[0m total:%-4d failed:%d\n" "$TOTAL" "$FAIL" ;;
  esac
  sleep 0.5
done

Build it

python3 server.py "SERVER 1" 8001 &
python3 server.py "SERVER 2" 8002 &
curl -s localhost:8001   # -> SERVER 1
curl -s localhost:8002   # -> SERVER 2

docker run --rm -d --name lb -p 8080:80 \
  --add-host=host.docker.internal:host-gateway \
  -v "$PWD/nginx.conf":/etc/nginx/nginx.conf:ro nginx:alpine

curl -s localhost:8080   # -> answered THROUGH the load balancer

Start the loop in a second terminal (or a tmux split):

bash loop.sh

You’ll see it alternate — SERVER 1, SERVER 2, SERVER 1, SERVER 2 — with failed:0. That’s round robin, live.

Kill it

kill -9 $(lsof -ti :8001)
curl -s --max-time 2 localhost:8001 || echo "SERVER 1 IS DEAD"

Watch the loop: every answer is now SERVER 2, and failed: stays at 0. Then bring it back:

python3 server.py "SERVER 1" 8001 &

Within ~2 seconds (our fail_timeout=2s), SERVER 1 rejoins the rotation.

How the failover actually works (the interview answer)

Open-source nginx uses passive health checks — it doesn’t ping your servers. When a request to a dead server fails with a connection error, proxy_next_upstream (the default) retries that same request on the next server, so the client never sees the failure. After max_fails failures it benches the dead server for fail_timeout, then tries it again with a real request.

Three honest caveats worth knowing:

1. “Zero downtime” isn’t guaranteed — a kill landing mid-response can fail that one in-flight request, and non-idempotent requests (like POST) are not retried by default.
2. Active health checks (regular pinging) are a paid NGINX Plus feature — open-source nginx only reacts to real failed requests.
3. This is an L7 (HTTP) load balancer. Big sites layer DNS, anycast, and L4 balancers on top of this idea — but the core mechanism you just built is the same one keeping them alive.

Clean up

kill -9 $(lsof -ti :8001) $(lsof -ti :8002) 2>/dev/null
docker rm -f lb