Infrastructure Insight

The formula

The adaptive load balancer is the reason this stage exists. A cron job reads node_load1 per web server from the Prometheus API and rewrites the Nginx upstream block:

weight = clamp(10 − round(node_load1 × 5), 1, 20)

Lower load produces a higher weight. The archived update sequence is:

1. write the generated block to a temporary file
2. nginx -t: on failure, log, discard, and do not reload
3. mv over the live config
4. systemctl reload nginx for a graceful reload

The updater's sudoers entry names three permitted command forms:

devops ALL=(root) NOPASSWD: /usr/sbin/nginx -t
devops ALL=(root) NOPASSWD: /bin/mv /home/devops/frontend_upstream_servers.conf.tmp /etc/nginx/conf.d/frontend_upstream_servers.conf
devops ALL=(root) NOPASSWD: /bin/systemctl reload nginx

Note the mv rule names both arguments. An earlier version matched only the command, and stopped matching the moment the temporary path changed.

The diagnostic application

The Go backend exposes system information through gopsutil. A bind mount of /etc/os-release provides the host's OS metadata explicitly; a container's hostname is a different piece of information. The Express frontend presents the diagnostic data through the application gateway.

The load experiment

The journal records stress-ng --cpu 1 --timeout 180s on web-server-1. Subsequent updater runs assigned that server a lower weight, with more traffic observed at web-server-2. After stopping the load, weights returned towards a more balanced state. These were observed config and browser checks, without a captured request-count series or benchmark showing a throughput improvement.

A file-restore check

I created important_config.txt on the application server, backed it up, checked its contents on the backup server and deleted the original. An rsync restore then returned the file to the application server. This verifies that file-restoration path, rather than complete recovery of a lost VM or database.

Cron exposed a separate permissions issue when reading protected configuration. The recorded fix configured the remote rsync command to use its permitted sudo path.

Configuration validation sequence

The script writes /home/devops/frontend_upstream_servers.conf.tmp, runs the main nginx -t check, moves the candidate over the live include and then reloads Nginx. Its comment assumes the temporary file is included indirectly, but the script does not explicitly point the test at a configuration containing that candidate. The sequence should not be presented as proof that the new include was validated before installation. It also reports a reload failure after the file has already been replaced, leaving recovery for inspection.

The rule that blocked everything

The intent was to gate the backend API on a secret header, so that only the load balancer could reach it:

proxy_set_header X-Secret-Handshake "I-Am-The-Trusted-Proxy";

sudo iptables -I DOCKER-USER 1 -p tcp --dport 5000 -m string --algo bm \
  ! --string "X-Secret-Handshake" -j DROP

The rule dropped connections before HTTP data was available:

The DROP rule was too effective. It was inspecting the very first packets of the TCP three-way handshake (SYN, SYN-ACK). These initial packets are for establishing a connection and contain no HTTP data, and therefore, no secret header. The firewall was dropping the connection before it could even be established.

Two further attempts followed: a multi-rule stateful approach, and the canonical UFW Docker patch. Both failed. The conclusion drawn is the actual lesson:

the core conflict between the TCP protocol's initial handshake and the firewall's stateless string matching proved to be the insurmountable hurdle

I reverted the filtering attempts and retained the application gateway. This does not make the backend unreachable to VPN clients; that remains a limitation of the lab.

A note on the interactive

LoadBalancerLab implements the real formula above. What it does not have are the measured node_load1 values. The project recorded the verification qualitatively ("script calculated lower weight for web-server-1") without capturing the numbers. The widget therefore runs on clearly-labelled illustrative figures rather than presenting invented numbers as captured ones.