Cluster Chronicles

The policy that did nothing

The verification test was supposed to fail. A temporary pod without the required label was started, and expected to time out reaching the backend.

It got an HTTP response instead.

Receiving an application-level HTTP response is definitive proof that the TCP connection was successful. A working NetworkPolicy should block the connection at the network layer, resulting in a Connection timed out error.

The kind of response carries information. An HTTP response such as a 404 proves that a TCP connection to the responding server completed. A timeout alone does not: it can occur while establishing the connection or while waiting for a response. In this test, the unexpected HTTP response showed that the intended network isolation was absent.

The object was valid and present in the API. The cluster was simply not enforcing it, because Minikube's default kindnet does not implement NetworkPolicy at all.

minikube delete --all --purge
minikube start --driver=docker --network-plugin=cni --cni=calico --cpus=4 --memory=10240

Re-verified: failed: Operation timed out.

Then it blocked too much

Switching CNI fixed the silence and immediately caused a different failure: app.local returning 504 Gateway Time-out. The first hypothesis was a deadlock in the Go handler.

That was disproved by simulating the ingress controller from inside its own namespace:

kubectl run -it --rm=true ingress-debug-pod --namespace=ingress-nginx --image=alpine:latest -- /bin/sh
# wget -O- --timeout=5 http://<BACKEND_POD_IP>:5000/api/info

It timed out. So:

The 504 error was not an application bug, but a network firewall issue.

The final policy has three ingress rules: from the frontend, from the ingress controller, and from Prometheus on the metrics port.

Autoscaling under synthetic load

The journal records running hey -z 5m -c 50 against the frontend service while watching the HPA. Replica counts progressed from 2 → 4 → 8 → 10, reaching the configured maximum as CPU utilisation exceeded the target. The exercise also adjusted scale-down behaviour to make the demonstration easier to observe.

This checks the HPA control path in a single-node Minikube environment. Increasing pod replicas on that node does not add physical capacity or demonstrate multi-node failover, and the test is not a production throughput benchmark.

The ghost in the machine

Custom alerting rules appeared correctly after a cluster reset, then reverted to an older version minutes later.

The first diagnosis was a Helm chart name collision. The chart creates a PrometheusRule with the same metadata.name as the manually applied one. That was tested and disproved. The clue that broke it open was the state of the reverted rules: they were not disappearing, they were being replaced by an older version.

The ghost was the project's own CI. connect.sh deployed Jenkins, JCasC configured it to run its pipeline on startup, and that pipeline checked out main and ran kubectl apply -f manifests/, overwriting the correct local state with stale committed files.

When multiple automation systems (a local deployment script and a remote CI/CD pipeline) can affect the same resources, it is essential to ensure they are working from the same source of truth. The issue was not a bug in Helm or Kubernetes, but a flaw in the overall development workflow.

The alert that could not fire

A memory alert divided observed usage by Kubernetes' reported allocatable memory. Kubernetes said 16GiB. The Docker cgroup capped at 10GiB.

10 / 16 = 62.5%

So a > 0.9 threshold was arithmetically unreachable. The rule now divides by 10737418240 explicitly. A second discovery followed: node memory stops at about 89% in the recorded environment. The journal attributes the gap to reserved resources; the observed percentage is not a universal Kubernetes limit.

And one that silently did nothing

A Grafana dashboard ConfigMap was created with an empty data section:

data:
  app-performance-dashboard.json: ""

Helm did not error when it could not read the file. It created an empty resource.

Declarative systems can sometimes fail silently.

And the twelve rules, read against the twelve in the previous stage

Both rule files declare exactly twelve alerts: this one in three groups, sherlock.rules.yml in four. Counting them was easy; reading them side by side is where the interesting part is.

The same condition, two different thresholds.

# VM stage — 80% of the guest's own CPUs
(1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100 > 80

# Kubernetes stage — 80% of the container, normalised against a 16-CPU host
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 20

The annotation on the Kubernetes rule says "CPU load has been over 80%", and the number in the expression is 20. The file's comment attempts to justify this as host/container normalisation, but the stated counts of 16 and 8 CPUs do not yield the claimed 25% relationship. That arithmetic is not a verified explanation of the threshold. The rule needs resource-normalisation checks against the actual exporter and cgroup setup before being reused on another machine.

A general rule narrowed to one pod.

sum by(pod) (container_memory_working_set_bytes{pod="container-mem-test-pod"})
/
sum by(pod) (kube_pod_container_resource_limits{pod="container-mem-test-pod", resource="memory"})

Both sides are pinned to container-mem-test-pod, the workload used to verify that memory limits are enforced. The VM stage has the same rule covering every container with a limit set, page cache excluded, and an explicit limit > 0 guard against dividing by zero. The general case was not generalised here; it was replaced by the test case. On this cluster, a container genuinely leaking memory would not have fired it.

Three rules carry no for clause or for: 0m. BackendPodFrequentlyRestarting has no for at all and fires on first evaluation; ElasticsearchStatusRed and its VM counterpart write for: 0m explicitly to say the same thing. One is a default nobody chose, the other is a decision. They behave identically. That is why the AlertGallery records both forms rather than normalising them.

And the fix above, in context. Replacing 16GiB with 10737418240 did make the memory alert reachable on Docker Desktop, where the cgroup happens to be 10GiB. It is not a general fix; it is a second hardcoded constant that happens to be right for the machine the cluster ran on. The page called it a fix because at the time, on that machine, it was one. An appropriate allocatable or cgroup-limit series would avoid hardcoding capacity, but the numerator and denominator must still describe the same resource boundary.