7 stages, one evolving system

Every entry below says what broke and what became of it. An unresolved one is listed as unresolved rather than dropped.

1. fixed

   Vagrant could not drive VirtualBox from WSL at all. More than seven hours were lost before it worked

   `Vagrant is unable to use the VirtualBox provider from the Windows Subsystem for Linux without access to the Windows environment.`

   then: The VMs were built on Windows proper and then driven from WSL over the host-only network. The whole `192.168.56.0/24` topology exists to support that path.

2. fixed

   Hardening locked Vagrant out of its own machines

   `AllowUsers devops` → `vagrant@192.168.56.10: Permission denied (publickey)`. Fixed: `AllowUsers devops vagrant`.

   then: The SSH configuration lists `vagrant` alongside `devops` in `AllowUsers`, preserving provisioning access in this manually configured stage.

3. fixed

   Disabling the NAT interface to force egress through a proxy broke Vagrant entirely. Three hypotheses were tested and disproven before the cause surfaced

   `50-cloud-init.yaml` contained `dhcp4: false`, which I had set during the earlier proxy work. The VM had lost its IPv4 address on `enp0s3`.

   then: A config the earlier proxy work had introduced, reverted so the interface keeps its DHCP lease. The three wrong hypotheses cost more than the fourth would have.

4. fixed

   Fail2Ban was not enabled at boot, so the sshd jail silently did nothing

   `Failed to access socket path: /var/run/fail2ban/fail2ban.sock. Is fail2ban running?` The jail was configured correctly and enforcing nothing.

   then: Started the service, then proved it rather than assuming it: six failed logins produced `Connection refused`, and `fail2ban-client status sshd` reported the ban.

Every entry below says what broke and what became of it. An unresolved one is listed as unresolved rather than dropped.

1. fixed

   An SSH host-key collision forced the entire address range to change

   `Offending ECDSA key in /home/juho/.ssh/known_hosts:3`. Host key verification failed.

   then: The six VMs moved from `.10–.15` to `.20–.25` rather than editing `known_hosts`, so the stale keys were sidestepped instead of trusted. That renumbering is why this stage and the one before use different ranges.

2. fixed

   An Ansible task used `group:` instead of `groups:`, silently destroying sudo membership

   `id devops` → `uid=1002(devops) gid=998(docker)`. The task overwrote the primary group rather than appending to it.

   then: Corrected to `groups: sudo` in the common role. The tell was `gid=998(docker)`: a supplementary-group mistake shows up in the primary GID because the wrong key replaced the whole group list.

3. workaround

   WireGuard routing loop, and the fix that was worse than the bug

   `THIS IS NOT FIX!!!! BROKE OTHER THINGS!!!!`

   then: Resolved on the host, not in the repo: the VPN client no longer routes `192.168.56.0/24` into the tunnel, because that range is the same one the tunnel endpoint lives on. The journal heading is accurate. An earlier attempt made it worse.

4. fixed

   Docker publishes ports through FORWARD, bypassing UFW INPUT entirely

   Docker's iptables rules for published ports take precedence in the FORWARD chain, bypassing UFW INPUT chain rules.

   then: UFW rules written for a published port were moving into the `DOCKER-USER` chain instead, so the firewall rules for container ports live there now. Docker consults that chain before its own.

Every entry below says what broke and what became of it. An unresolved one is listed as unresolved rather than dropped.

1. unresolved

   A firewall rule gated on an HTTP header killed every connection, including legitimate ones

   The DROP rule inspected the first packets of the TCP handshake, SYN and SYN-ACK, which carry no HTTP data and therefore no secret header. The connection was dropped before the header could ever be sent.

   then: Unresolved by design. Header-gating at packet level cannot work, and the rule was removed rather than replaced with something that looks like a fix.

2. unresolved

   The same problem survived two more attempts: a more elaborate stateful rule, and the canonical UFW Docker patch

   `This proved to be extremely brittle and still resulted in a blocked connection.`

   then: Both abandoned. Three attempts is the point at which reverting is the cheaper option. The next entry records that.

3. workaround

   Resolving it was to revert and document the limitation rather than pretend it was solved

   `iptables-persistent` was purged for a clean state. The backend port remains reachable on the private VPN, and the write-up says so.

   then: Deliberately unresolved. `iptables-persistent` was purged and the backend is reachable over the VPN only. The limitation is documented rather than papered over.

Every entry below says what broke and what became of it. An unresolved one is listed as unresolved rather than dropped.

1. fixed

   Filebeat ingested its own logs, and the loop filled the disk

   `/var/log/syslog` reached 18GB and provisioning died with `No space left on device`. Logstash and Filebeat wrote to syslog, Filebeat shipped it back, Logstash logged about it.

   then: The harvest paths are restricted to the specific logs intended for shipping, so the pipeline no longer reads its own output.

2. fixed

   Elasticsearch was killed by the JVM heap it asked for

   `There is insufficient memory for the Java Runtime Environment to continue.`

   then: Heap sizes and VM resources were adjusted. The archived monitor has 6 GiB and 5 vCPU, with Elasticsearch and Logstash heaps set to 1g. The later X-Pack startup issue is recorded separately.

3. workaround

   Then Kibana starved, and then everything starved at once

   `uptime` showed load average 45.55 with all vCPUs pinned, while every Filebeat agent shipped its whole backlog simultaneously and all three ELK components ran their own initialisation.

   then: Mitigated by provisioning order and heap sizing rather than eliminated. Every agent still ships its backlog at once on a cold start, and that is what pins the vCPUs.

4. fixed

   Three community Grafana dashboards failed against this cAdvisor version, one after another

   ID 14282 queried metric names this cAdvisor does not provide; ID 13946 did arithmetic between metrics whose labels do not match one-to-one.

   then: Resolved by reading the candidate dashboards' JSON from the Grafana API and matching their PromQL against the metric names this cAdvisor actually emits. ID 193 was compatible and is provisioned, and three hand-built dashboards now cover the VM, container and application panels directly.

Every entry below says what broke and what became of it. An unresolved one is listed as unresolved rather than dropped.

1. fixed

   A NetworkPolicy that silently did nothing at all

   An HTTP response came back from an unlabelled pod when a timeout was expected. Minikube's default kindnet does not support NetworkPolicy.

   then: Minikube is now started with `--cni=calico`, which enforces policies. kindnet accepts the manifest and ignores it, so nothing failed and nothing worked.

2. fixed

   After switching to Calico, the same policy worked too well and took down the application

   `app.local` returned `504 Gateway Time-out`. Simulating the ingress controller from inside `ingress-nginx` proved the policy was blocking it.

   then: Ingress controller namespace exempted from the policy, found by reproducing the request from inside the controller rather than from a laptop.

3. fixed

   Custom alerting rules reverted to old values minutes after being applied, and the first diagnosis was wrong

   The initial hypothesis, a Helm chart name collision, was disproved by experiment. The real cause: Jenkins, configured via JCasC to run on startup, checked out main and re-applied the old committed manifests.

   then: The startup pipeline was identified as the writer applying the older committed files. Local deployment and CI must use the same committed manifests; ArgoCD ownership is introduced in the subsequent GitOps stage.

4. workaround

   A memory alert that could never fire, because the arithmetic compared two different truths

   Kubernetes reported 16GiB allocatable; the Docker cgroup capped at 10GiB. The maximum expressible value was 10/16 = 62.5%, so a 90% threshold was unreachable.

   then: Divided by `10737418240` explicitly, which makes it fire on Docker Desktop. That is a second hardcoded constant rather than a fix. It is right for the machine this ran on and wrong on a real node.

Every entry below says what broke and what became of it. An unresolved one is listed as unresolved rather than dropped.

1. fixed

   Application resources raced each other for CRDs that did not exist yet

   `monitoring-custom-config` tried to create ServiceMonitors before `monitoring-stack` had created the CRDs. Fixed with sync waves: 0 for the stack, 1 for the config.

   then: Sync waves: `0` for the monitoring stack and `1` for its configuration, declared in `argocd/app-monitoring-custom.yaml`.

2. workaround

   Rollback experiments conflicted with automatic reconciliation

   The history contains `ci: Disable root selfHeal during rollback` and repeated ROLLBACK commits that pause or disable Image Updater.

   then: The experiments coordinated root self-heal with pausing Image Updater to prevent recovery actions from being undone. The archived root manifest enables self-heal; this is a workflow constraint in this setup, not an inherent incompatibility between GitOps and rollback.

Every entry below says what broke and what became of it. An unresolved one is listed as unresolved rather than dropped.

1. unresolved

   PostgreSQL consumed far more CPU than the workload could explain

   P95 3.40 vCPU at 100 users while the entire test generated only about 6.8 MB of backend egress.

   then: Never explained. The measurement stands as recorded, and sizing conservatively on it is the decision: a 48-vCPU instance priced from a number nobody can account for.

2. fixed

   Two of the four hypotheses were ruled out by reading the application’s own source

   `email` is declared `TEXT UNIQUE NOT NULL`, and `sql.Open("pgx", ...)` returns a persistent pool. These contradict the hypotheses of an absent email index or absent pool; they do not establish the CPU anomaly's cause.

   then: Both were eliminated on inspection rather than by experiment, so two of the four hypotheses were dropped without a reproduction.

3. unresolved

   The root cause remains unidentified

   What survives: default container database configuration, a measurement artefact in the exporter, and background maintenance overhead such as autovacuum and checkpointing.

   then: Still unidentified, and carried forward as the number one post-migration risk rather than closed with a guess.