Automation Alchemy

The renumbering

I changed the host addresses so the old and new environments could coexist.

Project 1 used 192.168.56.10 for its app server. SSH recorded that host key in known_hosts. When this project started at the same address, SSH refused to connect:

Offending ECDSA key in /home/juho/.ssh/known_hosts:3
Host key verification failed.

The available fix was ssh-keygen -R. It was rejected in favour of renumbering the whole fleet to .20–.25, and the reason given is the useful part. It was not correctness, but the ability to run both environments at once:

I want to be able to easily use both projects VMs, so I'm just changing the ips in this project

Note the subnet did not move; only the host octets did. .20 was chosen as a block rather than as an offset. The journal's own phrase is "both projects VMs", so this is coexistence, not an inference.

The fix that broke other things

The development record includes a WireGuard routing problem that blocked delegated Ansible tasks whenever the host VPN was up, because traffic for the entire subnet was being sent to the gateway through the gateway.

Split tunnelling was attempted. It is recorded in capitals:

THIS IS NOT FIX!!!! BROKE OTHER THINGS!!!!

Browser access to services broke during that attempt. The journal later records allowing SSH from the VPN subnet on the backup server; the portfolio's issue record also identifies a host-side routing workaround. These are adjustments to this environment rather than a general VPN-routing solution.

Private-address webhooks were another external constraint. The school policy blocked them, so the recorded workaround used the public address to trigger CI.

Application delivery and backups

I split configuration into service roles with shared variables, and used JCasC for Jenkins configuration and job definitions. The pipeline built the Go backend and Node frontend images, ran linting and security tools, pushed images to Docker Hub and deployed the application.

For backups, I chose a push model after comparing it with a central pull process. Client roles install rsync scripts and scheduled jobs that send selected paths to the dedicated backup server. The roles manage the supporting user, directories and SSH keys. This describes the configured backup workflow; a later observability stage does not retain those active backup plays.

Container firewall rules

Docker's published ports use forwarding rules, so UFW's INPUT rules were not the control point I had assumed. The recorded configuration moves these rules into DOCKER-USER, before Docker's own forwarding rules. Rule placement matters alongside the VPN access limitation below; it is not a claim that every access path was closed.

Changed=0

I set a concrete target for repeat provisioning:

ensure the automation was perfectly idempotent, meaning a second run should result in changed=0 in the final playbook recap

The retrospective records several changes, including:

• get_url with force: false "was found to be unreliable", replaced by a stat + block + when pattern across every role
• state: touch always reports changed; state: file reports truthfully

The outcome is described as a "sea of green" ok statuses with changed=0.

Still open

Because the WireGuard gateway NATs traffic, an admin's VPN client can still reach the backend port directly in this setup. A separate gateway VM was proposed to make that boundary easier to enforce, but was not added to this archived project.

What this became

Sherlock Logs starts from this exact fleet: same machines, same Ansible layout, same load balancer. It spends the backup-server on a dedicated monitor instead. The backups described above are not carried over; the backup roles survive there as commented-out plays with an empty backup_paths, pointing at a host its Vagrantfile no longer defines.