Skip to content

System walkthroughs

Explore how these systems fit together. Each animation reconstructs a documented mechanism; it is not a live deployment. Open a card to step through the flow, then explore the project for the decisions, results and limitations.

Step 2 of 7 · the DevOps journey 2025

Automation Alchemy

Six virtual machines provisioned entirely by Ansible, including a Jenkins server configured as code.

  • Repeat lab provisioning reached changed=0.
  • Application deployment and scheduled backups automated.
  • Ansible
  • Jenkins
  • JCasC
  • Vagrant
  • Go
  • +5
platform

Automation Alchemy: one command, six machines

Replay a step

vagrant up → runs the playbook itself

app app web-1 web1 web-2 web2 lb lb backup bkp ci ci

192.168.56.20–.25 · forwarded ssh :2222–:2227

  1. 01 common app · web-1 · web-2 · lb · backup · ci ✓ ✓ ✓ ✓ ✓ ✓
  2. 01 fail2ban app · web-1 · web-2 · lb · backup · ci ✓ ✓ ✓ ✓ ✓ ✓
  3. 01 node_exporter app · web-1 · web-2 · lb · backup · ci ✓ ✓ ✓ ✓ ✓ ✓
  4. 02 docker app · web-1 · web-2 · ci ✓ ✓ ✓ · · ✓
  5. 03 tinyproxy_server lb · · · ✓ · ·
  6. 04 prometheus_server lb · · · ✓ · ·
  7. 05 wireguard_server backup · · · · ✓ ·
  8. 06 grafana_server lb · · · ✓ · ·
  9. 07 nginx_lb lb · · · ✓ · ·
  10. 08 jenkins_server ci · · · · · ✓
  11. 09 backup_server_config backup · · · · ✓ ·
  12. 10 backup_client_config app · web-1 · web-2 · lb · ci ✓ ✓ ✓ ✓ · ✓
  1. browser lb :80 nginx
  2. /

    upstream frontend_app_servers no balancing directive is set

    1 2 3 4 5 6 web-1 .21:3000 web-2 .22:3000

    Six requests, strictly alternating — nginx's default round-robin. The app renders the host it got in Served by:, which is how this was checked.

  3. /api/ app :5000 one backend — an upstream with a single server, so there is nothing to balance /api/metrics /metrics rewrite ^/api/(.*)$ /$1
  4. anything else app :5000 ✗ dropped in DOCKER-USER — UFW's input chain does not govern Docker's forward chain
  1. 01 poll Gitea every 5 min — no webhook, the school blocks private-IP webhooks
  2. 02 build backend image juhonis/infra-insight-backend:${BUILD_ID} skipped on rollback
  3. 03 build frontend image juhonis/infra-insight-frontend:${BUILD_ID} skipped on rollback
  4. 04 scan hadolint, gosec — both non-fatal Trivy is the only fatal gate skipped on rollback
  5. 05 push to Docker Hub two tags: ${BUILD_ID} and latest skipped on rollback
  6. 06 ssh, then docker pull/run app, web-1, web-2 — raw commands over ssh
  7. 07 health gate 60s, then dump logs and fail
ci
app web-1 web-2

ssh, then docker pull / run on each

0 2 * * * on every one of these

  • app-server /etc
  • web-server-1 /etc
  • web-server-2 /etc
  • load-balancer /var/lib/grafana, /etc/nginx, /etc/prometheus
  • ci-server set in a vault-encrypted vars file — unverifiable

backupusr@backup-server /srv/backups/<hostname>/

Cron runs as devops and elevates to root with sudo for rsync; the remote side is backupusr. Config only — no volumes, database, uploads or logs.

--delete makes this a mirror rather than a backup: one copy per host, overwritten nightly. A file removed on a host is removed here the same night, and there is no earlier copy to fall back to. Nothing prunes it because nothing is ever kept.

prometheus :9090 on load-balancer — alongside nginx, Grafana and tinyproxy, so the machine that loses the site also loses the monitoring

  • app 192.168.56.20:9100
  • web-1 192.168.56.21:9100
  • web-2 192.168.56.22:9100
  • lb 192.168.56.23:9100
  • backup 192.168.56.24:9100
  • ci 192.168.56.25:9100

Every 15 seconds, 30-day retention, node_exporter on all six hosts. Grafana gets one dashboard — a community one, fetched at deploy time, not committed here.

Nothing alerts. No Alertmanager is installed anywhere, and rule_files and alerting are commented-out placeholders — a host could die at three in the morning and no one would be told.

The app's own /metrics endpoint — the one nginx rewrites to in scene 2 — is never scraped, and node_exporter carries no per-request data. Nothing here can show whether traffic is still reaching both frontends.

Six machines, ten plays, one command. Everything after that runs on its own: a pipeline every five minutes, a backup every night at two, a scrape every fifteen seconds. Four things are worth carrying out of here — the firewall that was never supposed to exist, round-robin that is two servers only because nothing said otherwise, a backup that deletes as faithfully as it copies, and monitoring that runs on the machine it is watching.
open Automation Alchemy →
Step 4 of 7 · the DevOps journey 2025

Sherlock Logs

A full observability platform. Prometheus, Grafana, Alertmanager and ELK, deployed from twenty Ansible roles.

  • Fixed a log-shipping loop that grew syslog to 18 GB.
  • Alert rules include API errors and projected disk exhaustion.
  • Ansible
  • Prometheus 2.53
  • Grafana
  • Alertmanager
  • Elasticsearch 8.18
  • +5
platform

Sherlock Logs: six machines, one of them watching

Replay a step

vagrant up → runs the playbook itself

app app web1 web1 web2 web2 lb lb ci ci mon mon

192.168.56.20–.26 · forwarded ssh :2222–:2225, :2227, :2228

  1. 01 common + fail2ban app · web1 · web2 · lb · ci · mon ✓ ✓ ✓ ✓ ✓ ✓
  2. 02 docker app · web1 · web2 · ci · mon ✓ ✓ ✓ · ✓ ✓
  3. 03 node_exporter app · web1 · web2 · lb · ci · mon ✓ ✓ ✓ ✓ ✓ ✓
  4. 04 cadvisor app · web1 · web2 · ci · mon ✓ ✓ ✓ · ✓ ✓
  5. 05 prometheus mon · · · · · ✓
  6. 06 grafana mon · · · · · ✓
  7. 07 alertmanager mon · · · · · ✓
  8. 08 elasticsearch mon · ci · · · · ✓ ✓
  9. 09 kibana mon · · · · · ✓
  10. 10 logstash mon · · · · · ✓
  11. 11 filebeat app · web1 · web2 · lb · ci · mon ✓ ✓ ✓ ✓ ✓ ✓
  12. 12 nginx + wireguard lb · · · ✓ · ·
  13. 13 jenkins ci · · · · ✓ ·
  14. — backup client commented out · · · · · ·
  15. — backup server commented out · · · · · ·

On 9 July 2025 those three moved here and backup-server was commented out — which is why :2226 is missing from the port list above, and why the last two rows are empty. In Automation Alchemy they ran on the load balancer, so losing a host took the monitoring with it.

  1. prometheus :9090 on monitor-server · scrapes every 15s · 30-day retention
  2. :9100
  3. node_exporter all six hostsnode_cpu_seconds_total — host and hardware
  4. :8088
  5. cAdvisor the five Docker hostscontainer_memory_working_set_bytes — containers, which node_exporter cannot see
  6. :9091
  7. go_backend_app app-serverbackend_http_requests_total — the application itself, the only tier that is about the app rather than the machine

The third tier is the one the previous figure admitted it did not have: an application exposing /metrics that nothing ever asked for.

  1. filebeat on all six VMs/var/lib/docker/containers/*/*.log — container input, stream: all/var/log/syslog — tagged system · /var/log/auth.log — SSH failures to structured events
  2. beats protocol, port 5044
  3. logstash :5044 system in tags goes to grok, everything else to jsonclient_ip becomes app_source_ip, path becomes app_request_path
  4. one index a day
  5. elasticsearch filebeat-%{+YYYY.MM.dd}
  6. search
  7. kibana five dashboards, imported as NDJSON

Nothing ever deletes them. There is no index lifecycle policy, so metrics expire at thirty days and logs never do.

  1. four of the twelve rules sherlock.rules.ymlInstanceDown for 1m · ElasticsearchClusterHealthRed for 0mHighApiErrorRate for 5m · DiskWillFillUpIn24Hours for 15m
  2. evaluated every 15s, and fires once the for clause has passed
  3. alertmanager route to email-notifications · group by [alertname, job]
  4. group_wait 30s · repeat_interval 4h
  5. email send_resolved: true
  6. and again when it clears
  7. resolved the recovery is as loud as the failure

Most alerting tells you something broke and then goes quiet, which is indistinguishable from the system healing on its own.

Three things this cannot do yet, and three it leaves rough. None of them stop the pipelines running, and each is a small, bounded piece of work.

  • Log retention An ILM policy: hot for seven days, warm, delete at ninety. The project README describes exactly this and it was never written.
  • Backups, or their absence Either re-enable the two plays or delete them. A working backup in the repo that can never run is worse than none, because it reads as covered.
  • Reproducible provisioning Generate the WireGuard key inside the role. Its own comment says "we will aim for generating if not exists", and it was never implemented — alongside the vault password, both currently sit outside version control.
  • A staggered cold start Every Filebeat agent ships its whole backlog the moment the VM boots, which is what pins the vCPUs. The page calls this mitigated rather than eliminated, and staggering start-up is the rest of it.
  • The test rule out of production TestDiskWillFillUpIn30Minutes ships with severity "testing" and pages the same address as everything else. Route it to a different receiver, or drop it.
  • Pinned dashboards Dashboards 1860 and 193 are fetched from revisions/latest, so a rebuild can hand you a different dashboard than the one you reviewed.
The same six machines as Automation Alchemy, and the machine that used to be the backup server is now the one watching all the others. Two pipelines in opposite directions, twelve rules with somewhere to go, and three things that are still not finished.
open Sherlock Logs →
Step 5 of 7 · the DevOps journey 2025

Cluster Chronicles

Six virtual machines replaced by one Kubernetes cluster, declared in manifests and Helm charts.

  • Network-policy enforcement checked after switching to Calico.
  • Frontend scaled from 2 to 10 replicas under local test load.
  • Kubernetes
  • Minikube
  • Calico
  • Helm
  • Prometheus Operator
  • +5
platform

Cluster Chronicles: six machines, and one path through them

Replay a step

./connect.sh → in order, because each step waits on the last

  1. minikube start --cni=calico --cpus=4 --memory=10240 --driver=docker
  2. minikube addons enable ingress
  3. kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=controller --timeout=120s
  4. kubectl apply namespaces/ rbac/
  5. kubectl apply alertmanager-secret.yml before Prometheus, to avoid a race
  6. helm upgrade --install prometheus installs the CRDs
  7. kubectl wait --for=condition=ready pod -l …name=prometheus-operator --timeout=120s
  8. kubectl apply application/ Ingress and ServiceMonitors
  • default frontend · backend · ingress · Jenkins in the repo
  • ingress-nginx ingress controller Minikube addon
  • monitoring Prometheus · Grafana · Alertmanager in the repo
  • logging Fluent Bit · Elasticsearch · Kibana in the repo

One node, Docker driver, Calico named on the command line because it enforces NetworkPolicy. The rest of the script adds Jenkins, Elasticsearch, Kibana and Fluent Bit.

  1. app.local:8080 browservia the minikube tunnel
  2. ingress-nginx-controller matches on path
  3. one host, two paths
  4. frontend-service:3000 Node.js frontend-deployment #1 frontend-deployment #2
  5. /
  6. backend-service:5000 Go backend-deployment #1
  7. /api/info
  8. 200 OK back out through both services

One host and one ingress object, two paths. The service has pods behind it — two of the frontend, one of the backend — and the next stages are about what keeps those two numbers honest.

backend-allow-ingress · namespace: default · policyTypes: [Ingress]

  1. frontend pod same namespace 5000 matches on app: frontend a podSelector alone — same namespace needs nothing else
  2. prometheus pod monitoring 9091 matches on app.kubernetes.io/name: prometheus across namespaces, so it needs both selectors
  3. any pod ingress-nginx 5000 matches on kubernetes.io/metadata.name: ingress-nginx the whole namespace — no pod label needed
backend app: backend Go · :5000 · 1 pod

[Ingress] only — the rule set governs what reaches the backend, and says nothing about what the backend reaches.

Three rules and no catch-all, so anything not listed is denied. The second one exists for the scraper in the last stage.

HPA · frontend-deployment

minReplicas: 2 · maxReplicas: 10 · target: 50% average CPU

  • scaleTargetRef frontend-deployment
  • minReplicas 2
  • maxReplicas 10
  • target averageUtilization: 50 · CPU
  • the floor the Deployment itself sets replicas: 2

average CPU

2 replicas10 replicas2 replicas

pods · frontend-deployment

backend-deployment pinned at 1 — not in the HPA

An autoscaler needs the metrics API, so step 2 of the deploy enables metrics-server — without it this controller has nothing to watch.

The controller watches one number and moves one other. Each half of this is a drawing of the same loop: the meter crossing the dashed line is the row gaining a pod.

Metrics

  1. Prometheus kube-prometheus-stack
  2. ServiceMonitor app-backend-sm
  3. backend :9091 the Go service’s own metrics port
  4. evaluate
  5. 12 rules · 3 groups kubelet cAdvisor on
  6. fires → Alertmanager
  7. Grafana 3 dashboard ConfigMapseach labelled grafana_dashboard=1

Logs

  1. Fluent Bit tail /var/log/containers/*.log
  2. 9200
  3. Elasticsearch 2 replicas · 512m heap
  4. index
  5. elasticsearch-exporter cluster health as metrics
  6. → Prometheus
  7. Kibana HTTPS, chart-issued CA

Metrics are pulled on 9091; logs are pushed from every node. One is the policy’s second rule, the other never passes through it.

The two columns are not independent: the exporter turns Elasticsearch's own health into metrics, which is what two of the twelve rules read. The Prometheus Operator claims objects by label rather than by namespace, so the Fluent Bit ServiceMonitor gets release=prometheus added to it — after the deploy waits for it to exist.

Six VMs become one node with four namespaces: a routed request, the policy governing it, an autoscaler on one deployment and not the other, and two pipelines collecting from everything.
open Cluster Chronicles →
2026

Full-Stack E-Commerce Platform

A modular-monolith e-commerce backend and React frontend, with an asynchronous payment pipeline.

  • Concurrent refresh test produced one winner from 10 requests.
  • Orders, stock deduction and cart clearing share one transaction.
  • Go
  • Gin
  • PostgreSQL
  • GORM
  • Redis
  • +6
application

Full-Stack E-Commerce Platform: a 202 before the work is done

Replay a step
  1. client checkout
  2. POST /payments validate order
  3. 202 Accepted returned now
  4. RabbitMQ queue
  5. worker consume
  6. Stripe charge
  7. order + email updated
Two separate AMQP channels, so publishing cannot block on the consumer, and manual acknowledgements, so a message unacked when the worker dies is redelivered rather than lost. The 202 does not mean the payment succeeded — it means the request is now somebody else's problem.
open Full-Stack E-Commerce Platform →
2024

match-me

A Go and React matchmaking app with preference scoring, profile setup and realtime chat.

  • Location filtering respects both users' search radii.
  • Shared games and languages gate preference scoring.
  • Go
  • Gin
  • PostgreSQL
  • PostGIS
  • JWT
  • +2
application

match-me: the same score, and two different decisions

Replay a step

No shared game

You Score They
  1. Rocket League
  2. Competitive
  3. Advanced
  4. Evening, Night
  5. PC, Steam Deck
  6. Discord, Voice Chat
  7. English, Finnish
  8. Shooter, Action, Platformer
  9. Duo
  10. Ranked
  1. Games — blocked
  2. Gaming style —
  3. Skill —
  4. Available —
  5. Platform —
  6. Comms —
  7. Language —
  8. Genres —
  9. Modes —
  10. Competitive —
  1. Valorant
  2. Competitive
  3. Advanced
  4. Evening, Night
  5. PC
  6. Discord
  7. English
  8. Shooter, Strategy
  9. Duo
  10. Ranked

never scored

Gates cleared, little in common

You Score They
  1. Rocket League
  2. Competitive
  3. Advanced
  4. Evening, Night
  5. PC, Steam Deck
  6. Discord, Voice Chat
  7. English, Finnish
  8. Shooter, Action, Platformer
  9. Duo
  10. Ranked
  1. Games +2.0 gate
  2. Gaming style —
  3. Skill —
  4. Available —
  5. Platform —
  6. Comms —
  7. Language +2.0 gate
  8. Genres —
  9. Modes —
  10. Competitive —
  1. Rocket League
  2. Casual
  3. Newbie
  4. Monday
  5. Mobile
  6. Text Chat
  7. English
  8. Puzzle
  9. Squad
  10. Casual

4.0 Both gates passed, and it still only earns the entry fee

Gates cleared, little in common dismissed

They liked you first

You Score They
  1. Rocket League
  2. Competitive
  3. Advanced
  4. Evening, Night
  5. PC, Steam Deck
  6. Discord, Voice Chat
  7. English, Finnish
  8. Shooter, Action, Platformer
  9. Duo
  10. Ranked
  1. Games +2.0 gate
  2. Gaming style +1.0
  3. Skill +1.0
  4. Available +2.0
  5. Platform +2.0
  6. Comms +2.0
  7. Language +2.0 gate
  8. Genres +1.0 +0.5
  9. Modes +1.0
  10. Competitive —
  1. Rocket League
  2. Competitive
  3. Advanced
  4. Evening
  5. PC
  6. Discord, Text Chat
  7. English, Swedish
  8. Shooter, Action
  9. Duo
  10. Competitive

14.5 One attribute disagrees — you want Ranked, they play Competitive

They liked you first accepted

Requested Accepted the second like is the accept, and a chat row is created with it

/chats/:id/ws receives messages, sends typing
/ws/connections pushes the full connection list on change
you Hello them
you typing… them
you hi them

Sent over POST /chats/:id/messages, not the socket. The socket carries it out, with chat_id, sender_id, content and created_at.

Everyone is asked the same ten questions, and both answers are compared. Two of those questions are pass or fail: you have to share at least one game and at least one language, or the candidate is dropped before anything is scored. The rest earn points, and a second match on the same question is worth half of the first — which is why the two shared genres are 1.0 and then 0.5.

That gives three outcomes. One candidate shares nothing and is never scored at all. One clears both gates and still only reaches 4.0, so it is dismissed. One reaches 14.5 and you accept — and that only becomes a match because they had already liked you, making yours the second like. Accepting also creates the chat automatically, and that is what the two sockets are for.
open match-me →
2024

Literary Lions Forum

A book-discussion forum in Go. Server-rendered on SQLite, with no JavaScript.

  • Server-rendered discussions without client-side JavaScript.
  • Database-backed sessions removed on logout.
  • Go
  • SQLite
  • bcrypt
  • Docker
application

Literary Lions Forum: one binary, one session row, and a 303

Replay a step

docker build -t forum-app .

  1. FROM golang:1.23.0
  2. WORKDIR /app
  3. RUN go mod download
  4. RUN go build -o main .
  5. EXPOSE 8080
  6. CMD ["./main"]
./main — one process, listening on :8080
  • 19 routes ServeMux
  • 8 templates HTML/
  • 12 stylesheets Css/
  • 0 JavaScript files anywhere

internal/ is 2,117 lines and main.go is 67. Seven of the eight templates are parsed once into package-level vars at startup, so a request renders from memory; the eighth is read from disk again on every visit.

  1. POST /like-post a form on /postsReferer: /posts
  2. :8080
  3. mux.HandleFunc one of 19 literal pathsno path patterns
  4. session_id cookie
  5. getUserIDFromSession every handler starts here
  6. bound parameters
  7. Likes one parameterised statement
  8. 303 See Other
  9. back to the Referer POST / redirect / GET · no fetch

The redirect target is the Referer header, so the handler sends you back to whichever page you submitted from — falling back to / when there isn't one.

if userID <= 0 → invalid user ID

  1. BEGIN
  2. INSERT INTO Posts user_id, title, content, book_id, created_at
  3. LastInsertId the id every category row needs
  4. INSERT INTO PostCategoryRelations post_id = 42
  5. INSERT INTO PostCategoryRelations post_id = 42
  6. INSERT INTO PostCategoryRelations post_id = 42
  7. COMMIT

any leg fails → ROLLBACK, nothing is written

303 See Other → /posts

One post is several things at once: the row, then a link row per category, each carrying the id the row before it returned. Written together or not at all.

SELECT like_type FROM Likes WHERE user_id = ? AND post_id = ?

  1. no row yet INSERT Likes(user 4, post 9, +1) first vote
  2. row, sign differs UPDATE Likes(user 4, post 9, -1) the sign flips
  3. row, same sign nothing happens Likes(user 4, post 9, +1) — unchanged not an undo

and the number on the page is not stored either — it is recomputed on every view:

  • SELECT COUNT(*) FROM Likes WHERE post_id = p.id AND like_type = 1
  • SELECT COUNT(*) FROM Likes WHERE post_id = p.id AND like_type = -1

One signed integer, one row per user per post. Pressing the same button twice changes nothing — it is a value you set, not a switch you flip.

  1. session_id cookie a UUID v4 — and the primary key
  2. Sessions row id · user_id · expires_at · 24h
  3. read at request time
  4. time.Now().After(expires_at) the whole check, in Go
  5. true → not logged in
  6. logout DELETE FROM Sessions WHERE id = ?
  7. revoked for real

in the UI

one boolean, {{if .LoggedIn}}, across all 8 templates

  • the nav
  • the like buttons
  • the comment box
  • the add-post form

in the profile tab

the same rows the counters read, joined back to their posts

SELECT … FROM Likes INNER JOIN Posts WHERE Likes.user_id = ? AND like_type = 1

A vote is written once and read twice: once as a number on the post, and once as a list on the account that cast it.

One binary, one database file and nineteen routes take a form submission, write it inside a transaction, set a vote as a value rather than a toggle, and know who you are from a row behind a cookie.
open Literary Lions Forum →

Selected work All projects →