GitOps Galaxy

Git-backed cluster configuration

The repository declares twelve Application manifests, including the root, with destinations across seven namespaces:

argocd   database   monitoring   jenkins   dev   vault   vault-operator

I used an App-of-Apps to manage the child Applications, with automated sync, pruning and self-heal. The root manages Application manifests; the children manage their respective workloads. Normal deployment changes go through Git.

Self-heal, demonstrated

Git says two frontend replicas:

kubectl scale deployment infrastructure-insight-dev-frontend -n dev --replicas=3

Within seconds the ArgoCD application goes OutOfSync, and the UI shows a diff between live replicas: 3 and desired replicas: 2. Doing nothing:

kubectl get deployment infrastructure-insight-dev-frontend -n dev -w

The count returns to 2, one pod enters Terminating, and the application goes back to Healthy and Synced. This checks the controller's response to a deliberate manual change in the lab.

Two triggers, both ending in Git

Code-triggered. Jenkins polls the source repository, builds and pushes images with a 1.0.x tag, then rewrites the chart values and commits the change back:

yq -i '.backend.image.tag = "1.0.4"' infrastructure/app-chart/values.yaml
git commit -m "ci: Update image tags to 1.0.4"
git push origin main

ArgoCD sees the commit and syncs.

Image-triggered. ArgoCD Image Updater watches the same chart for new tags matching regexp:^1\.0\.[0-9]+$ and writes the change to Git itself. The filter was tightened once, deliberately, so that 1.1.0 and 2.0.0 could not be picked up. The journal shows the debug output confirming one surviving tag.

Deployment ordering and stateful resources

The monitoring configuration initially tried to create ServiceMonitors before the Prometheus Operator had installed their CRDs. I assigned sync wave 0 to monitoring-stack and 1 to monitoring-custom-config so the dependencies were installed first.

Large Prometheus CRDs also exceeded the client-side apply annotation limit. The monitoring Application uses ServerSideApply=true to avoid that annotation and excludes the PVC field /spec/volumeName from drift comparison. These are separate adjustments for apply size and stateful-resource differences.

Automated rollback in the lab

A breaking change was committed to trigger a build, which produced a CrashLoopBackOff. The pipeline's failure detector, three consecutive checks of restartCount > 2 with a 30-second grace, moved the build to its catch block:

yq -i '.backend.image.tag = "1.0.3"' infrastructure/app-chart/values.yaml
git commit -m "ci: ROLLBACK to 1.0.3 [skip ci]"
sh "argocd app sync ${appName} --force --prune --timeout 180"

It also pauses the image updater, because a rollback that immediately re-updates is not a rollback.

The failure was injected deliberately, and the pipeline executed the rollback. The development record reports the application returning to Healthy and Synced on the previous image. Repeated rollback commits record the iterations needed to get that workflow working.

Rollback coordination

Image Updater could otherwise select the failed image again after rollback. The history records attempts to pause or disable it, along with a temporary change to root self-heal while recovery was being tested.

The final manifests enable root self-heal and exclude the Image Updater deployment's /spec/replicas field from drift comparison. That lets the pipeline pause the updater without ArgoCD immediately restoring its replica count. The constraint is coordinating these control loops; reverting an image tag in Git is compatible with reconciliation.

Recovering a deleted Application

kubectl delete application.argoproj.io infrastructure-insight-dev -n argocd

With the root app's self-heal temporarily disabled so the test could observe the window. The next pipeline run produces:

application.argoproj.io/infrastructure-insight-dev created

The created output verifies that the pipeline recreated the missing Application resource. The root app's auto-sync was then re-enabled. This exercise checks resource recreation, not restoration of database contents or recovery of an entire lost cluster.

Database secrets from Vault

The Banzai Cloud Vault Operator was the first choice. It was dropped after research found the Helm repository deprecated following a Cisco acquisition, and HashiCorp's own Vault Secrets Operator was used instead.

I supplied the database password from a git-ignored bootstrap script, wrote it to Vault via vault kv put, and synced it to a Kubernetes Secret. The policy restricts read access to secret/data/database/* for one service account in one namespace.

Vault developer mode

The archived argocd/app-vault.yaml explicitly sets server.dev.enabled: true and server.ha.enabled: false. This is a single-pod, in-memory Vault setup. It makes the Kubernetes authentication and secret-delivery path convenient to exercise, but does not establish durable secret storage or high availability.

Scoped access, tested

RBAC was tightened from cluster-admin to four Roles across three namespaces, and one rule is restricted by name:

- apiGroups: [""]
  resourceNames: ["argocd-initial-admin-secret"]
  resources: ["secrets"]
  verbs: ["get"]

Separately, scripts/rbac-permission-test.sh uses the image-updater-user ArgoCD token to check both sides of its policy. The recorded output shows argocd app get succeeding and argocd app sync being denied. This verifies those actions for that identity, rather than asserting that every permission in the cluster was exhaustively tested.

TLS recovery after a restart

The restart investigation recorded missing or corrupted TLS material in argocd-secret. The server started with tls: false, so clients expecting HTTPS could not complete their handshakes.

The archived scripts/fix-argocd.sh scales down the ArgoCD server, deletes the problem secret and runs a Helm upgrade with --reuse-values to regenerate it. It then scales the server back up and waits for readiness. That restores this lab after the failure; the record does not establish a durable prevention of the restart problem.