Sherlock Logs

Two paradigms, one machine

The stage's premise is running both monitoring approaches side by side: a pull pipeline scraping exporters into Prometheus, and a push pipeline shipping logs to Logstash. They have genuinely different failure modes. That turned out to be the lesson rather than a detail.

Four separate memory episodes, escalating:

Each of those was an intermediate step, and the numbers moved again afterwards. What the repo actually ships is monitor-server at 6144 MB and 5 vCPU, with both the Elasticsearch and Logstash heaps pinned to 1g. The 512m and 256m values quoted in the project README are stale. They survive only in a comment in the Logstash role that was never updated when the value changed.

The fourth had the most misleading symptom. Kibana showed an enrollment screen that had nothing to do with Kubernetes. The cause was an apt upgrade inside a shared role silently upgrading the whole ELK stack. The VM crashed before the configuration that disabled X-Pack could be applied.

The fix is a health check rather than a sleep:

polling /_cluster/health until the cluster status reports as yellow, guaranteeing that Elasticsearch is fully initialized before the playbook proceeds to start Logstash

The 18GB file

Filebeat ships /var/log/syslog. Logstash writes to /var/log/syslog. Filebeat ships those lines. Logstash logs about them.

# /etc/filebeat/filebeat.yml
exclude_lines: ['logstash', 'filebeat']

The template marks this line as the critical fix. Before it, a single vagrant provision filled the root partition:

Logstash and Filebeat were writing their own operational logs to /var/log/syslog. Filebeat was then reading these same log entries and shipping them back to Logstash, which would process them and log about it, creating an infinite loop that rapidly filled the disk.

troubleshooting.md keeps the lesson in one line: if /var/log is the largest directory, a service is generating excessive logs. That was the cause here.

Picking a dashboard that works

Three community Grafana dashboards failed before a fourth was chosen. The strategy that worked was not guesswork:

download the candidate's JSON model from the Grafana API, inspect its PromQL, and pick one whose queries use the exact base metric names that our cAdvisor instance provides

The deployment also used container_label_restartcount where the expected restart counter was not available. That is a property of this exporter setup, not a general claim that all cAdvisor versions expose restart counts this way.

Application and security signals

HighApiErrorRate consumes the Go service's backend_http_requests_total counter, so it can identify an application-error condition separately from host health. The Logstash pipeline also parses SSH failures into categories such as invalid users, failed passwords and rejected keys. These are implemented rules and parsing paths; they are not evidence of monitoring a production incident.

Sherlock, and what its twelve rules actually say

sherlock.rules.yml declares twelve alerts across four groups: HostAlerts, ContainerAlerts, ObservabilityStackAlerts, and an AdvancedAlerts group that is where the interesting two live.

The disk alert combines a forecast with a recent-trend check.

- alert: DiskWillFillUpIn24Hours
  expr: |
    predict_linear(node_filesystem_free_bytes{mountpoint="/"}[4h], 24 * 3600) <= 0
    and
    deriv(node_filesystem_free_bytes{mountpoint="/"}[15m]) < 0

predict_linear fits a line to four hours of free-space samples and predicts the value 24 hours into the future. The first clause checks whether that forecast is zero or below; it does not test for a zero crossing in the past.

The second clause requires a negative free-space trend over the last 15 minutes. That shorter window can suppress an alert when recent cleanup has reversed the longer-term decline. Both clauses use fitted trends, so this is a conditional forecast, not a guarantee that the disk will fill.

The same file carries a scratch copy of it. TestDiskWillFillUpIn30Minutes runs over a 10-minute window with severity: "testing", retained beside the normal rules in this lab configuration. Twelve declared rules include this scratch rule, and the reason grep -c 'alert:' on this file reports thirteen: line 110 is a comment beginning "# Advanced trend-based alert:". The count that reached the project page was the right one.

One rule depends on the application, not the exporters.

- alert: HighApiErrorRate
  expr: |
    sum(rate(backend_http_requests_total{code=~"5.."}[5m]))
    / sum(rate(backend_http_requests_total[5m])) * 100 > 5

backend_http_requests_total is instrumented in the Go service rather than supplied by node_exporter. It is the only rule in either file that cannot work unless the application is exposing metrics. The file does not document that dependency. It is also what makes this rule the most valuable of the twelve: it is the only one that notices the backend returning 500s while every host is green.

And a typo that shipped. LowDiskSpaceWarning, the rule that fires below 50% free, has the summary:

summary: "Dick space if below 50% on {{ $labels.instance }}"

That notification text remains in the archived rule. Reviewing the expression is only part of reviewing an alert; its message also needs to be useful to the recipient.

Against Automation Alchemy

This project starts from the fleet Automation Alchemy already built and keeps that foundation almost unchanged. The same six machines on 192.168.56.20–25, the same vagrant up → Ansible path, the same nginx round-robin and DOCKER-USER rule, the same Jenkins pipeline. What differs is what the sixth machine became, and everything that follows from that.

The swap happened in a single commit on 9 July 2025. backup-server was commented out of both the Vagrantfile and the inventory. monitor-server appeared at .26 in its place. WireGuard moved to the load balancer, and Prometheus and Grafana moved with it.

Two of those rows close gaps this site already describes elsewhere. The /metrics endpoint that Automation Alchemy exposes and never scrapes is exactly what HighApiErrorRate consumes here. The same page that says nothing could observe the round-robin split is the page that fixed it.

What did not carry over. Backups are the real one. The two roles are still in the repository, complete with the cron job and the rsync script. Both plays are commented out of site.yml, backup_paths is empty, and backup_server_host still points at a machine the Vagrantfile no longer defines. Nothing can run it. Log retention is worse than absent. The daily filebeat-* indices have no ILM policy and no deletion phase, on the same disk that filled up once already.

An unconfigured fresh clone cannot provision this project without the WireGuard private key and Ansible vault password. Keeping secrets outside Git is appropriate, but they remain prerequisites for reproducing this archived environment.

This stage kept the fleet and replaced the backup host with better observability. The disabled backup plays and absent retention policy are important qualifications when comparing it with the preceding configuration.