observability
Metrics, probes and models of an estate — knowing what a system is doing.
-
The Whole Estate in One Article: How Every Layer Fits Together
Each layer of a self-hosted platform is well documented in isolation, but nothing describes the order they have to arrive in or why getting that order wrong fails weeks later rather than immediately. This article walks the full dependency chain from bare metal to self-hosted model serving, and demonstrates the ordering discipline with a runnable Compose file.
-
Turning Alertmanager Webhooks Into Retained MQTT State
Alertmanager's webhook delivers an event to whoever happens to be listening at the moment it fires, which is no help to a system that connects later and wants to know what is currently active. This builds a small bridge that keeps current alert state in Redis and republishes it to MQTT as retained messages, with a complete setup that runs on a laptop.
-
A Dead-Man's Switch for the Monitoring System Itself
A monitoring pipeline that is completely down looks identical to one where everything is fine, because both states produce zero alerts. This explains the dead-man's-switch pattern that closes that gap, and gives a complete, self-hosted example that a reader can run and deliberately trip on a laptop.
-
Probing the Names You Publish, Not the Services You Run
A service that checks only its own process health can report perfectly healthy while nobody outside it can reach it at all. This covers why external, black-box probing against the published name catches an entire class of failure that internal health checks structurally cannot, with a complete blackbox_exporter setup that runs on a laptop.
-
Prometheus HTTP Service Discovery Backed by a Live Inventory
A static Prometheus targets file is correct on the day someone last edited it and wrong every day after that infrastructure changes. This walks through Prometheus's HTTP service discovery mechanism, backed by a live inventory rather than a file, with a complete example that runs on a laptop.
-
Host Probes via the Textfile Collector Pattern
A single custom host measurement does not justify a dedicated exporter process, but it still needs to reach the same scrape pipeline as everything else. The textfile collector is node_exporter's answer, and it has two failure modes that only show up under load. This covers both, with a complete, runnable example.
-
A Rules File for Health Evaluation, Validated at Load Time
Hardcoding a threshold per metric works until you have more than a handful of them, and a hand-edited rules file that is only checked when a rule fires will eventually ship a typo straight into production. Here is a small Go health-evaluation engine that validates its entire rules file before it evaluates a single metric.
-
Merging Concurrent Polls Without Ever Serving a Half-Built Graph
A console that polls host inventory, storage, network and alerting concurrently must never show a reader a graph where some sources landed and others did not. This walks through why updating shared state field by field under a mutex fails silently, and how to publish a fully-built snapshot with a single atomic swap instead, with a runnable Go program and a race-tested invariant to prove it.
-
An Entity Graph as the Data Model for a Heterogeneous Estate
Hosts, volumes, links and alerts have genuinely different shapes, but an observability console still needs one model that can answer "what does this depend on" and "what is this node's health" without a type-specific code path for every kind of thing. This walks through an entity-graph model in Go that gets there, with a complete runnable implementation.
-
Separating 'Cannot Ever' From 'Currently Broken' in Sweep Automation
A sweep that checks a list of targets for a feature has to decide what a failure means, and a response code alone cannot tell it whether that failure was expected or is a regression. Here is how to make that decision explicit, keep it loud when it should be, and test it so the escape hatch does not become a second way to hide problems.
-
A DNS Canary CronJob as a Regression Test for Split-Horizon Resolution
A pod's search domain or CoreDNS configuration can resolve a public name to an internal blackhole, and once that bug is fixed it tends to come back the next time someone touches the Corefile for an unrelated reason. This walks through a CronJob that resolves a small, deliberate set of names on the exact path real pods use, so the regression fails a Job instead of a customer's request.
-
Alerting on Symptoms Instead of Metrics: Designing Alerts People Do Not Ignore
Most monitoring failures are not missing data, they are the wrong alert on good data, and the class of outage nobody catches is the one where every metric stays green. This works through the difference between alerting on a resource number and alerting on what a user experiences, and includes a runnable Prometheus and blackbox_exporter stack plus a tested DNS check that catches a failure no metric-based alert can see.