alerting
Deciding what is worth waking someone for, and proving the alerting itself is alive.
-
Turning Alertmanager Webhooks Into Retained MQTT State
Alertmanager's webhook delivers an event to whoever happens to be listening at the moment it fires, which is no help to a system that connects later and wants to know what is currently active. This builds a small bridge that keeps current alert state in Redis and republishes it to MQTT as retained messages, with a complete setup that runs on a laptop.
-
A Dead-Man's Switch for the Monitoring System Itself
A monitoring pipeline that is completely down looks identical to one where everything is fine, because both states produce zero alerts. This explains the dead-man's-switch pattern that closes that gap, and gives a complete, self-hosted example that a reader can run and deliberately trip on a laptop.
-
Alerting on Symptoms Instead of Metrics: Designing Alerts People Do Not Ignore
Most monitoring failures are not missing data, they are the wrong alert on good data, and the class of outage nobody catches is the one where every metric stays green. This works through the difference between alerting on a resource number and alerting on what a user experiences, and includes a runnable Prometheus and blackbox_exporter stack plus a tested DNS check that catches a failure no metric-based alert can see.