reliability
Behaving correctly when something is already broken.
-
A Dead-Man's Switch for the Monitoring System Itself
A monitoring pipeline that is completely down looks identical to one where everything is fine, because both states produce zero alerts. This explains the dead-man's-switch pattern that closes that gap, and gives a complete, self-hosted example that a reader can run and deliberately trip on a laptop.
-
Merging Concurrent Polls Without Ever Serving a Half-Built Graph
A console that polls host inventory, storage, network and alerting concurrently must never show a reader a graph where some sources landed and others did not. This walks through why updating shared state field by field under a mutex fails silently, and how to publish a fully-built snapshot with a single atomic swap instead, with a runnable Go program and a race-tested invariant to prove it.
-
Importing Pre-Existing Storage as a Static PersistentVolume
Dynamic provisioning assumes every PersistentVolume starts empty, which is the wrong assumption the moment you need to mount storage that already holds data. Here is how to bind a PVC to that data explicitly, and how to make sure deleting the PVC does not take the data with it.
-
Alerting on Symptoms Instead of Metrics: Designing Alerts People Do Not Ignore
Most monitoring failures are not missing data, they are the wrong alert on good data, and the class of outage nobody catches is the one where every metric stays green. This works through the difference between alerting on a resource number and alerting on what a user experiences, and includes a runnable Prometheus and blackbox_exporter stack plus a tested DNS check that catches a failure no metric-based alert can see.
-
The Transactional Outbox: Never Losing a Job You Already Committed
Committing a database change and then enqueueing a job as two separate steps leaves a gap in which the change is permanent but nothing ever runs the job and nothing notices. This works through the transactional outbox pattern and gives a complete, runnable Postgres, Redis Streams and Python implementation that survives a crash at any point in that gap.
-
Running Ceph on Commodity Hardware: Failure Domains and the Four Things That Bite
Ceph replaces one expensive box with a support contract for replication across ordinary machines you already own, and the trade is real in both directions: it costs a fraction as much and it comes with a set of failure modes that appear only when things are already going wrong. This covers the four that will bite, and includes a runnable check that alerts on the fullest disk in the cluster rather than the average, which is the one that actually decides whether writes keep working.
-
Idempotent Payment Webhooks with a Deduplication Table
Payment providers deliver webhooks at least once and retry on anything but a 2xx response, so a handler that is not explicitly idempotent will eventually process the same payment twice. This covers signature verification, a deduplication table with a unique constraint, and a transactional handler that makes redelivery a no-op, with a complete example and a test that proves it.
-
MQTT Persistent Sessions and Retained Messages for Offline Subscribers
A subscriber that drops off the network for a minute can miss every message published in that window, and a newly-connecting client sees nothing at all until the next publish. Persistent sessions and retained messages fix both, and this walks through the broker configuration and the client flags that make it work, plus a small Redis-backed cache for consumers that never speak MQTT at all.
-
UUID Case Normalization as a Recurring Cross-Service Bug
An uppercase UUID from one service and a lowercase one from another look identical to a person and different to a dict, a cache, or a database index, which turns one entity into two without raising an error. Here is a minimal reproduction of the bug and a boundary-normalisation fix with a test that keeps it fixed.
-
Sizing Redis maxmemory Against a Stream Trim Threshold
Redis Streams trim themselves as a side effect of XADD, and XADD is refused once maxmemory is reached under the default eviction policy. That leaves a gap where a stream can hit the memory ceiling before it ever gets trimmed down, and stay stuck there. Here is how to reproduce that state, why maxmemory-policy will not save you, and how to size and sequence trimming so it cannot happen.
-
A Single-Writer Rule for Configuration Held in Redis
When several services can write the same Redis key, a read after a write can return someone else's value and nobody owns the truth. This walks through a lease-and-version pattern that makes one process the only writer, with a complete Python example a reader can run on a laptop.
-
Tuning Ceph Recovery: The Trade Between Degraded Time and Client Latency
Recovery I/O competes with client I/O on the same disks exactly when hardware is already reduced, and the wrong balance either starves applications or leaves the cluster degraded for longer than the next failure can wait. This explains how the mClock scheduler's profiles replace the old manual throttles, and gives a reproducible cephadm setup for measuring the trade-off directly.
-
Capacity Planning Against the Fullest OSD, Not the Average
Ceph's full_ratio applies per OSD, so placement variance means the fullest disk in a cluster can be far ahead of the cluster-wide average, and one OSD crossing that threshold stops writes cluster-wide. This works through why the variance exists, how to see it before it becomes an incident, and gives a script that alerts on the tail of the distribution instead of the mean.
-
Infrastructure as Code with Ansible: Making a Host Reproducible from the Repository
Infrastructure as code only works if the repository is the single source of truth for a host's configuration, and that discipline is easy to state and easy to break under pressure. This walks through why partial coverage buys almost none of the benefit, why idempotence is the actual product rather than a nice property, and ends with a complete Ansible role, run against a throwaway container, that a reader can use to watch drift get corrected.
-
A Minimal Go HTTP Service with chi, Timeouts and Graceful Shutdown
http.ListenAndServe has no default timeouts and no way to drain in-flight requests on SIGTERM, so a rolling deploy or a container orchestrator's stop signal cuts connections mid-response. This builds a small chi-based service with explicit server timeouts and a shutdown sequence that waits for in-flight work to finish, with a test that proves a slow request survives a shutdown signal instead of being killed by it.