PhilippTheServer

observability

Metrics, probes and models of an estate — knowing what a system is doing.

  1. nginx: The Map Variable That Fixes the URI in proxy_pass

    nginx: [emerg] invalid URI prefix in proxy_pass: a regex location needs a bare upstream, and a map on $uri keeps a path-specific Cache-Control header.

    docker networking observability ci-cd

  2. A Daily Digest of What Actually Got Merged, Built from MQTT and a Local LLM

    paho-mqtt subscriber to SQLite plus a local LLM: a daily digest that summarises merged pull requests per repository from diff stats, no cloud API.

    ci-cd observability python architecture

  3. The Whole Estate in One Article: How Every Layer Fits Together

    Self-hosted platform build order: why mesh, cluster, Ceph storage, identity and observability must precede model serving, shown with Compose depends_on.

    infrastructure-as-code kubernetes ceph observability architecture

  4. Turning Alertmanager Webhooks Into Retained MQTT State

    Alertmanager webhook to MQTT retained messages: clear stale state on resolve with an empty retained payload, keeping current alerts in Redis by fingerprint.

    alerting redis observability api-design

  5. A Dead-Man's Switch for the Monitoring System Itself

    Alertmanager dead man's switch with expr: vector(1) == 1: an always-firing Heartbeat alert and an external receiver that alerts when pings stop.

    alerting observability reliability

  6. Probing the Names You Publish, Not the Services You Run

    blackbox_exporter __param_target relabel_configs: probe the published hostname to catch DNS, TLS expiry and proxy errors that /healthz cannot see.

    observability dns

  7. Prometheus HTTP Service Discovery Backed by a Live Inventory

    Prometheus http_sd_configs: the targets and labels JSON an inventory endpoint must return, refresh_interval, and why targets persist when it is down.

    observability automation networking

  8. Host Probes via the Textfile Collector Pattern

    node_exporter --collector.textfile.directory: torn .prom writes fixed by write-temp-then-rename, and stale files caught with a last-run timestamp alert.

    observability linux

  9. A Rules File for Health Evaluation, Validated at Load Time

    Go rules file with gopkg.in/yaml.v3: yaml.Unmarshal accepts operator: greter. Validate operators and metric names at load time so startup fails.

    observability go

  10. Merging Concurrent Polls Without Ever Serving a Half-Built Graph

    atomic.Pointer swap of a fully merged snapshot so concurrent polls never serve a torn read, a bug go test -race cannot see. errgroup with timeouts.

    go performance observability reliability

  11. An Entity Graph as the Data Model for a Heterogeneous Estate

    Go entity graph data model: one Entity struct with Kind and Attrs, typed contains, dependsOn and runsOn edges, health rollup and cycle detection.

    observability architecture go

  12. Separating 'Cannot Ever' From 'Currently Broken' in Sweep Automation

    Sweep automation: a 404 cannot tell "never supported" from "currently broken". An explicit cannot-ever allowlist with required reasons, and its tests.

    automation testing api-design observability

  13. A DNS Canary CronJob as a Regression Test for Split-Horizon Resolution

    CoreDNS Corefile rewrite regressions caught by a busybox nslookup CronJob that checks internal, external and NXDOMAIN names and fails the Job.

    kubernetes dns observability testing

  14. Alerting on Symptoms Instead of Metrics: Designing Alerts People Do Not Ignore

    probe_success == 0 via blackbox_exporter instead of CPU alerts, plus a dnspython check that treats NOERROR with an empty answer as down.

    observability alerting reliability testing

All topics · All writing