PhilippTheServer

reliability

Behaving correctly when something is already broken.

  1. Automating Web Shop Order Processing Before the Catalogue

    Web shop order processing automated first: Stripe webhook payment confirmation, stock reservations, outbox-driven DHL labels, two manual steps.

    automation architecture reliability python

  2. Traefik's 60-Second readTimeout Covers the Whole Request Body

    "Expected filesize of 104857600 bytes but read 29491200": Traefik's readTimeout defaults to 60s and bounds the whole request body, so slow uploads get cut.

    kubernetes networking reliability testing

  3. The NTP Clock Jump That Made the Scheduler Skip a Week of Measurements

    time.time() vs time.monotonic() in a Python scheduler: a backwards NTP step delayed runs by nine days. Monotonic durations and a jump-injecting test.

    embedded reliability python linux

  4. fsGroupChangePolicy: OnRootMismatch and the setgid Invariant That Makes It Sound

    Pod stuck in ContainerCreating from fsGroup recursive chown: fsGroupChangePolicy: OnRootMismatch, and why the setgid bit on the volume root makes it safe.

    kubernetes storage performance reliability

  5. kube-proxy in iptables Mode and Native nftables Rules in the Forward Chain

    kube-proxy iptables-nft sync silently aborts on native nftables rules a mesh agent adds to the FORWARD chain. Fix: nft delete rule by handle, ip and ip6.

    kubernetes networking linux reliability

  6. A Dead-Man's Switch for the Monitoring System Itself

    Alertmanager dead man's switch with expr: vector(1) == 1: an always-firing Heartbeat alert and an external receiver that alerts when pings stop.

    alerting observability reliability

  7. Merging Concurrent Polls Without Ever Serving a Half-Built Graph

    atomic.Pointer swap of a fully merged snapshot so concurrent polls never serve a torn read, a bug go test -race cannot see. errgroup with timeouts.

    go performance observability reliability

  8. Importing Pre-Existing Storage as a Static PersistentVolume

    persistentVolumeReclaimPolicy: Retain with volumeName and claimRef binds a PVC to existing data. What Released means and how to rebind safely.

    kubernetes storage ceph reliability

  9. Alerting on Symptoms Instead of Metrics: Designing Alerts People Do Not Ignore

    probe_success == 0 via blackbox_exporter instead of CPU alerts, plus a dnspython check that treats NOERROR with an empty answer as down.

    observability alerting reliability testing

  10. The Transactional Outbox: Never Losing a Job You Already Committed

    Transactional outbox in Postgres: commit-then-enqueue loses jobs on a crash. A SKIP LOCKED relay to Redis Streams and an idempotent consumer.

    python architecture reliability databases

  11. Running Ceph on Commodity Hardware: Failure Domains and the Four Things That Bite

    ceph osd df --format json: alert on the fullest OSD, not the mean, because one full OSD stops writes. Plus CRUSH failure domains and monitor quorum.

    ceph storage reliability performance

  12. Idempotent Payment Webhooks with a Deduplication Table

    Payment webhook processed twice on retry: dedupe on the provider event ID with a primary key, committed in the same transaction as the side effect.

    python reliability api-design

  13. MQTT Persistent Sessions and Retained Messages for Offline Subscribers

    MQTT clean_session=False only persists with a stable client ID, QoS 1 and Mosquitto persistence true. Retained messages give new subscribers the last value.

    redis reliability embedded

  14. UUID Case Normalization as a Recurring Cross-Service Bug

    Uppercase vs lowercase UUID keys cause silent cache and database lookup misses between services. Normalise once at the boundary with str(uuid.UUID(value)).

    reliability architecture python testing

  15. Sizing Redis maxmemory Against a Stream Trim Threshold

    OOM command not allowed when used memory exceeds maxmemory on XADD MAXLEN: why the stream never trims itself, and why a standalone XTRIM still works.

    redis performance databases reliability

  16. A Single-Writer Rule for Configuration Held in Redis

    SET NX PX lease plus a Lua script that checks the fencing token, bumps a version and publishes, so exactly one process writes a Redis config key.

    redis infrastructure-as-code architecture reliability

  17. Tuning Ceph Recovery: The Trade Between Degraded Time and Client Latency

    osd_max_backfills does nothing under mClock unless osd_mclock_override_recovery_settings is set. Choosing and measuring osd_mclock_profile for recovery.

    ceph storage performance reliability

  18. Capacity Planning Against the Fullest OSD, Not the Average

    Ceph full_ratio applies per OSD: one OSD at 95% stops writes while ceph df shows 70%. Alerting on the fullest OSD in ceph osd df, plus the upmap balancer.

    ceph storage performance reliability

  19. Infrastructure as Code with Ansible: Making a Host Reproducible from the Repository

    ansible-playbook --check --diff and a changed=0 second run as proof a host matches its repository, with a container demo that corrects a hand edit.

    infrastructure-as-code ansible linux reliability

  20. A Minimal Go HTTP Service with chi, Timeouts and Graceful Shutdown

    Go http.Server ReadHeaderTimeout and srv.Shutdown with signal.NotifyContext: draining in-flight requests on SIGTERM in a chi service, with a test.

    go reliability

All topics · All writing