PhilippTheServer

kubernetes

Running workloads on a cluster: scheduling, storage, networking and the failure modes.

  1. Traefik's 60-Second readTimeout Covers the Whole Request Body

    "Expected filesize of 104857600 bytes but read 29491200": Traefik's readTimeout defaults to 60s and bounds the whole request body, so slow uploads get cut.

    kubernetes networking reliability testing

  2. fsGroupChangePolicy: OnRootMismatch and the setgid Invariant That Makes It Sound

    Pod stuck in ContainerCreating from fsGroup recursive chown: fsGroupChangePolicy: OnRootMismatch, and why the setgid bit on the volume root makes it safe.

    kubernetes storage performance reliability

  3. kube-proxy in iptables Mode and Native nftables Rules in the Forward Chain

    kube-proxy iptables-nft sync silently aborts on native nftables rules a mesh agent adds to the FORWARD chain. Fix: nft delete rule by handle, ip and ip6.

    kubernetes networking linux reliability

  4. The Whole Estate in One Article: How Every Layer Fits Together

    Self-hosted platform build order: why mesh, cluster, Ceph storage, identity and observability must precede model serving, shown with Compose depends_on.

    infrastructure-as-code kubernetes ceph observability architecture

  5. External Secrets with Offline JWT Validation When Vault Cannot Reach the Cluster

    Vault kubernetes auth "failed to validate token" tokenreviews i/o timeout: jwt auth with jwt_validation_pubkeys from sa.pub for External Secrets.

    kubernetes vault secrets-management security

  6. cert-manager for Mesh-Only Names That No ACME Challenge Can Reach

    Waiting for HTTP-01 challenge propagation, no such host: ACME cannot issue for private names. A cert-manager CA ClusterIssuer and trust distribution.

    kubernetes tls

  7. How a VPN Client Silently Shrinks Your Pod Network MTU

    Frag needed and DF set (mtu = 1420): a VPN hop below the CNI's auto-detected 1500 MTU stalls large pod transfers. Reproducing it and pinning the MTU.

    kubernetes networking

  8. ndots and the Accidental Search-Domain Leak in Pod DNS

    dnsConfig.searches with ndots:5 lets an internal wildcard answer external names. Why ndots:1 only half fixes it and a trailing dot fully does.

    kubernetes dns networking

  9. A DNS Canary CronJob as a Regression Test for Split-Horizon Resolution

    CoreDNS Corefile rewrite regressions caught by a busybox nslookup CronJob that checks internal, external and NXDOMAIN names and fails the Job.

    kubernetes dns observability testing

  10. fsGroup Recursive chown Hangs on Large Volumes

    fsGroupChangePolicy: OnRootMismatch stops kubelet re-chowning every file on each mount. Why fsGroup pods restart slowly, and when fsGroupPolicy None applies.

    kubernetes storage ceph performance

  11. Object Storage as the Durability Boundary for a Container Registry

    MANIFEST_UNKNOWN after losing a registry PVC: REGISTRY_STORAGE s3 on MinIO keeps blobs through helm uninstall. What it does not fix: garbage-collect.

    kubernetes storage docker

  12. Importing Pre-Existing Storage as a Static PersistentVolume

    persistentVolumeReclaimPolicy: Retain with volumeName and claimRef binds a PVC to existing data. What Released means and how to rebind safely.

    kubernetes storage ceph reliability

  13. Ceph CSI StorageClasses: RBD for Block, CephFS for Shared Volumes

    Multi-Attach error for volume: Ceph RBD is ReadWriteOnce by design. When to use rook-ceph-block and when rook-cephfs for ReadWriteMany.

    kubernetes ceph storage

  14. Breaking the Circular Dependency in GitOps Secret Delivery

    External Secrets SecretStore tokenSecretRef needs a Vault token before GitOps can deliver one. Breaking the loop with one out-of-band secret.

    kubernetes gitops secrets-management

  15. Sync Waves and the CRD-Before-Consumer Ordering Problem

    no matches for kind "Widget" in version: CRD and CR in one Argo CD sync. sync-wave "-1" fixes ordering; Prune=false stops a CRD prune wiping CRs.

    kubernetes gitops testing

  16. ApplicationSets: Onboarding a Workload With a Directory and a Pull Request

    ApplicationSet git directory generator on services/*: one Argo CD Application per directory, requeueAfterSeconds, and where sync errors show up.

    kubernetes gitops automation

  17. Argo CD App-of-Apps: Bootstrapping a Bare Cluster from One Root Application

    Argo CD app-of-apps: a root Application with directory recurse over apps/ bootstraps a bare cluster from one kubectl apply, with pruning.

    kubernetes gitops automation

  18. Docker Compose or Kubernetes: What the Control Loop Actually Buys You

    Docker Compose restart: always vs a Kubernetes Deployment: what happens when the host dies, and what the control loop costs in debugging.

    kubernetes docker architecture gitops

  19. Configuration Precedence: Docker Secrets, Kubernetes Secrets, Environment, .env, Default

    Docker secrets in /run/secrets, Kubernetes secret mounts, env vars and .env: a Python resolver with fixed precedence that reports which source won.

    python infrastructure-as-code docker kubernetes

All topics · All writing