kubernetes
Running workloads on a cluster: scheduling, storage, networking and the failure modes.
-
The Whole Estate in One Article: How Every Layer Fits Together
Each layer of a self-hosted platform is well documented in isolation, but nothing describes the order they have to arrive in or why getting that order wrong fails weeks later rather than immediately. This article walks the full dependency chain from bare metal to self-hosted model serving, and demonstrates the ordering discipline with a runnable Compose file.
-
External Secrets with Offline JWT Validation When Vault Cannot Reach the Cluster
Vault's default Kubernetes auth method needs to call back into the cluster's API server on every login, which cannot work once Vault and the cluster sit in deliberately isolated networks. Here is how to configure Vault's JWT auth method to validate service-account tokens offline, against a static public key, and wire External Secrets Operator to use it.
-
cert-manager for Mesh-Only Names That No ACME Challenge Can Reach
ACME's HTTP-01 and DNS-01 challenges both assume the certificate authority can reach or resolve something public, which is exactly what a mesh-only hostname does not have. This article shows why a cert-manager CA Issuer is the correct tool for that case, what it costs in trust distribution, and a complete kind-based example that issues and verifies a certificate for a private name.
-
How a VPN Client Silently Shrinks Your Pod Network MTU
A CNI that auto-detects MTU from the node's default-route interface bakes in that value once, at start-up, and never revisits it. When a VPN client later puts a lower-MTU hop in the path, the mismatch fails silently rather than with an error, and this article shows how to reproduce, diagnose and fix it.
-
ndots and the Accidental Search-Domain Leak in Pod DNS
Kubernetes gives every pod a DNS search list and ndots:5 by default, which is usually harmless. Add a custom internal domain to that search list and it stops being harmless: an ordinary external hostname can resolve through the internal domain's own records before it is ever tried as written. Here is why, and three ways to close it off.
-
A DNS Canary CronJob as a Regression Test for Split-Horizon Resolution
A pod's search domain or CoreDNS configuration can resolve a public name to an internal blackhole, and once that bug is fixed it tends to come back the next time someone touches the Corefile for an unrelated reason. This walks through a CronJob that resolves a small, deliberate set of names on the exact path real pods use, so the regression fails a Job instead of a customer's request.
-
fsGroup Recursive chown Hangs on Large Volumes
Setting fsGroup on a pod makes kubelet walk the entire volume at every mount, not just the first one, and on a volume with millions of files that walk can take hours. Here is why the second mount is the one that hurts, and how to stop paying for it on every restart.
-
Object Storage as the Durability Boundary for a Container Registry
A self-hosted registry backed by a single PersistentVolumeClaim keeps exactly one copy of every image layer, and routine maintenance is enough to delete it. This walks through why the fix is moving the durability guarantee to object storage rather than the volume, and gives a complete, runnable example that proves the registry container itself has become disposable.
-
Importing Pre-Existing Storage as a Static PersistentVolume
Dynamic provisioning assumes every PersistentVolume starts empty, which is the wrong assumption the moment you need to mount storage that already holds data. Here is how to bind a PVC to that data explicitly, and how to make sure deleting the PVC does not take the data with it.
-
Ceph CSI StorageClasses: RBD for Block, CephFS for Shared Volumes
A database needs an exclusive block device and a multi-pod workload needs a volume several nodes can write to at once, and a single Ceph cluster can serve both, but only if each workload uses the right CSI provisioner. This works through why RBD and CephFS answer different access patterns and gives a complete, reproducible Rook-Ceph setup on a local kind cluster proving both, including the failure that shows up when they're swapped.
-
Breaking the Circular Dependency in GitOps Secret Delivery
A secrets operator deployed through GitOps is the natural way to get credentials onto a cluster without committing them to git, but that operator needs its own credential to reach the secret store, and nothing has delivered that one yet. This works through where the circularity actually breaks, and gives a complete, reproducible bootstrap using a Vault dev server and the External Secrets Operator on a local kind cluster.
-
Sync Waves and the CRD-Before-Consumer Ordering Problem
Applying a CustomResourceDefinition and a custom resource that depends on it in the same Argo CD sync can fail intermittently, and removing the CRD from git can prune it in a way that deletes every instance of that resource cluster-wide, not just the ones this Application owns. This works through sync waves and a prune guard that make both failures impossible, with a reproducible kind cluster test proving it.
-
ApplicationSets: Onboarding a Workload With a Directory and a Pull Request
Hand-writing a new Argo CD Application object for every microservice is repetitive, and the step is easy to forget entirely when a new service is added. This shows how an ApplicationSet's git directory generator turns onboarding a workload into adding a directory and merging a pull request, with a complete, reproducible setup on a local kind cluster.
-
Argo CD App-of-Apps: Bootstrapping a Bare Cluster from One Root Application
A fresh Kubernetes cluster has nothing on it, and a runbook of kubectl apply commands run in the right order does not survive being handed to someone else or run against a rebuilt cluster months later. This works through the app-of-apps pattern, where one root Argo CD Application manages every other Application, and gives a complete, reproducible setup on a local kind cluster.
-
Docker Compose or Kubernetes: What the Control Loop Actually Buys You
Kubernetes is usually sold as the next step up from Docker Compose, which makes it sound like a bigger version of what you already have. It is a trade: you hand over control of where things run in exchange for a system that keeps working when a machine dies. This works through what that trade actually costs, with a runnable example showing the control loop doing its job and a Compose stack that cannot do the same thing.
-
Configuration Precedence: Docker Secrets, Kubernetes Secrets, Environment, .env, Default
A service that reads configuration only from environment variables forces every deployment target to shoehorn secrets into that one mechanism, and nothing in the code says which value wins when two sources disagree. Here is a small, tested resolver with a fixed precedence order, and the tests that prove it behaves the same in Compose, in Kubernetes and on a laptop.