storage
Volumes, object stores and the durability questions that decide where data lives.
-
fsGroup Recursive chown Hangs on Large Volumes
Setting fsGroup on a pod makes kubelet walk the entire volume at every mount, not just the first one, and on a volume with millions of files that walk can take hours. Here is why the second mount is the one that hurts, and how to stop paying for it on every restart.
-
Object Storage as the Durability Boundary for a Container Registry
A self-hosted registry backed by a single PersistentVolumeClaim keeps exactly one copy of every image layer, and routine maintenance is enough to delete it. This walks through why the fix is moving the durability guarantee to object storage rather than the volume, and gives a complete, runnable example that proves the registry container itself has become disposable.
-
Importing Pre-Existing Storage as a Static PersistentVolume
Dynamic provisioning assumes every PersistentVolume starts empty, which is the wrong assumption the moment you need to mount storage that already holds data. Here is how to bind a PVC to that data explicitly, and how to make sure deleting the PVC does not take the data with it.
-
Ceph CSI StorageClasses: RBD for Block, CephFS for Shared Volumes
A database needs an exclusive block device and a multi-pod workload needs a volume several nodes can write to at once, and a single Ceph cluster can serve both, but only if each workload uses the right CSI provisioner. This works through why RBD and CephFS answer different access patterns and gives a complete, reproducible Rook-Ceph setup on a local kind cluster proving both, including the failure that shows up when they're swapped.
-
Running Ceph on Commodity Hardware: Failure Domains and the Four Things That Bite
Ceph replaces one expensive box with a support contract for replication across ordinary machines you already own, and the trade is real in both directions: it costs a fraction as much and it comes with a set of failure modes that appear only when things are already going wrong. This covers the four that will bite, and includes a runnable check that alerts on the fullest disk in the cluster rather than the average, which is the one that actually decides whether writes keep working.
-
Tuning Ceph Recovery: The Trade Between Degraded Time and Client Latency
Recovery I/O competes with client I/O on the same disks exactly when hardware is already reduced, and the wrong balance either starves applications or leaves the cluster degraded for longer than the next failure can wait. This explains how the mClock scheduler's profiles replace the old manual throttles, and gives a reproducible cephadm setup for measuring the trade-off directly.
-
Capacity Planning Against the Fullest OSD, Not the Average
Ceph's full_ratio applies per OSD, so placement variance means the fullest disk in a cluster can be far ahead of the cluster-wide average, and one OSD crossing that threshold stops writes cluster-wide. This works through why the variance exists, how to see it before it becomes an incident, and gives a script that alerts on the tail of the distribution instead of the mean.
-
CRUSH Hybrid Rules: Pinning the Read Primary to SSD Without Rebalancing
Metadata-heavy pools suffer when their primary replica sits on a spinning disk, but re-pointing the whole pool at SSD means paying for all-flash capacity and triggering a full rebalance. This shows how to write a CRUSH rule that places only the primary on SSD and the remaining replicas on HDD, and how to test it with crushtool before touching a live map.
-
SMR Drives in a Replicated Cluster: Finding the Disk That Ruins the Pool
Drive-managed SMR disks report the same capacity, connector and SMART attributes as conventional drives, then fall off a latency cliff once their persistent cache is exhausted by sustained random writes. This walks through why that pattern matches exactly what a Ceph OSD does to its backing disk, and gives a reproducible benchmark that catches the problem before the disk is holding data.
-
A Layered Storage Benchmark: Isolating Device, Network and Protocol Bottlenecks
A single throughput number from an RBD volume or a CephFS mount hides whether the limit is a slow disk, a saturated replication network, or overhead in Ceph's own protocol path, and each of those has a completely different fix. This builds a benchmark methodology that measures the device, the network and the protocol separately before measuring the full stack, using fio, iperf3 and rados bench against a single-node Ceph container a reader can run without a real cluster.