ceph
Distributed storage on commodity hardware: placement, failure domains, recovery.
-
The Whole Estate in One Article: How Every Layer Fits Together
Self-hosted platform build order: why mesh, cluster, Ceph storage, identity and observability must precede model serving, shown with Compose depends_on.
-
fsGroup Recursive chown Hangs on Large Volumes
fsGroupChangePolicy: OnRootMismatch stops kubelet re-chowning every file on each mount. Why fsGroup pods restart slowly, and when fsGroupPolicy None applies.
-
Importing Pre-Existing Storage as a Static PersistentVolume
persistentVolumeReclaimPolicy: Retain with volumeName and claimRef binds a PVC to existing data. What Released means and how to rebind safely.
-
Ceph CSI StorageClasses: RBD for Block, CephFS for Shared Volumes
Multi-Attach error for volume: Ceph RBD is ReadWriteOnce by design. When to use rook-ceph-block and when rook-cephfs for ReadWriteMany.
-
Running Ceph on Commodity Hardware: Failure Domains and the Four Things That Bite
ceph osd df --format json: alert on the fullest OSD, not the mean, because one full OSD stops writes. Plus CRUSH failure domains and monitor quorum.
-
Tuning Ceph Recovery: The Trade Between Degraded Time and Client Latency
osd_max_backfills does nothing under mClock unless osd_mclock_override_recovery_settings is set. Choosing and measuring osd_mclock_profile for recovery.
-
Capacity Planning Against the Fullest OSD, Not the Average
Ceph full_ratio applies per OSD: one OSD at 95% stops writes while ceph df shows 70%. Alerting on the fullest OSD in ceph osd df, plus the upmap balancer.
-
CRUSH Hybrid Rules: Pinning the Read Primary to SSD Without Rebalancing
step take default class ssd, chooseleaf firstn 1, then class hdd firstn 0: a CRUSH rule with an SSD primary and HDD replicas, tested with crushtool.
-
SMR Drives in a Replicated Cluster: Finding the Disk That Ruins the Pool
lsblk ZONED=none does not rule out drive-managed SMR. A 30-minute fio randwrite probe and p99 latency ratio catch the cliff before it hits a Ceph OSD.
-
A Layered Storage Benchmark: Isolating Device, Network and Protocol Bottlenecks
fio, iperf3, rados bench, then fio on RBD: benchmarking device, network, Ceph protocol and full stack separately to find which layer caps throughput.