Running Ceph on Commodity Hardware: Failure Domains and the Four Things That Bite

The problem

The pitch for a SAN is that storage becomes somebody else’s problem: buy a box, it has redundant everything, there is a phone number to call. The pitch is honest, and so is the price — for a lot of organisations it is the single largest line in the infrastructure budget.

Ceph is the other trade. Ordinary machines, ordinary disks, replication done in software, and storage becomes your problem in exchange for costing a fraction as much. Whether that is a good deal depends entirely on whether you learn it properly, because half-learned Ceph is more dangerous than the SAN it replaced: the SAN fails in ways a vendor has already documented, and Ceph fails in ways that only show up once you are already relying on it.

The trap that catches almost everyone is the same shape every time: a setting that looks safe in isolation — three replicas, a sensible-looking CRUSH rule, a cluster that reports HEALTH_OK — turns out to protect against a narrower failure than assumed, and the gap is invisible until the exact day it matters.

Working through it

The model, briefly

Ceph stores objects across a cluster and computes where an object lives rather than looking it up in a table. There is no metadata server to become a bottleneck or a single point of failure, because placement is a function, not a lookup. The pieces that matter day to day: OSDs (one per disk, roughly — these hold data, and losing one is normal), monitors (hold the cluster map and vote on it — these need quorum), and the CRUSH map, which is where you tell Ceph what “independent” means: which disks share a host, which hosts share a rack.

Failure domains are the whole design

Take one thing away: Ceph will faithfully protect you against exactly the failure you described to it, and no other. Three replicas sounds safe. Three replicas that happen to land on three disks in the same machine protect against a disk failing and not against that machine’s power supply failing. Default CRUSH rules usually do something sensible at the host level — but “usually” is doing a lot of work in that sentence, and the moment your hosts are not uniform it stops being true.

So the first real design decision is not “how many replicas” — it is replicas across what. Disks, hosts, racks, rooms? Write it down, configure it explicitly, and verify it by asking the cluster where an object actually lives rather than assuming the rule did what you meant.

Why a full OSD is a cluster event, not a disk event

Ceph refuses to run out of space quietly, because a full cluster cannot rebalance its way out of trouble. Hit the threshold and writes stop cluster-wide, not just on that disk — everything sharing the pool goes read-only, which in practice is everything.

The part that catches people is that utilisation is not even. Placement is pseudo-random, so some OSDs run hotter than the mean, and it is the fullest disk that decides whether writes still work, not the average. A cluster that looks 70% full in a dashboard can have one disk at 90%. Monitoring the mean is monitoring the wrong number.

Recovery competes with the workload it is protecting

When a disk dies, Ceph starts making new copies immediately — that is the system working. It is also a large amount of I/O landing at exactly the moment there is less hardware than usual to absorb it. Too aggressive and a degraded-but-working cluster feels down. Too gentle and you sit at reduced redundancy for a long time, and the second failure during that window is the one that costs data. There is no universally correct setting, but there is a wrong one: not having decided in advance.

Monitor quorum fails differently than disk loss

Lose OSDs and you lose redundancy on the data they held. Lose monitor quorum and you lose the cluster — everything blocks, however healthy the data actually is, because nobody can agree on the map. Run an odd number of monitors, spread across genuine failure domains, and remember that two monitors is worse than one: one gives you no redundancy, two gives you no redundancy plus a way to lose quorum by rebooting either machine.

The network is a storage component

Every write is replicated, so a write is several network transfers, and latency between nodes becomes latency in the storage layer. A cluster that behaves fine at low load can fall apart the moment recovery traffic and client traffic contend on the same link. This is the one people skip because network gear is boring, and it is the one that produces the most baffling symptoms: storage that is slow in a way that correlates with nothing on the storage nodes themselves.

The solution

A single-node demo cluster is enough to see the shape of the fullest-disk problem without needing real hardware. It starts a self-contained cluster in a container:

docker run -d --name ceph-demo \
  -e MON_IP=127.0.0.1 \
  -e CEPH_PUBLIC_NETWORK=0.0.0.0/0 \
  -e CEPH_DEMO_UID=demo \
  -e CEPH_DEMO_ACCESS_KEY=demo-access-key \
  -e CEPH_DEMO_SECRET_KEY=demo-secret-key \
  -e CEPH_DEMO_BUCKET=demo \
  quay.io/ceph/demo:latest-quincy
 
# give it a minute to finish bootstrapping, then:
docker exec ceph-demo ceph -s
docker exec ceph-demo ceph osd tree
docker exec ceph-demo ceph osd df --format json

ceph osd df --format json is the number that actually matters, and here is a check that reads it correctly — by fullest OSD, not by mean — as something a monitoring system can call directly:

#!/usr/bin/env python3
# check_osd_capacity.py
"""Alert on the fullest OSD, not the cluster mean.
 
Usage:
    ceph osd df --format json | ./check_osd_capacity.py
"""
import json
import sys
 
WARN_THRESHOLD = 70.0
CRIT_THRESHOLD = 85.0
 
 
def evaluate(osd_df: dict) -> tuple[str, str]:
    nodes = osd_df.get("nodes", [])
    if not nodes:
        return "unknown", "no OSDs reported"
 
    fullest = max(nodes, key=lambda n: n["utilization"])
    mean = sum(n["utilization"] for n in nodes) / len(nodes)
    pct = fullest["utilization"]
    message = (
        f"fullest OSD is osd.{fullest['id']} at {pct:.1f}% "
        f"(cluster mean is {mean:.1f}%)"
    )
 
    if pct >= CRIT_THRESHOLD:
        return "critical", message
    if pct >= WARN_THRESHOLD:
        return "warning", message
    return "ok", message
 
 
def main() -> int:
    osd_df = json.load(sys.stdin)
    status, message = evaluate(osd_df)
    print(f"{status.upper()}: {message}")
    return {"ok": 0, "warning": 1, "critical": 2, "unknown": 3}[status]
 
 
if __name__ == "__main__":
    raise SystemExit(main())

Run it against the live demo cluster:

docker exec ceph-demo ceph osd df --format json | python3 check_osd_capacity.py
# OK: fullest OSD is osd.0 at 12.4% (cluster mean is 12.4%)

A single-OSD demo cluster cannot show an uneven cluster, so here is the test that proves the logic does what the article claims — that a healthy-looking mean can hide a dangerous fullest disk:

# test_check_osd_capacity.py
from check_osd_capacity import evaluate
 
 
def test_mean_hides_the_real_problem():
    osd_df = {
        "nodes": [
            {"id": 0, "utilization": 40.0},
            {"id": 1, "utilization": 45.0},
            {"id": 2, "utilization": 92.0},  # the one that actually matters
        ]
    }
    status, message = evaluate(osd_df)
    assert status == "critical"
    assert "osd.2" in message
    # the mean (59%) would have looked comfortable; the fullest disk is not
    assert "92.0%" in message
 
 
def test_even_cluster_is_ok():
    osd_df = {"nodes": [{"id": 0, "utilization": 30.0}, {"id": 1, "utilization": 32.0}]}
    status, _ = evaluate(osd_df)
    assert status == "ok"

pytest test_check_osd_capacity.py
# 2 passed

That is the concrete version of “plan capacity against the fullest OSD, not the mean” — not advice to remember, but a check that fails a build or pages someone before the cluster does it for you.

Conclusion

Setting the warnings aside, because they are warnings about a system worth running: Ceph gets you storage that grows by adding a machine instead of a purchase order, and it gets you the thing that is hardest to price — when something is slow, you can find out why, because there is no opaque appliance in the way.

Three things generalise past Ceph specifically:

A redundancy guarantee is only as good as the failure domain it was tested against. “Three replicas” is not a safety property on its own; “three replicas, on three hosts, on two power circuits” is. Say the domain out loud and verify it, rather than trusting the default.

Monitor the number that determines the failure, not the number that looks representative. The mean utilisation is comforting and mostly irrelevant. The maximum is what decides whether writes still work.

Meet failure modes on purpose, before they meet you under load. Take a disk out, watch recovery, fill a test pool until writes stop, on a Tuesday afternoon with nobody depending on it. Ceph rewards operators who have already seen its failure modes and punishes the ones meeting them for the first time in production.