Ceph without a vendor's price list

The pitch for a SAN is that storage becomes somebody else’s problem. You buy a box, it has redundant everything, and there is a phone number. The pitch is honest. The price is also honest, and for a lot of organisations it is the single largest line in the infrastructure budget.

Ceph is the other trade: ordinary machines, ordinary disks, replication in software, and storage becomes your problem in exchange for costing a fraction as much. Whether that is a good deal depends entirely on whether you are willing to learn it properly. Half-learned Ceph is more dangerous than the SAN you were trying to avoid.

The model, briefly

Ceph stores objects across a cluster and computes where an object lives rather than looking it up. That sounds like a detail and it is the central idea: there is no metadata server to become a bottleneck or a single point of failure, because placement is a function, not a table.

The pieces you actually care about day to day:

• OSDs — one per disk, roughly. These hold the data. Losing one is normal.
• Monitors (MONs) — hold the cluster map and vote on it. These need quorum.
• The CRUSH map — the description of your failure domains: which disks are in which host, which host in which rack. This is where you tell Ceph what “independent” means.

Everything above that — block devices, a filesystem, an S3-compatible gateway — sits on top of the same object store.

Failure domains are the whole design

If you take one thing away: Ceph will faithfully protect you against exactly the failure you described, and no other.

Three replicas sounds safe. Three replicas that happen to live on three disks in the same machine protect you against disk failure and not against that machine’s power supply. The default CRUSH rules usually do something sensible at the host level, but “usually” is doing a lot of work in that sentence, and the moment your hosts are not uniform it stops being true.

So the first real design decision is not how many replicas — it is replicas across what. Across disks, hosts, racks, rooms? Write it down, configure it explicitly, and then verify it by asking the cluster where a given object actually lives rather than assuming.

The related trap: a cluster that cannot satisfy its own rule will simply refuse to place data, and tell you it is degraded, and wait. That is correct behaviour and it looks exactly like a broken cluster if you were not expecting it.

Four things that will bite you

1. A full OSD stops the cluster, not just the disk.

Ceph is protective about running out of space, because a full cluster cannot rebalance itself out of trouble. Hit the threshold and writes stop — cluster-wide, not on that disk. Everything sharing that pool becomes read-only, which in practice means everything.

Worse, utilisation is not even. Placement is pseudo-random, so some OSDs run hotter than average, and the fullest disk is what matters, not the mean. A cluster that looks 70% full can have a disk at 90%.

Plan capacity against the fullest OSD, alert well before the threshold, and never let a cluster get comfortable above roughly two-thirds. You need the headroom to survive losing a node — because when a node goes, its data re-replicates onto the remaining disks, and if there is no room for it, one failure becomes an outage.

2. Recovery competes with your workload.

When a disk dies, Ceph starts making new copies. That is the system working. It is also a large amount of I/O arriving at exactly the moment you have less hardware than usual.

Left at aggressive settings, recovery can make a degraded-but-working cluster feel like a down cluster. Left too gentle, you stay at reduced redundancy for a long time, and the second failure is the one that costs you data. There is no universally right answer, but there is a wrong one: not having decided before it happens.

3. MON quorum is a different failure than OSD failure.

Lose disks and you lose redundancy. Lose monitor quorum and you lose the cluster — everything blocks, regardless of how healthy the data is, because nobody can agree on the map.

Run an odd number, spread them across genuine failure domains, and understand that two monitors is worse than one: one gives you no redundancy, two gives you no redundancy and a way to lose quorum by rebooting either machine.

4. The network is a storage component.

Every write is replicated, so a write is several network transfers. Latency between nodes is latency in your storage. A cluster that behaves fine at low load can fall apart when recovery traffic and client traffic contend on the same link.

This is the one people skip because network gear is boring and expensive, and it is the one that produces the most baffling symptoms — storage that is slow in a way that correlates with nothing on the storage nodes.

What it actually gives you

Setting the warnings aside, because they are warnings about a system worth running:

You get storage that grows by adding a machine instead of by a purchase order. You get to lose a disk on a Tuesday and not care. You get to lose a node and still not care, if you built the failure domains right. You get one pool of storage that presents as block devices, as a filesystem, and as an object store, instead of three separate products.

And you get the thing that is hardest to price: you understand your storage. When something is slow, you can find out why, because there is no opaque appliance in the way. That is worth a great deal at three in the morning, and it is precisely what you gave up when you bought the box with the phone number.

If you are starting

Start with more nodes and fewer disks each, rather than the reverse. Node count is what gives you real failure domains, and a cluster of three fat machines has fewer independent things to lose than one of six thin ones.

Do not put the first production workload on it. Put the second. Run something real but survivable, lose a disk on purpose, watch recovery, fill it up in a test pool until writes stop so you have seen what that looks like. Ceph rewards operators who have already met its failure modes and punishes the ones meeting them for the first time under load.

Then read the health output every day until it is boring. HEALTH_WARN is not a state to live in. It is the cluster telling you what will hurt next.