That is the reason atlas exists: a self-hosted inference stack on a machine I own, an OpenAI-compatible API in front of it, benchmarks to tell me whether the models are actually any good, and a daemon that hands them real work. Every agent session I run — editor, terminal, CI — goes to that endpoint rather than out of the building.
Data locality. The strongest argument. Debugging context is exactly the material you would least like to send somewhere else. Local inference makes that a non-question rather than a policy you have to remember.
Predictable cost. Hosted inference is priced per token, which means your bill is a function of how useful you find it. That is a strange incentive — it makes people ration a tool that gets better the more you use it. A GPU is a fixed cost and marginal use is free.
Latency and availability. No round trip, no rate limits, no dependency on somebody else’s capacity planning.
It still works when the internet does not. Which is precisely when you are debugging.
The honest counterweight: the best hosted models are better than what you can run locally, and the gap on hard reasoning is real. Local models are excellent at the bulk of practical work — summarising, transforming, drafting, structured extraction, code you will review anyway — and noticeably weaker at the genuinely difficult problems. Pretending otherwise leads to disappointment. Use both, and know which is which.
The practical shape is a model server behind an API. Three things matter more than the choice of server:
Model swapping. You cannot hold every model in VRAM at once, and you do not want a separate service per model. A layer that loads on demand and evicts what is idle turns a fixed allocation into a pool. It costs a cold-start delay on the first request, which is almost always the right trade.
An OpenAI-compatible API. Not because that API is well designed, but because everything already speaks it. Adopting it means every tool, library and editor plugin works against your own hardware with a changed base URL and nothing else. Compatibility with a widespread interface beats a better interface nobody implements.
Quantisation is the lever. The difference between a model that does not fit and one that fits comfortably is usually quantisation, and the quality cost at sensible levels is much smaller than people expect. This is what decides whether a given GPU is useful.
This is the part I would push hardest.
Model evaluation by vibes is the norm and it is worthless. You try a new model, ask it a few things, form an impression, and swap. Your impression is shaped by the last thing you happened to ask, by prompt phrasing, and by wanting the new thing to be better.
So atlas has benchmarks: a fixed set of tasks representative of what I actually use models for, run against every served model, repeatably. Not academic benchmarks — those measure something real but not necessarily the thing you need. Your own tasks, scored consistently.
What this buys you is the ability to answer did that upgrade help? with evidence. It also produces genuinely surprising results. Bigger is not reliably better for a specific task. A quantisation step you assumed was lossy may cost nothing measurable on your workload. The model everyone recommends may be worse at your actual job than a smaller one.
You cannot learn any of that from impressions.
The interesting and dangerous part. An agent that can read logs, query metrics and propose changes is enormously useful. An agent that can apply changes is a different risk category, and the difference is worth being deliberate about.
The rules I hold to:
Read-only by default, and generously so. Reading logs, metrics, configuration and state covers most of the value. Most of what makes infrastructure work slow is gathering context, not typing the fix. An agent that only reads is already worth having.
Dry run before apply, always. Every serious infrastructure tool can show you what it would do. Agents should be required to use it, and the output is the thing a human reads. This is not agent-specific — it is the same discipline that should apply to a person — but agents make it non-negotiable, because an agent’s confidence is uncorrelated with its correctness.
A human approves anything that changes state. Not a rubber stamp: the dry-run output, read, then approved. The agent’s job is to do the work and present the evidence. The decision stays with a person.
Never on a base branch, never without a trail. Agent work goes through the same process as human work — a branch, a pull request, a review, a merge. Partly for safety, mostly so that in six months you can find out why something is the way it is.
Evidence, not assertion. The failure mode that matters is not an agent doing something destructive — that is what approval gates are for. It is an agent reporting that something is fixed, deployed or passing without having verified it. A claim without the command output behind it is worth nothing, and it is worse than nothing if it is believed. I would rather be told “I could not verify this” than be told a plausible falsehood.
The workflow that has actually stuck is narrower than “an agent that helps”, and the narrowness is the point: work is handed over as a GitHub issue, and comes back as a pull request.
A daemon watches for issues carrying a particular label. When one appears it drives a locally served model to do the work, and opens a pull request. I review the pull request the way I would review anyone’s. If the issue was badly written, the result is bad, and that is informative rather than dangerous — nothing has been applied to anything.
Two properties make this work, and neither is about the model.
The daemon is deterministic. Which issue gets picked up, in what order, and what happens to it is ordinary code with ordinary tests. The model is called at one point inside a process whose shape does not depend on it. That is what makes the system debuggable: when something goes wrong, it is nearly always the issue or the harness, and both are things you can read.
The label is a contract. Applying it is an explicit act by a person, so nothing is ever picked up because an agent decided it was in scope. Removing it takes the work back. It is a very small mechanism and it is the whole boundary.
What this buys is not speed of typing. It is that a piece of work can be described once, handed over, and collected later as a diff — with the review step exactly where it would be for a human contributor, because it is the review step for a human contributor.
That is a much less exciting story than autonomous operations, and it is the one that survives contact with production. The value is real and it is in the boring half.
The direction I find genuinely promising is not agents that act more autonomously. It is agents with better context — able to read the repository, the monitoring, the runbooks and the incident history together, and tell you what changed and what it correlates with.
Most infrastructure problems are not hard to fix once understood. They are hard to understand because the relevant information is spread across six systems and one person’s memory. That is a retrieval and correlation problem, and it is one that models are genuinely good at right now, on hardware you can own, without anything leaving the building.
]]>Now imagine someone leaves. What is the procedure? If the honest answer involves a checklist and a search, the accounts you forget will outlive the employment. That is not a hypothetical risk, it is the normal outcome, and it is the actual argument for running an identity provider. Not convenience. Offboarding.
With a central identity provider, services stop having users. They have sessions, handed to them by something else that did the authenticating.
A person exists in one place. Disable them there and every service goes dark at once, with no list to work through. Group membership becomes the thing that grants access, so “who can reach production” is a question with an answer rather than an investigation.
Second-factor enforcement becomes a single policy instead of twelve separate features of varying quality — half of which do not have it at all.
And people stop reusing one password across a dozen internal tools, because there is one password.
OIDC is the layer worth learning. A user hits a service, the service redirects them to the identity provider, they authenticate there, and come back with a signed token asserting who they are. The service validates the signature and trusts the claim. It never sees the password. It does not store credentials, cannot leak them, and has no opinion about password policy — that all lives in one place, maintained by people who thought about it.
The important consequence: your services stop being credential stores. Most internal tools have unremarkable auth code written once and never revisited. Deleting all of it and replacing it with “validate this token” is a large reduction in the amount of security code you are responsible for.
The failure mode with an identity provider is over-modelling. It will happily support fine-grained roles, nested groups, attribute mapping and per-client scopes, and you can build something so expressive that nobody can answer “what can this person do”.
Start with groups that mirror how you actually talk about access. If the sentence people say is “the infrastructure team can get to the cluster”, make a group called that. Grant things to groups, never to individuals — an individual grant is invisible in every review and outlives the reason it was made.
Resist per-service roles until a service actually needs to distinguish two kinds of user. Most do not. Most need “may this person in, yes or no”, and modelling three tiers of permission for a dashboard that has one page is work you will maintain forever for no benefit.
Some things cannot do modern auth. Old admin panels, appliances, dashboards with a single shared password. You will have several and they are usually the ones you most want behind authentication.
The pattern that solves this is a forward-auth proxy: the reverse proxy in front of the service asks an authentication service whether this request carries a valid session, and only passes it through if the answer is yes. The application behind it is untouched — it does not know authentication happened, and does not need to.
This is unreasonably useful. It puts real, group-controlled, second-factor-enforced authentication in front of software that has no concept of any of it, without modifying the software. A lot of “we cannot secure that, it is legacy” turns out to be false.
The caveat: the proxy is now load-bearing. A route that bypasses it — a second ingress, a port exposed directly, a health check path excluded a bit too generously — is a route with no authentication at all. Bypass paths deserve the same scrutiny as firewall rules.
It becomes a dependency of everything. When identity is down, nothing is reachable — not because those services are broken, but because nobody can prove who they are. That is a real concentration of risk and you should be honest about it rather than discovering it during an incident.
Which means: it needs to be genuinely reliable, it needs monitoring that checks the login flow end to end rather than just whether the process is running, and you need a documented way in when it is down. Break-glass access that does not depend on the thing that is broken. Written down, stored somewhere reachable without a login, and tested — an emergency procedure nobody has ever run is a procedure that does not work.
Sessions and tokens have subtle behaviour. Token lifetime, refresh, and what actually happens when you disable an account mid-session are worth understanding properly. Disabling a user does not always terminate their existing sessions immediately, and if your mental model is “I clicked disable, they are out”, you may be wrong for the length of a token lifetime. Find out which, before you need to rely on it.
Upgrades are real work. It sits in front of everything, so testing an upgrade means testing every login path. This is the price of centralising, and it is worth paying, but it should be planned rather than done casually on a Friday.
Immediately. Not for single sign-on, which is a pleasant side effect, but for the property that a person is one object and access is one decision.
The moment that justified it was the first time someone left and offboarding was: open one page, disable one account, done. No checklist. No wondering about the monitoring dashboard. Everything went dark at once, because everything had been asking the same question all along.
Everything else — the second factor, the audit log, the deleted auth code — is upside.
]]>An overlay mesh inverts that. Every machine gets a stable address on a private overlay network, and any two of them establish a direct encrypted tunnel — peer to peer, not through a hub. A control plane handles identity, key distribution and NAT traversal, then gets out of the path. WireGuard does the actual encryption.
The result is that “which network is this machine on” stops being a question you care about. A laptop on hotel wifi, a server in a rack, a device behind a carrier NAT — all of them have an address, and the address does not change when the network does.
Once every machine has a stable identity independent of its location, a whole category of configuration disappears.
You stop writing firewall rules against office IP ranges that change when the ISP feels like it. You stop maintaining a list of which subnet is which site. Automation stops needing different connection paths depending on where it runs. Services that should only be reachable internally can bind to the overlay address and simply be invisible from the internet — not firewalled off, not present.
And crucially, NAT traversal is handled for you. Getting a direct tunnel between two hosts that are both behind NAT is genuinely difficult, and the machinery to do it — STUN, hole punching, and a relay for the cases where it fails — is the actual value the control plane provides. When it works, traffic is direct. When it cannot, it falls back to a relay, and you get connectivity instead of a support ticket.
Every mesh gives you an internal naming scheme, so you can reach peers by name rather than by overlay address. The mesh resolver answers those names with the peer’s overlay address, and only for machines on the mesh. That is the design and it is good.
Here is what nobody puts in the quick-start guide.
Those names are usually subdomains of a domain you own publicly. And if that public zone has a catch-all wildcard — which an enormous number of zones do — then public DNS will answer for your internal names too. A DNS wildcard matches at any depth, not just one label. So a name intended to exist only inside the mesh gets a public answer, pointing at whatever the wildcard points at, which is typically your main public host.
While the mesh resolver is working, nobody notices. The mesh answers first, you get the right address, everything is fine. The problem appears the moment the mesh resolver is not answering: the client restarted, the tunnel dropped, someone disabled the resolver, the daemon is mid-reconnect. Then the query falls through to public DNS, and public DNS answers — confidently, successfully, with the address of a completely different machine.
No error. Nothing fails. The name resolves. It just points somewhere else now.
Sit with what that means for automation. Configuration management that connects to hosts by internal name, with credentials, running privileged operations. If a name silently resolves to a different machine, the automation connects to that machine and does what it was told to do. It may hand over a password on the way.
The fix is small and you should do it before you need it: publish an explicit record for the internal namespace in your public zone that points somewhere guaranteed to be unroutable.
A more specific wildcard beats the general catch-all for that subtree, so a single record covering the internal namespace takes it out of the catch-all’s reach without touching the records everything else depends on. Point it at documentation-reserved address space — addresses set aside by RFC precisely so they can never route anywhere.
The effect is that a fall-through now times out instead of reaching a live machine. That is a much better failure: slow and obviously broken, rather than fast and quietly wrong. Fail closed, not sideways.
Then add the second layer, because DNS alone is not enough. Host key checking is what turns “this is a different machine” from an invisible event into a loud one. If your automation is configured to accept any host key without complaint — and a surprising amount is, because it makes bootstrapping new machines convenient — then it will connect to the wrong host without a word.
The configuration you want accepts a key it has never seen (so provisioning a fresh machine still works unattended) but refuses loudly when a key changes. Those are different situations and they deserve different responses. A tool that treats them the same is one DNS blip away from an incident.
The control plane is infrastructure. It handles enrolment, key exchange and policy. If it is down, existing tunnels generally keep working, but nothing new can join and policy changes do not propagate. Treat it accordingly — it is not a nice-to-have service, it is a dependency of your ability to reach everything else.
Access policy is not the same as network reachability. The mesh gives every peer an address, and by default that can mean far more connectivity than you intended. A laptop should probably not be able to reach every storage node. Policy is where you say so, and it deserves the same review any firewall rule would get.
Overlay addressing overlaps with things. Meshes commonly use carrier-grade NAT space, which is not routable on the public internet but is very much in use inside some ISP networks. Know which range you are on before you debug a connectivity problem that only affects one person’s home connection.
Understand relayed versus direct. When hole punching fails, traffic goes through a relay, and your throughput and latency change character. It will still work, which is why you might not notice for months — until someone copies a large file and asks why it is slow.
Yes, and not marginally. Being able to address every machine you own by a stable name, encrypted end to end, regardless of where it physically sits, removes an entire class of network configuration from your life.
Just publish the blackhole record first. The trap costs one line of DNS to close and it is not the sort of thing you want to discover by finding out which machine your automation has been talking to.
]]>High CPU is not a problem. It is sometimes a sign of a problem, and it is sometimes a machine doing its job well. Alerting on it taught everyone the single worst lesson a monitoring system can teach: that alerts are things you glance at and dismiss. Once people learn that, the system is dead, and the outage it eventually catches correctly will be dismissed along with everything else.
Alert on what the user experiences. Graph everything else.
If a page is slow, alert. If a queue is not draining, alert. If a certificate expires in three days, alert. If CPU is at 90%, put it on a dashboard and leave me alone — I will look at it when I am investigating the alert that actually fired.
This distinction is not about which metrics you collect. Collect everything; it is cheap and you will want it during an incident. It is about which ones are allowed to wake someone. The test I use: if this fires and I do nothing, will anybody notice? If the honest answer is no, it is a graph, not an alert.
The corollary is uncomfortable: an alert that has fired ten times and been ignored ten times is not a monitoring gap, it is a monitoring bug, and the fix is to delete it or change it. Leaving it in place is choosing to train your team to ignore alerts.
White-box monitoring reads a system’s own account of itself: metrics the service exports, queue depths, internal error counters. It tells you why.
Black-box monitoring is an outside observer doing what a user does: fetch the URL, resolve the name, complete the handshake. It tells you whether.
You need both, and the mistake is thinking the first can replace the second. A service can report itself perfectly healthy while being completely unreachable, because everything it knows how to measure is inside the boundary that broke. Every layer between the process and the user — DNS, the load balancer, certificates, firewall rules, the ingress — is invisible to it.
Black-box checks are also the ones that catch the failures where nothing is broken at all, which brings me to the class of outage I find most interesting.
Here is a shape worth internalising, because it generalises far beyond the specific mechanism.
DNS wildcards do not work the way most people assume. A wildcard record synthesises an answer for a name only if nothing exists beneath that name. Put any record — any type at all — under a name, and the name now exists, and the wildcard is forbidden from answering for it. The name then resolves with a perfectly successful response code and an empty answer. No error. No failure. Just no address.
The practical consequence: a leftover record from a process that was supposed to clean up after itself can take a hostname off the internet, silently, while every neighbouring name served by the same wildcard keeps working perfectly. Query a random name under that wildcard and it answers. Query the real one and you get success-with-nothing.
Now consider what your monitoring says. The service is up. Its metrics are green. Its host is healthy. Certificates are valid. Every dashboard is fine, because the failure is not in anything any of those things measure — it is in the resolution step that happens before anyone reaches the service at all.
The only thing that catches this is an outside observer asking a public resolver for the name you actually publish and checking that the answer contains an address. Not that the query succeeded — that it returned something.
I like this example because it is not exotic. It is a category: failures where every component reports success and the composition still does not work. Certificate chains that validate individually and not together. A load balancer routing happily to a backend pool that is empty. Auth that returns 200 with a redirect loop. If your monitoring only ever asks components how they feel, none of these are visible.
An alert is a message to a tired person. Write it for them.
It should say what is broken in user terms, since that is what determines urgency. It should say how you know, so the first move is not re-deriving the query. It should point at the thing to look at. And it should be honest about severity — if it is not worth waking someone, it should not be able to.
The failure mode here is alerts written as an expression and a name and nothing else. Six
months later, at 3am, ProbeFailureRateHigh on a service you did not deploy tells you
nothing about whether to care.
The other way monitoring dies is by becoming too expensive to run, and the cause is almost always labels. Every distinct combination of label values is a separate time series. Put a user ID, a request path, or anything else unbounded in a label and you have built a system whose cost grows with your traffic in a way you did not intend.
The rule: labels are for things you would group by. If you would never write a query grouping on it, it is not a label, it is a log field.
Good monitoring is mostly deletion. The instinct when something breaks is to add an alert for it, and after two years you have three hundred alerts, most of which have never fired usefully, and a team that has learned the notification channel is noise.
Every alert should have to justify its continued existence. When was it last right? What did someone do about it? If it has never been right, it is not protecting you — it is using up the attention you will need for the alert that matters.
I would rather have twelve alerts that everyone trusts than three hundred that everyone ignores. The twelve will catch fewer things. They will catch them at 3am, when someone actually reads them.
]]>Ceph is the other trade: ordinary machines, ordinary disks, replication in software, and storage becomes your problem in exchange for costing a fraction as much. Whether that is a good deal depends entirely on whether you are willing to learn it properly. Half-learned Ceph is more dangerous than the SAN you were trying to avoid.
Ceph stores objects across a cluster and computes where an object lives rather than looking it up. That sounds like a detail and it is the central idea: there is no metadata server to become a bottleneck or a single point of failure, because placement is a function, not a table.
The pieces you actually care about day to day:
Everything above that — block devices, a filesystem, an S3-compatible gateway — sits on top of the same object store.
If you take one thing away: Ceph will faithfully protect you against exactly the failure you described, and no other.
Three replicas sounds safe. Three replicas that happen to live on three disks in the same machine protect you against disk failure and not against that machine’s power supply. The default CRUSH rules usually do something sensible at the host level, but “usually” is doing a lot of work in that sentence, and the moment your hosts are not uniform it stops being true.
So the first real design decision is not how many replicas — it is replicas across what. Across disks, hosts, racks, rooms? Write it down, configure it explicitly, and then verify it by asking the cluster where a given object actually lives rather than assuming.
The related trap: a cluster that cannot satisfy its own rule will simply refuse to place data, and tell you it is degraded, and wait. That is correct behaviour and it looks exactly like a broken cluster if you were not expecting it.
1. A full OSD stops the cluster, not just the disk.
Ceph is protective about running out of space, because a full cluster cannot rebalance itself out of trouble. Hit the threshold and writes stop — cluster-wide, not on that disk. Everything sharing that pool becomes read-only, which in practice means everything.
Worse, utilisation is not even. Placement is pseudo-random, so some OSDs run hotter than average, and the fullest disk is what matters, not the mean. A cluster that looks 70% full can have a disk at 90%.
Plan capacity against the fullest OSD, alert well before the threshold, and never let a cluster get comfortable above roughly two-thirds. You need the headroom to survive losing a node — because when a node goes, its data re-replicates onto the remaining disks, and if there is no room for it, one failure becomes an outage.
2. Recovery competes with your workload.
When a disk dies, Ceph starts making new copies. That is the system working. It is also a large amount of I/O arriving at exactly the moment you have less hardware than usual.
Left at aggressive settings, recovery can make a degraded-but-working cluster feel like a down cluster. Left too gentle, you stay at reduced redundancy for a long time, and the second failure is the one that costs you data. There is no universally right answer, but there is a wrong one: not having decided before it happens.
3. MON quorum is a different failure than OSD failure.
Lose disks and you lose redundancy. Lose monitor quorum and you lose the cluster — everything blocks, regardless of how healthy the data is, because nobody can agree on the map.
Run an odd number, spread them across genuine failure domains, and understand that two monitors is worse than one: one gives you no redundancy, two gives you no redundancy and a way to lose quorum by rebooting either machine.
4. The network is a storage component.
Every write is replicated, so a write is several network transfers. Latency between nodes is latency in your storage. A cluster that behaves fine at low load can fall apart when recovery traffic and client traffic contend on the same link.
This is the one people skip because network gear is boring and expensive, and it is the one that produces the most baffling symptoms — storage that is slow in a way that correlates with nothing on the storage nodes.
Setting the warnings aside, because they are warnings about a system worth running:
You get storage that grows by adding a machine instead of by a purchase order. You get to lose a disk on a Tuesday and not care. You get to lose a node and still not care, if you built the failure domains right. You get one pool of storage that presents as block devices, as a filesystem, and as an object store, instead of three separate products.
And you get the thing that is hardest to price: you understand your storage. When something is slow, you can find out why, because there is no opaque appliance in the way. That is worth a great deal at three in the morning, and it is precisely what you gave up when you bought the box with the phone number.
Start with more nodes and fewer disks each, rather than the reverse. Node count is what gives you real failure domains, and a cluster of three fat machines has fewer independent things to lose than one of six thin ones.
Do not put the first production workload on it. Put the second. Run something real but survivable, lose a disk on purpose, watch recovery, fill it up in a test pool until writes stop so you have seen what that looks like. Ceph rewards operators who have already met its failure modes and punishes the ones meeting them for the first time under load.
Then read the health output every day until it is boring. HEALTH_WARN is not a state to
live in. It is the cluster telling you what will hurt next.
The framing that gets everyone into trouble is that Kubernetes is what you graduate to when Compose stops being enough. That makes it sound like a bigger version of what you already have. It is not. It is a different bargain: you hand over control of where things run and when they restart, and in exchange you get a system that keeps working when a machine dies. If you do not need the second thing, you have paid for the first for nothing.
The thing nobody warns you about is that debugging changes shape.
With Compose, a service is a process on a host you can name. Something is wrong, you SSH in, you read the logs, you see the process. The mental model is one hop deep. With Kubernetes, “where is it running” is a question with a query attached, and the answer changes. Between you and the process there is now a scheduler, a CNI plugin, a service abstraction, an ingress, and a set of health checks any one of which can be the reason nothing is responding.
That is not a criticism. Every one of those layers is doing something you asked for. But it means your debugging is no longer “read the log”, it is “work out which layer is lying”. The first few times, that takes hours, and you will feel it acutely because the equivalent Compose problem would have taken minutes.
You also give up a certain kind of quick fix. On a single host, restarting a container to clear a bad state is a legitimate move. In a cluster, whatever put the container in that state will do it again, on a different node, at a worse time. The cluster removes your ability to paper over things — which is good, and which does not feel good.
Machine failure stops being an incident. This is the whole thing. On a single host, a dead disk is a phone call. In a cluster with a real failure domain, it is a rescheduling event you read about later. Everything else Kubernetes gives you is a consequence of the control loop that makes this work.
Deployment becomes declarative all the way down. With GitOps — a controller watching a repository and reconciling the cluster toward it — the deployment story becomes identical to the infrastructure story: the repository is the truth, and the cluster converges on it. Nobody deploys. They merge. The cluster notices.
That is a genuinely different operational posture. There is no deploy script that can be run with the wrong arguments, because there is no deploy script. There is no “did the staging change get applied to production” because the answer is in git. Rolling back is a revert.
Capacity becomes fungible. Once workloads are not pinned to hosts, adding a node adds capacity to everything at once, instead of to whichever service you decided lives there.
If you have one machine, Kubernetes gives you nothing except the layers. A single-node cluster has the failure characteristics of a single node plus the debugging surface of a cluster. That is the worst of both, and it is where an enormous number of installations actually sit.
If your workloads are not replicable — a database with local state, something with a license tied to a MAC address, a service that cannot tolerate being moved — then the scheduler cannot do the thing you are paying it for. You can pin them, and people do, but at that point you have a very elaborate way of running a process on a specific host.
If nobody on the team wants to learn it, it will rot. This is the one people find rude and it is the most reliable predictor I know. Kubernetes has a real learning curve and a fast release cadence. A cluster nobody is curious about becomes a cluster nobody upgrades, and an unpatched cluster is a worse liability than the Compose setup it replaced.
So the division is not ideological, it is about what each workload needs.
Things that are stateless, replicable, and benefit from surviving a node failure go to the cluster. Things that are pinned by their nature, or that are simple enough that the cluster would only add layers, stay on Docker. The test I apply is: if this host died right now, do I want the system to handle it, or do I want to be told? Both are legitimate answers. Pretending only one is legitimate is how you end up with a database in a pod that reschedules itself away from its disk.
The cost of running both is real — two deployment paths, two sets of habits. It is smaller than the cost of forcing everything into either one.
Learn the failure modes before you need them. Take a node out on purpose, on a weekday, while you are calm. Watch what happens. The whole value proposition is behaviour under failure, and if you have never seen it, you do not actually know whether you have it.
Resource requests are not paperwork. They are how the scheduler makes decisions. Leave them unset and you have asked it to pack your machines by guesswork, and it will guess wrong under exactly the load that made you care.
The ingress layer will surprise you. More outages come from the path into the cluster than from anything scheduling-related. Certificates, DNS, the exact behaviour of the controller when two things claim the same hostname — that is where the sharp edges are.
Do not let the cluster become the place where architecture goes to be forgotten. It is easy to add one more deployment, then another, until nobody can say what runs there or why. The cluster does not organise your system for you. It just makes disorganisation survive node failures.
Kubernetes is a good answer to a question a lot of people have not actually asked. Ask the question first: what do I want to happen when a machine dies? If the honest answer is “someone will notice and fix it, and that is fine” — then it is fine, and you have just saved yourself a great deal of YAML.
]]>Take the eleven seconds and you have just created a machine nobody can rebuild.
That is the whole argument for infrastructure as code, and it has very little to do with Ansible or Terraform. It is a claim about where the truth about a system lives. Either the repository describes the machine, or the machine does — and if it is the machine, then the knowledge is one disk failure away from gone.
I hold everything to one line: if a host cannot be rebuilt from the repository, it does not count as running.
It sounds absolute because it has to be. A rule with exceptions is not a rule, it is a preference, and preferences lose to tiredness. The value of infrastructure as code is not linear in how much of your infrastructure is covered. It is closer to a step function. Ninety percent coverage gives you almost none of the benefit, because you still cannot answer the only question that matters — can I rebuild this? — with yes. You have to answer with “mostly”, and “mostly” means you will find out which ten percent was missing at the worst possible time.
The practical test is not whether you have a playbook. It is whether you would be willing to wipe the machine right now and run it.
The failure mode is not that people do not write automation. Almost everyone writes automation. The failure is drift: the code and the machine start out identical and then quietly diverge, one eleven-second fix at a time.
Drift is nasty because it is invisible until you need it not to be. The playbook still runs green. It just no longer describes reality, because reality has grown a hand-added sysctl, a firewall rule someone opened for a debugging session in March, a package installed to test something. None of it is written down. All of it is load-bearing by the time you find out.
The defence is to make the code the only path. Not the preferred path — the only one. In practice that means:
That last point is the one people underrate. Once a playbook is genuinely idempotent, a
changed=0 run is a proof — the machine matches the code, right now, verified by
execution rather than by hope. That signal is worth more than the automation itself. I
would rather have a slow, ugly, idempotent playbook than an elegant one that always
reports changes.
Anything touching real infrastructure gets a dry run first. --check, plan, --diff,
whatever the tool calls it. Not because I expect the change to be wrong, but because the
difference between what I think a change does and what it does is exactly where
outages live.
There is a stronger version of this that I have come to rely on: to prove a change is only what you think it is, run it with your change removed and confirm the tool reports no difference at all. If the tool reproduces the entire existing state byte for byte, then whatever it reports with your change back in is genuinely only your change. That turns “I think this is safe” into something closer to a measurement.
It has caught things I would have sworn were fine. A refactor that was supposed to be cosmetic, quietly changing the order of records. A default that looked inert and was not.
Secrets. Ever. Not encrypted-in-a-pinch, not “it is a private repo”, not base64 — which is not encryption and everyone knows it. Secrets live in a secret store and are referenced by name. The repository says which secret, never what it is.
This is worth being rigid about because the failure is unrecoverable in a specific way: once a secret is in git history it is in every clone, every fork, every backup, and every laptop that ever pulled. Rotating it is the only fix, and rotation is exactly the thing nobody wants to do at the moment they discover the problem.
Infrastructure as code is not an argument for describing everything. Some things are genuinely better as data, some as documentation, and some should not exist at all.
The trap is building an abstraction layer over a mess instead of removing the mess. A role with fourteen boolean flags to accommodate four hosts that drifted apart is not automation, it is drift with a YAML interface. The honest fix is usually to make the four hosts the same.
I have written that role. Twice. The second time I noticed sooner.
The pitch for infrastructure as code is normally disaster recovery, and that is real but it undersells it. I have rebuilt a machine from scratch maybe a handful of times. I have read the repository to find out how something works hundreds of times.
That is the actual return. A system whose configuration is written down is a system you can reason about without logging in, hand over without a three-hour walkthrough, and change without the specific fear that comes from not knowing what you are about to break. Recovery is the insurance policy. Comprehensibility is what you use every day.
And the eleven-second fix at three in the morning? Do it, if the alternative is staying down. Then write the four-minute version before you go to bed, while you still remember what you did. The rule is not that you never touch a machine by hand. It is that the machine never keeps a secret from the repository overnight.
]]>The part worth writing about is not the API. It is the wiki.
The wiki lives in its own repository as Markdown files, and is published from there to wiki.opentaberna.de. It is not a wiki in the sense of a database somebody edits through a browser. It is text under version control, and it moves through pull requests like everything else.
That one decision changes the character of the documentation completely.
It can be wrong in a way that shows. A page in a hosted wiki drifts silently — there is no diff, no blame, no review, and no moment where somebody has to look at the change. A page in a repository is reviewed by the same person reviewing the code that made it stale.
It can be required. If the docs are in the repository, “the docs are updated in the same change” is a rule a reviewer can actually enforce, because the change is right there in the diff. If the docs are somewhere else, that rule is a hope. I have watched the hope version fail on every project that tried it, including mine.
It can be run locally. docker compose up brings up the same wiki software the
published site runs, serving this repository’s pages, with no login and no setup wizard.
So a page looks locally exactly as it will look published. That sounds like a small
convenience and it is the difference between people previewing their writing and people
guessing.
The structure is the interesting bit, because it maps to the questions people actually arrive with:
Two of those deserve comment.
“The schema as it is actually built” is a deliberate phrase. Schema documentation drifts faster than anything else in a project, because it is written during design and then the migrations happen. A page that promises to describe reality has to be revisited when reality changes, and saying so in the title makes that obligation explicit.
“Every setting and where it can come from” matters more than it sounds. Most configuration documentation lists the settings and omits the precedence — file, environment, default, flag — and precedence is exactly what you need at the moment configuration is not doing what you expect. That is the only moment anyone reads the page.
An API that returns a bare object here and a wrapped one there, an error as a string in one place and an object in another, is an API where every client writes its own special cases. The special cases are where bugs live, and they are invisible until a client hits the one endpoint that is shaped differently.
Deciding the envelope and the error model once, writing it on a page, and then holding every endpoint to it is unglamorous and it is most of what makes an API pleasant. It also makes the documentation shorter, because the shape is described once instead of per endpoint.
The same argument applies to authorization. Roles and clients defined centrally, with the API enforcing them, means “who can do this” is answerable by reading one page rather than by grepping decorators.
Working on something public raises the bar in a specific way: you cannot rely on anybody knowing anything.
There is no colleague to ask, no shared context, no “obviously it needs the database running first”. Everything a person needs has to be written down, and the fastest way to find out whether it is written down is that somebody tries and fails.
That is uncomfortable and it is the most useful review a project gets. Every internal project I have worked on had gaps that were invisible precisely because everyone had already been told the missing thing in a conversation. A public project has no conversations to lean on.
The related discipline is repository conventions — how issues are written, what a commit message says, what “done” means. Internally you can get away with these living in somebody’s head. Publicly they have to be written down, and once they are written down you notice they were never really agreed.
The wiki-as-repository pattern, without hesitation.
Documentation that is not under version control is documentation that will be wrong, and the only question is when. Putting it in a repository does not make anybody write more — it makes not writing visible in a diff, which turns out to be the same thing.
]]>