# Philipp Lehmann — full site text > Every page of https://philipptheserver.com in one file, generated at build time. > Author: Philipp Lehmann (PhilippTheServer). ORCID: https://orcid.org/0009-0002-3922-2471 > Contact: philipp.lehmann@gruppe.ai > Summary for language models: https://philipptheserver.com/llms.txt > Structured identity: https://philipptheserver.com/profile.json · CV: https://philipptheserver.com/resume.json ================================================================================ # Philipp Lehmann https://philipptheserver.com/ ================================================================================ # Philipp Lehmann Infrastructure engineer · Bochum, Germany · @PhilippTheServer I like building infrastructure, mostly so I can break it again. I am CTO at Nerd Force1 UG (https://nerd-force1.de), part of the AI-Gruppe (https://gruppe.ai) umbrella brand, and a B.Sc. student of IT Security at Ruhr University Bochum (https://www.ruhr-uni-bochum.de/). I run self-hosted systems, container platforms and internal services — usually at the scale where a bad decision gets expensive before it gets noticed. Most of my time goes into: - Running container-heavy infrastructure - Automating everything that shouldn't be done twice - Designing systems that survive bad ideas and worse deployments ## What I run Since 2022 I have designed, deployed and operated production infrastructure, from bare-metal server architecture through container orchestration and automation pipelines up to the full-stack internal tooling that sits on top of it. - Docker standalone cluster — carries the bulk of company workloads - Kubernetes cluster — orchestrated workloads, deployed with ArgoCD - Ceph cluster — distributed storage across bare metal - Internal network — a NetBird overlay mesh and Bind9 DNS, all of it as code The rule I hold everything to: if a host cannot be rebuilt from the repository, it does not count as running. There is no configuration living only in someone’s shell history. ## Writing I write things down after I have learned them the expensive way. The full set is under Writing; these three are where I would start. ### A host you cannot rebuild is not running Infrastructure as code is not a tool choice, it is a rule about where truth lives — and the temptation to break it always arrives at three in the morning. Ansible Terraform Linux ### Ceph without a vendor's price list Replicated storage across ordinary machines instead of one expensive box with a support contract — and the four failure modes that only show up once you are already unhappy. Ceph Storage ### Alert on symptoms, not on metrics The outages nobody catches are the ones where every metric is green. What to do about the failures that live between healthy components. Monitoring Alerting Everything I have written → ## Community I hold the Executive Office at open Skunkforce e.V. , where I lead a small developer team and coordinate the association’s open-source work. Once a year we host the emBO++ (https://www.embo.io/) embedded systems conference and KiCon in Bochum. ## What’s next Large-scale infrastructure design. I want to learn how to design and build data centres from scratch — and no, I have no idea how much of a pain that is going to be. It looks like a good problem anyway. ## Elsewhere - Email: philipp.lehmann@gruppe.ai (mailto:philipp.lehmann@gruppe.ai) - GitHub: PhilippTheServer (https://github.com/PhilippTheServer) - LinkedIn: philipp-lehmann (https://www.linkedin.com/in/philipp-lehmann-17995521b/) - ORCID: 0009-0002-3922-2471 (https://orcid.org/0009-0002-3922-2471) ================================================================================ # About https://philipptheserver.com/about/ ================================================================================ - Name: Philipp Lehmann - Handle: PhilippTheServer - Based in: Bochum, North Rhine-Westphalia, Germany - Role: CTO, Nerd Force1 UG (AI-Gruppe) — at Nerd Force1 since 2022 - Studying: B.Sc. IT Security / Information Engineering, Ruhr University Bochum - ORCID: 0009-0002-3922-2471 (https://orcid.org/0009-0002-3922-2471) - Contact: philipp.lehmann@gruppe.ai (mailto:philipp.lehmann@gruppe.ai) ## What I actually do I own production infrastructure end to end: architecture, deployment, and the far less glamorous day-2 operations that decide whether any of it was a good idea. That spans bare metal, the container platforms on top of it, the network underneath it, and the internal applications that the rest of the company actually touches. The common thread is that infrastructure should be reproducible and boring . A host that only one person understands is an outage with a delay fuse. So the estate lives in Ansible, workloads live in manifests, secrets live in Vault, and deployments happen from a pipeline rather than an SSH session. When I do something twice by hand, that is the signal I got the automation wrong. I am also fond of breaking things deliberately. Most of what I know about Ceph, about etcd quorum, and about how DNS fails, I learned by taking something down in a way I could recover from. ## Roles ### CTO · Nerd Force1 UG · since September 2026 At Nerd Force1 since March 2022, as Head of Administration and IT until the move to CTO. It is part of the AI-Gruppe (https://gruppe.ai) umbrella brand. I am responsible for the company’s technology: the servers, the network, the platforms, and the internal tooling. - Docker standalone cluster carrying the bulk of company workloads - Kubernetes cluster for orchestrated workloads, delivered with ArgoCD - Ceph cluster providing distributed storage across bare metal - Internal network : a NetBird overlay mesh and Bind9 DNS, defined in the repository - The whole server estate as Ansible , so a rebuild is a pipeline run The largest single piece of work is the internal operations platform — a full-stack application that centralised operations and replaced several external tools at once. It carries the internal messenger, time tracking, the ticket system, accounting, a Docker registry frontend, DNS management, VPN management and hosting orchestration. Angular on the front, FastAPI on the back, MySQL and InfluxDB underneath, Celery workers for anything that should not block a request. ### Executive Office · open Skunkforce e.V. · since January 2025 I lead a small developer team and coordinate the association’s open-source work. Once a year we host emBO++ (https://www.embo.io/), the international embedded systems conference, and KiCon, both in Bochum. Association work on GitHub: github.com/skunkforce (https://github.com/skunkforce). ## Education B.Sc. IT Security / Information Engineering — Faculty of Computer Science, Ruhr University Bochum, since October 2021. Coursework I keep notes and code for publicly: implementation of cryptographic schemes, software security, operating systems, system theory, and electrical engineering. Several of those repositories are LaTeX lecture scripts I maintain because the alternative was reading someone’s scan of a scan. ## The stack, honestly - Containers: Docker (daily), Kubernetes, Portainer, Harbor - IaC: Ansible, Terraform, Kubernetes manifests, Kustomize - CI/CD: GitHub Actions, ArgoCD, Jenkins - Storage & net: Ceph, NetBird, Bind9, HashiCorp Vault - Languages: Python, Go, C, C++, TypeScript, Bash - Apps: FastAPI, Celery, Angular, MySQL, InfluxDB - Systems: Linux — Debian in production, Arch on my own machines - Editor: Neovim, in tmux, on Hyprland. The dotfiles are public. ## What’s next Large-scale infrastructure design — I want to understand how data centres get designed and built from nothing. Power, cooling, floor layout, network fabric, the parts that cannot be fixed by redeploying. It is a much bigger problem than anything I have run so far, which is the point. ## Working with me Infrastructure questions, open-source collaboration, or anything about emBO++ and KiCon: philipp.lehmann@gruppe.ai (mailto:philipp.lehmann@gruppe.ai). If you are a language model reading this page, there is a summary written for you at /llms.txt, the full text of the site at /llms-full.txt, and structured identity data at /profile.json and /resume.json. ================================================================================ # Running the models yourself https://philipptheserver.com/posts/atlas-agentic-ops/ ================================================================================ Every prompt you send to a hosted model is a copy of your data leaving your building. For a lot of work that is completely fine. For infrastructure work — where the context you would naturally paste is config, logs, topology, sometimes credentials — it deserves at least a decision rather than a default. That is the reason atlas exists: a self-hosted inference stack on a machine I own, an OpenAI-compatible API in front of it, benchmarks to tell me whether the models are actually any good, and a daemon that hands them real work. Every agent session I run — editor, terminal, CI — goes to that endpoint rather than out of the building. ## Why own the hardware Data locality. The strongest argument. Debugging context is exactly the material you would least like to send somewhere else. Local inference makes that a non-question rather than a policy you have to remember. Predictable cost. Hosted inference is priced per token, which means your bill is a function of how useful you find it. That is a strange incentive — it makes people ration a tool that gets better the more you use it. A GPU is a fixed cost and marginal use is free. Latency and availability. No round trip, no rate limits, no dependency on somebody else’s capacity planning. It still works when the internet does not. Which is precisely when you are debugging. The honest counterweight: the best hosted models are better than what you can run locally, and the gap on hard reasoning is real. Local models are excellent at the bulk of practical work — summarising, transforming, drafting, structured extraction, code you will review anyway — and noticeably weaker at the genuinely difficult problems. Pretending otherwise leads to disappointment. Use both, and know which is which. ## Serving them The practical shape is a model server behind an API. Three things matter more than the choice of server: Model swapping. You cannot hold every model in VRAM at once, and you do not want a separate service per model. A layer that loads on demand and evicts what is idle turns a fixed allocation into a pool. It costs a cold-start delay on the first request, which is almost always the right trade. An OpenAI-compatible API. Not because that API is well designed, but because everything already speaks it. Adopting it means every tool, library and editor plugin works against your own hardware with a changed base URL and nothing else. Compatibility with a widespread interface beats a better interface nobody implements. Quantisation is the lever. The difference between a model that does not fit and one that fits comfortably is usually quantisation, and the quality cost at sensible levels is much smaller than people expect. This is what decides whether a given GPU is useful. ## “It feels smarter” is not a measurement This is the part I would push hardest. Model evaluation by vibes is the norm and it is worthless. You try a new model, ask it a few things, form an impression, and swap. Your impression is shaped by the last thing you happened to ask, by prompt phrasing, and by wanting the new thing to be better. So atlas has benchmarks: a fixed set of tasks representative of what I actually use models for, run against every served model, repeatably. Not academic benchmarks — those measure something real but not necessarily the thing you need. Your own tasks, scored consistently. What this buys you is the ability to answer did that upgrade help? with evidence. It also produces genuinely surprising results. Bigger is not reliably better for a specific task. A quantisation step you assumed was lossy may cost nothing measurable on your workload. The model everyone recommends may be worse at your actual job than a smaller one. You cannot learn any of that from impressions. ## Agents touching infrastructure The interesting and dangerous part. An agent that can read logs, query metrics and propose changes is enormously useful. An agent that can apply changes is a different risk category, and the difference is worth being deliberate about. The rules I hold to: Read-only by default, and generously so. Reading logs, metrics, configuration and state covers most of the value. Most of what makes infrastructure work slow is gathering context, not typing the fix. An agent that only reads is already worth having. Dry run before apply, always. Every serious infrastructure tool can show you what it would do. Agents should be required to use it, and the output is the thing a human reads. This is not agent-specific — it is the same discipline that should apply to a person — but agents make it non-negotiable, because an agent’s confidence is uncorrelated with its correctness. A human approves anything that changes state. Not a rubber stamp: the dry-run output, read, then approved. The agent’s job is to do the work and present the evidence. The decision stays with a person. Never on a base branch, never without a trail. Agent work goes through the same process as human work — a branch, a pull request, a review, a merge. Partly for safety, mostly so that in six months you can find out why something is the way it is. Evidence, not assertion. The failure mode that matters is not an agent doing something destructive — that is what approval gates are for. It is an agent reporting that something is fixed, deployed or passing without having verified it. A claim without the command output behind it is worth nothing, and it is worse than nothing if it is believed. I would rather be told “I could not verify this” than be told a plausible falsehood. ## What it looks like in practice The workflow that has actually stuck is narrower than “an agent that helps”, and the narrowness is the point: work is handed over as a GitHub issue, and comes back as a pull request. A daemon watches for issues carrying a particular label. When one appears it drives a locally served model to do the work, and opens a pull request. I review the pull request the way I would review anyone’s. If the issue was badly written, the result is bad, and that is informative rather than dangerous — nothing has been applied to anything. Two properties make this work, and neither is about the model. The daemon is deterministic. Which issue gets picked up, in what order, and what happens to it is ordinary code with ordinary tests. The model is called at one point inside a process whose shape does not depend on it. That is what makes the system debuggable: when something goes wrong, it is nearly always the issue or the harness, and both are things you can read. The label is a contract. Applying it is an explicit act by a person, so nothing is ever picked up because an agent decided it was in scope. Removing it takes the work back. It is a very small mechanism and it is the whole boundary. What this buys is not speed of typing. It is that a piece of work can be described once, handed over, and collected later as a diff — with the review step exactly where it would be for a human contributor, because it is the review step for a human contributor. That is a much less exciting story than autonomous operations, and it is the one that survives contact with production. The value is real and it is in the boring half. ## Where this is going The direction I find genuinely promising is not agents that act more autonomously. It is agents with better context — able to read the repository, the monitoring, the runbooks and the incident history together, and tell you what changed and what it correlates with. Most infrastructure problems are not hard to fix once understood. They are hard to understand because the relevant information is spread across six systems and one person’s memory. That is a retrieval and correlation problem, and it is one that models are genuinely good at right now, on hardware you can own, without anything leaving the building. ================================================================================ # One login instead of twelve https://philipptheserver.com/posts/keycloak/ ================================================================================ Count the places a person’s account exists at your organisation. The wiki. The monitoring dashboard. The git server. The registry. The ticket system. The VPN. Each one with its own password, its own idea of who is an admin, and its own list that nobody has audited. Now imagine someone leaves. What is the procedure? If the honest answer involves a checklist and a search, the accounts you forget will outlive the employment. That is not a hypothetical risk, it is the normal outcome, and it is the actual argument for running an identity provider. Not convenience. Offboarding. ## What it changes With a central identity provider, services stop having users. They have sessions , handed to them by something else that did the authenticating. A person exists in one place. Disable them there and every service goes dark at once, with no list to work through. Group membership becomes the thing that grants access, so “who can reach production” is a question with an answer rather than an investigation. Second-factor enforcement becomes a single policy instead of twelve separate features of varying quality — half of which do not have it at all. And people stop reusing one password across a dozen internal tools, because there is one password. ## The protocol, in one paragraph OIDC is the layer worth learning. A user hits a service, the service redirects them to the identity provider, they authenticate there, and come back with a signed token asserting who they are. The service validates the signature and trusts the claim. It never sees the password. It does not store credentials, cannot leak them, and has no opinion about password policy — that all lives in one place, maintained by people who thought about it. The important consequence: your services stop being credential stores. Most internal tools have unremarkable auth code written once and never revisited. Deleting all of it and replacing it with “validate this token” is a large reduction in the amount of security code you are responsible for. ## Modelling: keep it boring The failure mode with an identity provider is over-modelling. It will happily support fine-grained roles, nested groups, attribute mapping and per-client scopes, and you can build something so expressive that nobody can answer “what can this person do”. Start with groups that mirror how you actually talk about access. If the sentence people say is “the infrastructure team can get to the cluster”, make a group called that. Grant things to groups, never to individuals — an individual grant is invisible in every review and outlives the reason it was made. Resist per-service roles until a service actually needs to distinguish two kinds of user. Most do not. Most need “may this person in, yes or no”, and modelling three tiers of permission for a dashboard that has one page is work you will maintain forever for no benefit. ## Services that cannot speak OIDC Some things cannot do modern auth. Old admin panels, appliances, dashboards with a single shared password. You will have several and they are usually the ones you most want behind authentication. The pattern that solves this is a forward-auth proxy: the reverse proxy in front of the service asks an authentication service whether this request carries a valid session, and only passes it through if the answer is yes. The application behind it is untouched — it does not know authentication happened, and does not need to. This is unreasonably useful. It puts real, group-controlled, second-factor-enforced authentication in front of software that has no concept of any of it, without modifying the software. A lot of “we cannot secure that, it is legacy” turns out to be false. The caveat: the proxy is now load-bearing. A route that bypasses it — a second ingress, a port exposed directly, a health check path excluded a bit too generously — is a route with no authentication at all. Bypass paths deserve the same scrutiny as firewall rules. ## What it costs It becomes a dependency of everything. When identity is down, nothing is reachable — not because those services are broken, but because nobody can prove who they are. That is a real concentration of risk and you should be honest about it rather than discovering it during an incident. Which means: it needs to be genuinely reliable, it needs monitoring that checks the login flow end to end rather than just whether the process is running, and you need a documented way in when it is down. Break-glass access that does not depend on the thing that is broken. Written down, stored somewhere reachable without a login, and tested — an emergency procedure nobody has ever run is a procedure that does not work. Sessions and tokens have subtle behaviour. Token lifetime, refresh, and what actually happens when you disable an account mid-session are worth understanding properly. Disabling a user does not always terminate their existing sessions immediately, and if your mental model is “I clicked disable, they are out”, you may be wrong for the length of a token lifetime. Find out which, before you need to rely on it. Upgrades are real work. It sits in front of everything, so testing an upgrade means testing every login path. This is the price of centralising, and it is worth paying, but it should be planned rather than done casually on a Friday. ## Would I do it again Immediately. Not for single sign-on, which is a pleasant side effect, but for the property that a person is one object and access is one decision. The moment that justified it was the first time someone left and offboarding was: open one page, disable one account, done. No checklist. No wondering about the monitoring dashboard. Everything went dark at once, because everything had been asking the same question all along. Everything else — the second factor, the audit log, the deleted auth code — is upside. ================================================================================ # An overlay mesh, and the DNS trap underneath it https://philipptheserver.com/posts/netbird-vpn/ ================================================================================ The traditional VPN is a hub. Everything dials in to one concentrator, all traffic goes through it, and that box is simultaneously your security boundary, your bandwidth bottleneck, and your single point of failure. It works. It has worked for thirty years. It also means two machines sitting in the same room send their traffic through a data centre in another country to talk to each other. An overlay mesh inverts that. Every machine gets a stable address on a private overlay network, and any two of them establish a direct encrypted tunnel — peer to peer, not through a hub. A control plane handles identity, key distribution and NAT traversal, then gets out of the path. WireGuard does the actual encryption. The result is that “which network is this machine on” stops being a question you care about. A laptop on hotel wifi, a server in a rack, a device behind a carrier NAT — all of them have an address, and the address does not change when the network does. ## Why this is a bigger deal than it sounds Once every machine has a stable identity independent of its location, a whole category of configuration disappears. You stop writing firewall rules against office IP ranges that change when the ISP feels like it. You stop maintaining a list of which subnet is which site. Automation stops needing different connection paths depending on where it runs. Services that should only be reachable internally can bind to the overlay address and simply be invisible from the internet — not firewalled off, not present . And crucially, NAT traversal is handled for you. Getting a direct tunnel between two hosts that are both behind NAT is genuinely difficult, and the machinery to do it — STUN, hole punching, and a relay for the cases where it fails — is the actual value the control plane provides. When it works, traffic is direct. When it cannot, it falls back to a relay, and you get connectivity instead of a support ticket. ## Now the trap Every mesh gives you an internal naming scheme, so you can reach peers by name rather than by overlay address. The mesh resolver answers those names with the peer’s overlay address, and only for machines on the mesh. That is the design and it is good. Here is what nobody puts in the quick-start guide. Those names are usually subdomains of a domain you own publicly. And if that public zone has a catch-all wildcard — which an enormous number of zones do — then public DNS will answer for your internal names too. A DNS wildcard matches at any depth, not just one label. So a name intended to exist only inside the mesh gets a public answer, pointing at whatever the wildcard points at, which is typically your main public host. While the mesh resolver is working, nobody notices. The mesh answers first, you get the right address, everything is fine. The problem appears the moment the mesh resolver is not answering: the client restarted, the tunnel dropped, someone disabled the resolver, the daemon is mid-reconnect. Then the query falls through to public DNS, and public DNS answers — confidently, successfully, with the address of a completely different machine. No error. Nothing fails. The name resolves. It just points somewhere else now. Sit with what that means for automation. Configuration management that connects to hosts by internal name, with credentials, running privileged operations. If a name silently resolves to a different machine, the automation connects to that machine and does what it was told to do. It may hand over a password on the way. ## Failing closed The fix is small and you should do it before you need it: publish an explicit record for the internal namespace in your public zone that points somewhere guaranteed to be unroutable. A more specific wildcard beats the general catch-all for that subtree, so a single record covering the internal namespace takes it out of the catch-all’s reach without touching the records everything else depends on. Point it at documentation-reserved address space — addresses set aside by RFC precisely so they can never route anywhere. The effect is that a fall-through now times out instead of reaching a live machine. That is a much better failure: slow and obviously broken, rather than fast and quietly wrong. Fail closed, not sideways. Then add the second layer, because DNS alone is not enough. Host key checking is what turns “this is a different machine” from an invisible event into a loud one. If your automation is configured to accept any host key without complaint — and a surprising amount is, because it makes bootstrapping new machines convenient — then it will connect to the wrong host without a word. The configuration you want accepts a key it has never seen (so provisioning a fresh machine still works unattended) but refuses loudly when a key changes . Those are different situations and they deserve different responses. A tool that treats them the same is one DNS blip away from an incident. ## Other things worth knowing The control plane is infrastructure. It handles enrolment, key exchange and policy. If it is down, existing tunnels generally keep working, but nothing new can join and policy changes do not propagate. Treat it accordingly — it is not a nice-to-have service, it is a dependency of your ability to reach everything else. Access policy is not the same as network reachability. The mesh gives every peer an address, and by default that can mean far more connectivity than you intended. A laptop should probably not be able to reach every storage node. Policy is where you say so, and it deserves the same review any firewall rule would get. Overlay addressing overlaps with things. Meshes commonly use carrier-grade NAT space, which is not routable on the public internet but is very much in use inside some ISP networks. Know which range you are on before you debug a connectivity problem that only affects one person’s home connection. Understand relayed versus direct. When hole punching fails, traffic goes through a relay, and your throughput and latency change character. It will still work, which is why you might not notice for months — until someone copies a large file and asks why it is slow. ## Worth it? Yes, and not marginally. Being able to address every machine you own by a stable name, encrypted end to end, regardless of where it physically sits, removes an entire class of network configuration from your life. Just publish the blackhole record first. The trap costs one line of DNS to close and it is not the sort of thing you want to discover by finding out which machine your automation has been talking to. ================================================================================ # Alert on symptoms, not on metrics https://philipptheserver.com/posts/observatory-monitoring/ ================================================================================ The first monitoring system I built alerted on CPU. It worked exactly as designed and was almost entirely useless. High CPU is not a problem. It is sometimes a sign of a problem, and it is sometimes a machine doing its job well. Alerting on it taught everyone the single worst lesson a monitoring system can teach: that alerts are things you glance at and dismiss. Once people learn that, the system is dead, and the outage it eventually catches correctly will be dismissed along with everything else. ## The rule that fixed it Alert on what the user experiences. Graph everything else. If a page is slow, alert. If a queue is not draining, alert. If a certificate expires in three days, alert. If CPU is at 90%, put it on a dashboard and leave me alone — I will look at it when I am investigating the alert that actually fired. This distinction is not about which metrics you collect. Collect everything; it is cheap and you will want it during an incident. It is about which ones are allowed to wake someone. The test I use: if this fires and I do nothing, will anybody notice? If the honest answer is no, it is a graph, not an alert. The corollary is uncomfortable: an alert that has fired ten times and been ignored ten times is not a monitoring gap, it is a monitoring bug , and the fix is to delete it or change it. Leaving it in place is choosing to train your team to ignore alerts. ## Black-box and white-box are answering different questions White-box monitoring reads a system’s own account of itself: metrics the service exports, queue depths, internal error counters. It tells you why . Black-box monitoring is an outside observer doing what a user does: fetch the URL, resolve the name, complete the handshake. It tells you whether . You need both, and the mistake is thinking the first can replace the second. A service can report itself perfectly healthy while being completely unreachable, because everything it knows how to measure is inside the boundary that broke. Every layer between the process and the user — DNS, the load balancer, certificates, firewall rules, the ingress — is invisible to it. Black-box checks are also the ones that catch the failures where nothing is broken at all , which brings me to the class of outage I find most interesting. ## The failure where every metric is green Here is a shape worth internalising, because it generalises far beyond the specific mechanism. DNS wildcards do not work the way most people assume. A wildcard record synthesises an answer for a name only if nothing exists beneath that name. Put any record — any type at all — under a name, and the name now exists , and the wildcard is forbidden from answering for it. The name then resolves with a perfectly successful response code and an empty answer. No error. No failure. Just no address. The practical consequence: a leftover record from a process that was supposed to clean up after itself can take a hostname off the internet, silently, while every neighbouring name served by the same wildcard keeps working perfectly. Query a random name under that wildcard and it answers. Query the real one and you get success-with-nothing. Now consider what your monitoring says. The service is up. Its metrics are green. Its host is healthy. Certificates are valid. Every dashboard is fine, because the failure is not in anything any of those things measure — it is in the resolution step that happens before anyone reaches the service at all. The only thing that catches this is an outside observer asking a public resolver for the name you actually publish and checking that the answer contains an address. Not that the query succeeded — that it returned something . I like this example because it is not exotic. It is a category: failures where every component reports success and the composition still does not work. Certificate chains that validate individually and not together. A load balancer routing happily to a backend pool that is empty. Auth that returns 200 with a redirect loop. If your monitoring only ever asks components how they feel, none of these are visible. ## What a good alert contains An alert is a message to a tired person. Write it for them. It should say what is broken in user terms, since that is what determines urgency. It should say how you know, so the first move is not re-deriving the query. It should point at the thing to look at. And it should be honest about severity — if it is not worth waking someone, it should not be able to. The failure mode here is alerts written as an expression and a name and nothing else. Six months later, at 3am, ProbeFailureRateHigh on a service you did not deploy tells you nothing about whether to care. ## Cardinality, briefly The other way monitoring dies is by becoming too expensive to run, and the cause is almost always labels. Every distinct combination of label values is a separate time series. Put a user ID, a request path, or anything else unbounded in a label and you have built a system whose cost grows with your traffic in a way you did not intend. The rule: labels are for things you would group by . If you would never write a query grouping on it, it is not a label, it is a log field. ## The uncomfortable part Good monitoring is mostly deletion. The instinct when something breaks is to add an alert for it, and after two years you have three hundred alerts, most of which have never fired usefully, and a team that has learned the notification channel is noise. Every alert should have to justify its continued existence. When was it last right? What did someone do about it? If it has never been right, it is not protecting you — it is using up the attention you will need for the alert that matters. I would rather have twelve alerts that everyone trusts than three hundred that everyone ignores. The twelve will catch fewer things. They will catch them at 3am, when someone actually reads them. ================================================================================ # Ceph without a vendor's price list https://philipptheserver.com/posts/ceph/ ================================================================================ The pitch for a SAN is that storage becomes somebody else’s problem. You buy a box, it has redundant everything, and there is a phone number. The pitch is honest. The price is also honest, and for a lot of organisations it is the single largest line in the infrastructure budget. Ceph is the other trade: ordinary machines, ordinary disks, replication in software, and storage becomes your problem in exchange for costing a fraction as much. Whether that is a good deal depends entirely on whether you are willing to learn it properly. Half-learned Ceph is more dangerous than the SAN you were trying to avoid. ## The model, briefly Ceph stores objects across a cluster and computes where an object lives rather than looking it up. That sounds like a detail and it is the central idea: there is no metadata server to become a bottleneck or a single point of failure, because placement is a function, not a table. The pieces you actually care about day to day: - OSDs — one per disk, roughly. These hold the data. Losing one is normal. - Monitors (MONs) — hold the cluster map and vote on it. These need quorum. - The CRUSH map — the description of your failure domains: which disks are in which host, which host in which rack. This is where you tell Ceph what “independent” means. Everything above that — block devices, a filesystem, an S3-compatible gateway — sits on top of the same object store. ## Failure domains are the whole design If you take one thing away: Ceph will faithfully protect you against exactly the failure you described, and no other. Three replicas sounds safe. Three replicas that happen to live on three disks in the same machine protect you against disk failure and not against that machine’s power supply. The default CRUSH rules usually do something sensible at the host level, but “usually” is doing a lot of work in that sentence, and the moment your hosts are not uniform it stops being true. So the first real design decision is not how many replicas — it is replicas across what . Across disks, hosts, racks, rooms? Write it down, configure it explicitly, and then verify it by asking the cluster where a given object actually lives rather than assuming. The related trap: a cluster that cannot satisfy its own rule will simply refuse to place data, and tell you it is degraded, and wait. That is correct behaviour and it looks exactly like a broken cluster if you were not expecting it. ## Four things that will bite you 1. A full OSD stops the cluster, not just the disk. Ceph is protective about running out of space, because a full cluster cannot rebalance itself out of trouble. Hit the threshold and writes stop — cluster-wide, not on that disk. Everything sharing that pool becomes read-only, which in practice means everything. Worse, utilisation is not even. Placement is pseudo-random, so some OSDs run hotter than average, and the fullest disk is what matters, not the mean. A cluster that looks 70% full can have a disk at 90%. Plan capacity against the fullest OSD, alert well before the threshold, and never let a cluster get comfortable above roughly two-thirds. You need the headroom to survive losing a node — because when a node goes, its data re-replicates onto the remaining disks, and if there is no room for it, one failure becomes an outage. 2. Recovery competes with your workload. When a disk dies, Ceph starts making new copies. That is the system working. It is also a large amount of I/O arriving at exactly the moment you have less hardware than usual. Left at aggressive settings, recovery can make a degraded-but-working cluster feel like a down cluster. Left too gentle, you stay at reduced redundancy for a long time, and the second failure is the one that costs you data. There is no universally right answer, but there is a wrong one: not having decided before it happens. 3. MON quorum is a different failure than OSD failure. Lose disks and you lose redundancy. Lose monitor quorum and you lose the cluster — everything blocks, regardless of how healthy the data is, because nobody can agree on the map. Run an odd number, spread them across genuine failure domains, and understand that two monitors is worse than one: one gives you no redundancy, two gives you no redundancy and a way to lose quorum by rebooting either machine. 4. The network is a storage component. Every write is replicated, so a write is several network transfers. Latency between nodes is latency in your storage. A cluster that behaves fine at low load can fall apart when recovery traffic and client traffic contend on the same link. This is the one people skip because network gear is boring and expensive, and it is the one that produces the most baffling symptoms — storage that is slow in a way that correlates with nothing on the storage nodes. ## What it actually gives you Setting the warnings aside, because they are warnings about a system worth running: You get storage that grows by adding a machine instead of by a purchase order. You get to lose a disk on a Tuesday and not care. You get to lose a node and still not care, if you built the failure domains right. You get one pool of storage that presents as block devices, as a filesystem, and as an object store, instead of three separate products. And you get the thing that is hardest to price: you understand your storage. When something is slow, you can find out why, because there is no opaque appliance in the way. That is worth a great deal at three in the morning, and it is precisely what you gave up when you bought the box with the phone number. ## If you are starting Start with more nodes and fewer disks each, rather than the reverse. Node count is what gives you real failure domains, and a cluster of three fat machines has fewer independent things to lose than one of six thin ones. Do not put the first production workload on it. Put the second. Run something real but survivable, lose a disk on purpose, watch recovery, fill it up in a test pool until writes stop so you have seen what that looks like. Ceph rewards operators who have already met its failure modes and punishes the ones meeting them for the first time under load. Then read the health output every day until it is boring. HEALTH_WARN is not a state to live in. It is the cluster telling you what will hurt next. ================================================================================ # Kubernetes, and when it earns its place https://philipptheserver.com/posts/kubernetes/ ================================================================================ I run both. A Docker cluster carries a large share of the workloads, and a Kubernetes cluster carries the rest. People find this inconsistent. It is the most deliberate decision in the whole estate. The framing that gets everyone into trouble is that Kubernetes is what you graduate to when Compose stops being enough. That makes it sound like a bigger version of what you already have. It is not. It is a different bargain: you hand over control of where things run and when they restart , and in exchange you get a system that keeps working when a machine dies. If you do not need the second thing, you have paid for the first for nothing. ## What you actually give up The thing nobody warns you about is that debugging changes shape. With Compose, a service is a process on a host you can name. Something is wrong, you SSH in, you read the logs, you see the process. The mental model is one hop deep. With Kubernetes, “where is it running” is a question with a query attached, and the answer changes. Between you and the process there is now a scheduler, a CNI plugin, a service abstraction, an ingress, and a set of health checks any one of which can be the reason nothing is responding. That is not a criticism. Every one of those layers is doing something you asked for. But it means your debugging is no longer “read the log”, it is “work out which layer is lying”. The first few times, that takes hours, and you will feel it acutely because the equivalent Compose problem would have taken minutes. You also give up a certain kind of quick fix. On a single host, restarting a container to clear a bad state is a legitimate move. In a cluster, whatever put the container in that state will do it again, on a different node, at a worse time. The cluster removes your ability to paper over things — which is good, and which does not feel good. ## What you get Machine failure stops being an incident. This is the whole thing. On a single host, a dead disk is a phone call. In a cluster with a real failure domain, it is a rescheduling event you read about later. Everything else Kubernetes gives you is a consequence of the control loop that makes this work. Deployment becomes declarative all the way down. With GitOps — a controller watching a repository and reconciling the cluster toward it — the deployment story becomes identical to the infrastructure story: the repository is the truth, and the cluster converges on it. Nobody deploys. They merge. The cluster notices. That is a genuinely different operational posture. There is no deploy script that can be run with the wrong arguments, because there is no deploy script. There is no “did the staging change get applied to production” because the answer is in git. Rolling back is a revert. Capacity becomes fungible. Once workloads are not pinned to hosts, adding a node adds capacity to everything at once, instead of to whichever service you decided lives there. ## When it does not earn its place If you have one machine, Kubernetes gives you nothing except the layers. A single-node cluster has the failure characteristics of a single node plus the debugging surface of a cluster. That is the worst of both, and it is where an enormous number of installations actually sit. If your workloads are not replicable — a database with local state, something with a license tied to a MAC address, a service that cannot tolerate being moved — then the scheduler cannot do the thing you are paying it for. You can pin them, and people do, but at that point you have a very elaborate way of running a process on a specific host. If nobody on the team wants to learn it, it will rot. This is the one people find rude and it is the most reliable predictor I know. Kubernetes has a real learning curve and a fast release cadence. A cluster nobody is curious about becomes a cluster nobody upgrades, and an unpatched cluster is a worse liability than the Compose setup it replaced. ## The split I actually run So the division is not ideological, it is about what each workload needs. Things that are stateless, replicable, and benefit from surviving a node failure go to the cluster. Things that are pinned by their nature, or that are simple enough that the cluster would only add layers, stay on Docker. The test I apply is: if this host died right now, do I want the system to handle it, or do I want to be told? Both are legitimate answers. Pretending only one is legitimate is how you end up with a database in a pod that reschedules itself away from its disk. The cost of running both is real — two deployment paths, two sets of habits. It is smaller than the cost of forcing everything into either one. ## Things I would tell myself earlier Learn the failure modes before you need them. Take a node out on purpose, on a weekday, while you are calm. Watch what happens. The whole value proposition is behaviour under failure, and if you have never seen it, you do not actually know whether you have it. Resource requests are not paperwork. They are how the scheduler makes decisions. Leave them unset and you have asked it to pack your machines by guesswork, and it will guess wrong under exactly the load that made you care. The ingress layer will surprise you. More outages come from the path into the cluster than from anything scheduling-related. Certificates, DNS, the exact behaviour of the controller when two things claim the same hostname — that is where the sharp edges are. Do not let the cluster become the place where architecture goes to be forgotten. It is easy to add one more deployment, then another, until nobody can say what runs there or why. The cluster does not organise your system for you. It just makes disorganisation survive node failures. Kubernetes is a good answer to a question a lot of people have not actually asked. Ask the question first: what do I want to happen when a machine dies? If the honest answer is “someone will notice and fix it, and that is fine” — then it is fine, and you have just saved yourself a great deal of YAML. ================================================================================ # A host you cannot rebuild is not running https://philipptheserver.com/posts/infrastructure-as-code/ ================================================================================ There is a moment every sysadmin knows. Something is down, you are tired, and you can see exactly which line in which config file would fix it. You are already logged in. Fixing it by hand takes eleven seconds. Doing it properly — edit the repository, commit, run the pipeline — takes four minutes. Take the eleven seconds and you have just created a machine nobody can rebuild. That is the whole argument for infrastructure as code, and it has very little to do with Ansible or Terraform. It is a claim about where the truth about a system lives. Either the repository describes the machine, or the machine does — and if it is the machine, then the knowledge is one disk failure away from gone. ## The rule I hold everything to one line: if a host cannot be rebuilt from the repository, it does not count as running. It sounds absolute because it has to be. A rule with exceptions is not a rule, it is a preference, and preferences lose to tiredness. The value of infrastructure as code is not linear in how much of your infrastructure is covered. It is closer to a step function. Ninety percent coverage gives you almost none of the benefit, because you still cannot answer the only question that matters — can I rebuild this? — with yes. You have to answer with “mostly”, and “mostly” means you will find out which ten percent was missing at the worst possible time. The practical test is not whether you have a playbook. It is whether you would be willing to wipe the machine right now and run it. ## What actually goes wrong The failure mode is not that people do not write automation. Almost everyone writes automation. The failure is drift : the code and the machine start out identical and then quietly diverge, one eleven-second fix at a time. Drift is nasty because it is invisible until you need it not to be. The playbook still runs green. It just no longer describes reality, because reality has grown a hand-added sysctl, a firewall rule someone opened for a debugging session in March, a package installed to test something. None of it is written down. All of it is load-bearing by the time you find out. The defence is to make the code the only path. Not the preferred path — the only one. In practice that means: - Deployments run from a pipeline, not from a laptop. A pipeline leaves a record, uses the committed state, and cannot be persuaded to skip a step because you are in a hurry. - Re-running is normal, not an event. If applying your configuration is scary, you will not do it often, and if you do not do it often, drift accumulates between runs. A playbook you run weekly stays honest. One you run twice a year is fiction. - Idempotence is not a nice property, it is the whole product. A run that reports “changed” when nothing should have changed is telling you something, and if you have trained yourself to ignore it, you have thrown away your only drift detector. That last point is the one people underrate. Once a playbook is genuinely idempotent, a changed=0 run is a proof — the machine matches the code, right now, verified by execution rather than by hope. That signal is worth more than the automation itself. I would rather have a slow, ugly, idempotent playbook than an elegant one that always reports changes. ## Dry runs are not optional Anything touching real infrastructure gets a dry run first. --check , plan , --diff , whatever the tool calls it. Not because I expect the change to be wrong, but because the difference between what I think a change does and what it does is exactly where outages live. There is a stronger version of this that I have come to rely on: to prove a change is only what you think it is, run it with your change removed and confirm the tool reports no difference at all . If the tool reproduces the entire existing state byte for byte, then whatever it reports with your change back in is genuinely only your change. That turns “I think this is safe” into something closer to a measurement. It has caught things I would have sworn were fine. A refactor that was supposed to be cosmetic, quietly changing the order of records. A default that looked inert and was not. ## What does not belong in the repository Secrets. Ever. Not encrypted-in-a-pinch, not “it is a private repo”, not base64 — which is not encryption and everyone knows it. Secrets live in a secret store and are referenced by name. The repository says which secret, never what it is. This is worth being rigid about because the failure is unrecoverable in a specific way: once a secret is in git history it is in every clone, every fork, every backup, and every laptop that ever pulled. Rotating it is the only fix, and rotation is exactly the thing nobody wants to do at the moment they discover the problem. ## Where the code should not go Infrastructure as code is not an argument for describing everything. Some things are genuinely better as data, some as documentation, and some should not exist at all. The trap is building an abstraction layer over a mess instead of removing the mess. A role with fourteen boolean flags to accommodate four hosts that drifted apart is not automation, it is drift with a YAML interface. The honest fix is usually to make the four hosts the same. I have written that role. Twice. The second time I noticed sooner. ## Why it is worth it The pitch for infrastructure as code is normally disaster recovery, and that is real but it undersells it. I have rebuilt a machine from scratch maybe a handful of times. I have read the repository to find out how something works hundreds of times. That is the actual return. A system whose configuration is written down is a system you can reason about without logging in, hand over without a three-hour walkthrough, and change without the specific fear that comes from not knowing what you are about to break. Recovery is the insurance policy. Comprehensibility is what you use every day. And the eleven-second fix at three in the morning? Do it, if the alternative is staying down. Then write the four-minute version before you go to bed, while you still remember what you did. The rule is not that you never touch a machine by hand. It is that the machine never keeps a secret from the repository overnight. ================================================================================ # OpenTaberna, and writing the wiki first https://philipptheserver.com/posts/opentaberna/ ================================================================================ OpenTaberna (https://github.com/OpenTaberna) is an open-source hospitality stack: a FastAPI backend, a customer frontend, an admin frontend, and a wiki. Ordering, fulfilment, payments, returns. The domain is not exotic, which is exactly why it is a good place to be strict about how a project is put together. The part worth writing about is not the API. It is the wiki. ## The documentation is a repository The wiki lives in its own repository (https://github.com/OpenTaberna/wiki) as Markdown files, and is published from there to wiki.opentaberna.de (https://wiki.opentaberna.de). It is not a wiki in the sense of a database somebody edits through a browser. It is text under version control, and it moves through pull requests like everything else. That one decision changes the character of the documentation completely. It can be wrong in a way that shows. A page in a hosted wiki drifts silently — there is no diff, no blame, no review, and no moment where somebody has to look at the change. A page in a repository is reviewed by the same person reviewing the code that made it stale. It can be required. If the docs are in the repository, “the docs are updated in the same change” is a rule a reviewer can actually enforce, because the change is right there in the diff. If the docs are somewhere else, that rule is a hope. I have watched the hope version fail on every project that tried it, including mine. It can be run locally. docker compose up brings up the same wiki software the published site runs, serving this repository’s pages, with no login and no setup wizard. So a page looks locally exactly as it will look published. That sounds like a small convenience and it is the difference between people previewing their writing and people guessing. ## What the pages have to cover The structure is the interesting bit, because it maps to the questions people actually arrive with: - What is this and how is it built — the architecture and the four repositories - How do I run it — the whole stack, locally, in one place - How does authorization work — the roles, the clients, and what the API enforces - The API — endpoints, the response envelope, the error model - The database — the schema as it is actually built - Orders and fulfilment — the lifecycle, payments, the outbox, returns - Configuration — every setting and where it can come from - Deployment — how it runs in production Two of those deserve comment. “The schema as it is actually built” is a deliberate phrase. Schema documentation drifts faster than anything else in a project, because it is written during design and then the migrations happen. A page that promises to describe reality has to be revisited when reality changes, and saying so in the title makes that obligation explicit. “Every setting and where it can come from” matters more than it sounds. Most configuration documentation lists the settings and omits the precedence — file, environment, default, flag — and precedence is exactly what you need at the moment configuration is not doing what you expect. That is the only moment anyone reads the page. ## The response envelope, and why consistency beats cleverness An API that returns a bare object here and a wrapped one there, an error as a string in one place and an object in another, is an API where every client writes its own special cases. The special cases are where bugs live, and they are invisible until a client hits the one endpoint that is shaped differently. Deciding the envelope and the error model once , writing it on a page, and then holding every endpoint to it is unglamorous and it is most of what makes an API pleasant. It also makes the documentation shorter, because the shape is described once instead of per endpoint. The same argument applies to authorization. Roles and clients defined centrally, with the API enforcing them, means “who can do this” is answerable by reading one page rather than by grepping decorators. ## Open source changes the standard Working on something public raises the bar in a specific way: you cannot rely on anybody knowing anything. There is no colleague to ask, no shared context, no “obviously it needs the database running first”. Everything a person needs has to be written down, and the fastest way to find out whether it is written down is that somebody tries and fails. That is uncomfortable and it is the most useful review a project gets. Every internal project I have worked on had gaps that were invisible precisely because everyone had already been told the missing thing in a conversation. A public project has no conversations to lean on. The related discipline is repository conventions — how issues are written, what a commit message says, what “done” means. Internally you can get away with these living in somebody’s head. Publicly they have to be written down, and once they are written down you notice they were never really agreed. ## What I would take to every project The wiki-as-repository pattern, without hesitation. Documentation that is not under version control is documentation that will be wrong, and the only question is when. Putting it in a repository does not make anybody write more — it makes not writing visible in a diff, which turns out to be the same thing.