testing
Tests that fail when the thing they describe breaks, including for infrastructure.
-
Documentation as a Repository: Publishing a Wiki.js Site from Reviewed Markdown
Documentation kept in a hosted wiki drifts silently, because nothing forces a reviewer to look at it when the code it describes changes. Treating docs as a repository, reviewed through the same pull requests as code, turns that silence into a diff someone has to look at. This covers what changes when documentation moves into git — as it does in the open-source OpenTaberna project — with a runnable Wiki.js setup and a CI check that fails a pull request when source changes without its docs.
-
Testing Infrastructure Code by Executing Its Real Expressions
A test that reimplements an Ansible expression's logic in Python proves two independent implementations agree today, not that the shipped expression is correct, and the two can drift apart while both keep passing. This article renders the actual expression through Ansible's own Templar and filters, using a recursive dict merge as a concrete case where a hand-written paraphrase gets it wrong.
-
Keeping Documentation Honest with an OpenAPI Snapshot Diff
Hand-written API documentation and the code behind it drift apart silently, because nothing runs the docs to notice. This article generates the real OpenAPI schema from a FastAPI app, commits a normalised snapshot, and fails CI the moment the two disagree, with the full app, test and workflow.
-
Self-Hosted LLM Inference: Serving, Benchmarking and Agent Guardrails
Running models on hardware you own removes a category of decision about where debugging context goes, at the cost of a real gap on the hardest reasoning tasks, and neither fact is worth much without a way to measure it. This covers serving, a repeatable benchmark harness that replaces "it feels smarter" with a number, and the guardrails that have to exist before an agent is allowed near anything that changes state, with runnable code for all three.
-
Verifying an Agent's Work Against Reality, Not Its Own Report
An agent process can exit cleanly and report success while having changed nothing, or while its tests silently didn't run. This article builds a separate verification step that checks git state and re-runs the real test command, and wires it into CI as an independent job the agent cannot influence.
-
Testing an Agent Harness Without Ever Calling the Model
A coding agent's permission decisions are ordinary deterministic code, but testing them by running the model end to end is slow, expensive and non-reproducible. Separating the decision layer from the model and recording real tool-call shapes as fixtures makes the whole thing testable with an ordinary unit-test suite, in milliseconds, with no API key required.
-
Testing That a Canvas Diagram Actually Painted Pixels
A canvas element can exist, have the right size, and show a completely blank drawing surface, and a DOM-based test suite will pass every one of those assertions. This covers why canvas content needs a real browser and a pixel-level assertion to test properly, with a complete Playwright example.
-
DNS Wildcards and Empty Non-Terminals: The Answer That Is Not an Error
A stale ACME DNS-01 TXT record can turn part of a domain into an empty non-terminal, and RFC 1034's wildcard rule then refuses to cover it — not as a bug, but as specified behaviour. This walks through the closest-encloser algorithm behind that refusal, reproduces it with a BIND container and a small zone file, and gives a script and a record-lifecycle pattern that catch the problem before it reaches production.
-
Separating 'Cannot Ever' From 'Currently Broken' in Sweep Automation
A sweep that checks a list of targets for a feature has to decide what a failure means, and a response code alone cannot tell it whether that failure was expected or is a regression. Here is how to make that decision explicit, keep it loud when it should be, and test it so the escape hatch does not become a second way to hide problems.
-
A DNS Canary CronJob as a Regression Test for Split-Horizon Resolution
A pod's search domain or CoreDNS configuration can resolve a public name to an internal blackhole, and once that bug is fixed it tends to come back the next time someone touches the Corefile for an unrelated reason. This walks through a CronJob that resolves a small, deliberate set of names on the exact path real pods use, so the regression fails a Job instead of a customer's request.
-
Sync Waves and the CRD-Before-Consumer Ordering Problem
Applying a CustomResourceDefinition and a custom resource that depends on it in the same Argo CD sync can fail intermittently, and removing the CRD from git can prune it in a way that deletes every instance of that resource cluster-wide, not just the ones this Application owns. This works through sync waves and a prune guard that make both failures impossible, with a reproducible kind cluster test proving it.
-
Alerting on Symptoms Instead of Metrics: Designing Alerts People Do Not Ignore
Most monitoring failures are not missing data, they are the wrong alert on good data, and the class of outage nobody catches is the one where every metric stays green. This works through the difference between alerting on a resource number and alerting on what a user experiences, and includes a runnable Prometheus and blackbox_exporter stack plus a tested DNS check that catches a failure no metric-based alert can see.
-
A Multi-Service docker-compose That a New Contributor Can Actually Start
A backend that needs an identity provider, a database, a cache and a time-series store fails unpredictably under a naive docker-compose file because none of it waits for the others to actually be ready. This shows the healthcheck, seeding and end-to-end test that make `docker compose up` a reliable one-command environment.
-
A 500 That Was a Misconfigured Client, Not an Auth Failure
Code that assumes an OAuth2 introspection response always carries an active field works right up until the resource server's own client credentials are wrong, at which point every request that reaches it throws instead of returning a clean 401. This article separates the response shapes introspection can actually return and gives a tested client that handles all of them.
-
Building the Dry-Run Path First, and Testing That It Sends Nothing
A dry-run flag added after the real logic is written tends to fall out of sync as the real path grows new side effects nobody remembers to gate. This describes structuring a tool so planning and execution are separate from the start, and writing a test that proves dry-run mode sends nothing.
-
Scoring Tool Calls by Parsed Structure Instead of String Equality
Comparing a model's tool call against an expected one by exact string match fails on harmless formatting differences and a substring check lets wrong calls through by accident. This shows how to parse both sides into a structure first, canonicalise it, and compare field by field.
-
Why a Leaderboard Score Does Not Predict Your Workload
A public leaderboard score describes performance on a broad, general set of tasks that has little in common with a narrow production workload such as calling a fixed set of tools with a strict schema. This walks through building a small task-specific evaluation and running candidate models against it instead.
-
Making a Benchmark Deterministic Against a Server Tuned for Interactive Use
A model server tuned for a pleasant chat experience samples its output, so the same prompt run twice gives two different answers and a benchmark score moves for reasons that have nothing to do with a regression. This shows how to pin what can be pinned and measure variance for what cannot.
-
UUID Case Normalization as a Recurring Cross-Service Bug
An uppercase UUID from one service and a lowercase one from another look identical to a person and different to a dict, a cache, or a database index, which turns one entity into two without raising an error. Here is a minimal reproduction of the bug and a boundary-normalisation fix with a test that keeps it fixed.
-
A Layered Storage Benchmark: Isolating Device, Network and Protocol Bottlenecks
A single throughput number from an RBD volume or a CephFS mount hides whether the limit is a slow disk, a saturated replication network, or overhead in Ceph's own protocol path, and each of those has a completely different fix. This builds a benchmark methodology that measures the device, the network and the protocol separately before measuring the full stack, using fio, iperf3 and rados bench against a single-node Ceph container a reader can run without a real cluster.
-
chmod -R and the setgid Bit: Why a Four-Digit Mode Is Not Enough
Fixing shared-volume permissions with chmod -R 775 looks like a correct fix and passes an immediate check, but it clears the setgid bit that made new files inherit the directory's group, so the fix quietly regresses days later. This covers how the octal digits actually work, why the fix has to treat files and directories differently, and a way to test it that a one-off manual check will not catch.
-
Ansible Role Idempotence: Why a changed=0 Run Is the Only Proof You Have
A role that reports changed on every run has stopped detecting drift, and nothing in Ansible's exit code tells you so. Here is how that happens module by module, and how to make the second run's changed=0 an automated check rather than something you eyeball.