llm

Serving language models on your own hardware, and measuring whether they are any good.

• One Local Endpoint for Every Agent Session

  18 August 2026 llm security python

  An IDE assistant, a terminal agent and a background daemon can each be configured to send code to a different endpoint, and nobody can answer where code actually goes without checking every tool individually. This article routes all of them through one local proxy that logs and can refuse requests, with a complete Docker Compose example and an honest account of what it doesn't guarantee.

• Treating Issue Bodies as Untrusted Input

  14 August 2026 security agents llm

  An agent that reads issue bodies is reading text a contributor fully controls, and treating that text as instructions is an injection surface no amount of careful wording closes reliably. This article builds a typed action schema that rejects anything a compromised model output might try, and tests the rejection without ever calling a model.

• Self-Hosted LLM Inference: Serving, Benchmarking and Agent Guardrails

  7 August 2026 llm agents testing security

  Running models on hardware you own removes a category of decision about where debugging context goes, at the cost of a real gap on the hardest reasoning tasks, and neither fact is worth much without a way to measure it. This covers serving, a repeatable benchmark harness that replaces "it feels smarter" with a number, and the guardrails that have to exist before an agent is allowed near anything that changes state, with runnable code for all three.

• Scoring Tool Calls by Parsed Structure Instead of String Equality

  3 February 2026 llm testing

  Comparing a model's tool call against an expected one by exact string match fails on harmless formatting differences and a substring check lets wrong calls through by accident. This shows how to parse both sides into a structure first, canonicalise it, and compare field by field.

• Why a Leaderboard Score Does Not Predict Your Workload

  30 January 2026 testing llm

  A public leaderboard score describes performance on a broad, general set of tasks that has little in common with a narrow production workload such as calling a fixed set of tools with a strict schema. This walks through building a small task-specific evaluation and running candidate models against it instead.

• Making a Benchmark Deterministic Against a Server Tuned for Interactive Use

  27 January 2026 testing llm

  A model server tuned for a pleasant chat experience samples its output, so the same prompt run twice gives two different answers and a benchmark score moves for reasons that have nothing to do with a regression. This shows how to pin what can be pinned and measure variance for what cannot.

• An OpenAI-Compatible Proxy in Front of a Local Model Server

  23 January 2026 llm fastapi api-design identity

  Local model servers speak a shape close enough to the OpenAI API that existing SDKs and tools can point at them unmodified, but close enough is not the same as safe to expose. This builds a small FastAPI proxy that adds authentication and a health check that cannot itself trigger a model load.

• Serving Several Local Models on One GPU with On-Demand Loading

  20 January 2026 llm performance architecture

  One GPU can hold one large model at a time, but different tasks want different models and restarting a server by hand does not scale past a handful of requests. This describes a proxy that loads a model on first request, keeps it warm, and swaps it out for the next one automatically.

• Sizing One GPU for Local LLM Inference: VRAM, Quantisation and KV Cache

  16 January 2026 llm performance

  Fitting a local language model onto one GPU means budgeting three separate pools of VRAM — weights, activations and a KV cache that grows with context length — and the wrong cache format can silently multiply latency without raising an error. This walks through the arithmetic and a way to measure it.

All topics · All writing