PhilippTheServer

llm

Serving language models on your own hardware, and measuring whether they are any good.

  1. One Local Endpoint for Every Agent Session

    OPENAI_BASE_URL=http://localhost:8080/v1 for every AI tool: one local FastAPI proxy that logs each request and refuses internal-only projects.

    llm security python

  2. Treating Issue Bodies as Untrusted Input

    Prompt injection via GitHub issue bodies: fence untrusted text and accept model output only as a fixed JSON action schema, tested without a model.

    security agents llm

  3. Self-Hosted LLM Inference: Serving, Benchmarking and Agent Guardrails

    Ollama OpenAI-compatible /v1/chat/completions: a repeatable benchmark on your own tasks and a dry-run plus --approve guardrail for infrastructure agents.

    llm agents testing security

  4. Scoring Tool Calls by Parsed Structure Instead of String Equality

    Scoring LLM tool calls: parse the arguments JSON first so key order and 5 vs 5.0 pass, wrong tools fail, and extra fields follow per-case rules.

    llm testing

  5. Why a Leaderboard Score Does Not Predict Your Workload

    LLM leaderboard scores do not predict a narrow workload. A 20 to 50 case task eval with a deterministic JSON schema scorer instead of an LLM judge.

    testing llm

  6. Making a Benchmark Deterministic Against a Server Tuned for Interactive Use

    LLM output still varies at temperature 0 with a fixed seed because batching changes float order. Repeat each prompt and compare mean and stdev.

    testing llm

  7. An OpenAI-Compatible Proxy in Front of a Local Model Server

    /v1/chat/completions behind a FastAPI proxy: bearer API-key auth, unbuffered SSE streaming, and a /healthz that cannot trigger a model load.

    llm fastapi api-design identity

  8. Serving Several Local Models on One GPU with On-Demand Loading

    llama-swap config.yaml with cmd, ttl and healthCheckTimeout: loading llama-server models on demand and unloading idle ones on a single GPU.

    llm performance architecture

  9. Sizing One GPU for Local LLM Inference: VRAM, Quantisation and KV Cache

    llama-bench -ctk q8_0 -ctv q8_0 vs f16: estimating VRAM for GGUF weights plus a KV cache that grows with context, and catching slow KV cache types.

    llm performance

All topics · All writing