llm-benchmark-harness
A thin benchmark harness for local LLM endpoints. It drives a fixed prompt set through an OpenAI-compatible endpoint and reports the two things most benchmarks split apart: how fast a model answers — TTFT / decode / prefill as a distribution, correlated with real macOS memory pressure and thermal throttling — and how well it answers, scored by an LLM-as-judge against a use-case "pack". The same code runs on an M1 (LM Studio) and an M5 (mlx) — only the config changes.
Why this one
Most local-LLM benchmarks report tokens-per-second and stop there. But "fast" is the wrong single number on Apple Silicon, where latency stays flat until the model spills past unified memory and then jumps — so a mean hides exactly the thing you'd feel. And speed says nothing about whether a model is actually good for your use case. ramcheck reports the distribution (P50/P95, CV% for consistency), pins it to host-sampled memory pressure and throttling from a separate process (never estimated from the request thread), and then judges answer quality over a pack — e.g. a neurodivergence-assistant pack — into a scorecard you can publish and others can reproduce on their own hardware. The aim is comparable, contributable runs, not one-off numbers.
A recurring lesson shaped it: a model that "thinks" before it answers will starve a tight token budget and look broken. The fix was to let models answer freely by default — closer to real use — and to surface that transparently rather than silently scoring it a zero. Measuring the endpoint fairly, including how it really behaves, is the whole point.
Judging the judge
There's a second-order question most benchmarks never ask: is the judge any good? A local model scoring other models can be miscalibrated or hand-wavy about its reasons. So the harness now turns the lens on the judge itself — a meta-evaluation that checks how closely it tracks a strong reference and whether its verdicts are actually argued (citing what an answer did, naming what would make it better, engaging the safety-critical cases) rather than rubber-stamped. That feedback even fed back into a sharper judging prompt. The old thinking-model trap got a clean fix too: reasoning is suppressed transparently, so a "thinking" model serves as a usable judge instead of stalling. The throughline: numbers you publish should be ones you can defend.
Under the hood
OpenAI-compatible is the only interface (no engine-specific code in the hot path). Two decoupled producers — a latency runner and a host sampler — are merged by timestamp. A use case is a pack (one YAML, data not code); the judge is pluggable (local or cloud). Output is Markdown + CSV. Detail in the repo.
Online repository
git.jkaindl.de/jkaindl/llm-benchmark-harness: code, configs, and the technical detail.
See also: zettelkasten