Offline LLM regression gates · Concept
ModelLedger
A working TypeScript CLI scaffold that compares recorded model outputs and fails CI when quality, latency or cost gates regress.
Result — Deterministic quality, latency and cost gates that fail CI with a non-zero exit code the moment a recorded output regresses.
Problem — prompt and model changes ship on vibes. Without a gate, quality, latency or cost can quietly regress and nobody notices until users do.
What I built — an offline-first evaluation harness that checks recorded LLM outputs before a change ships. It validates fixtures, runs deterministic scorers and produces both a readable terminal brief and a stable JSON report.
Architecture — built-in gates cover exact match, required terms, valid JSON, latency budgets and estimated cost budgets. Passing suites return exit code zero; regressions return a non-zero code that blocks CI.
Tech — a TypeScript CLI with Zod validation and Vitest coverage that makes no provider calls and claims no semantic judgment. Typed scorer and output-loader boundaries leave a clear path toward live providers and model-based judges later.
