Eval Gauntlet
LLM regression testing · in-browser
Makes the case for per-case regression gates with a runnable proof: prompt v2 lifts all four aggregate scores yet breaks 2 cases v1 got right, and the gate names them, deterministically, on every run.
Synthetic benchmark, hand-authored mock model outputs. The parsing, scoring and regression logic is real and unit-tested.
Synthetic benchmark and mock model outputs. The harness logic is real and unit-tested.
Eval Gauntlet treats prompt changes the way CI treats code changes. The task under test is structured extraction: given a short infrastructure snippet, produce one security finding as strict JSON ({severity, control, evidence}). A hand-authored synthetic benchmark of 24 gold-labeled cases is scored against two static mock output sets standing in for prompt v1 and prompt v2, seeded with the failure modes real eval work surfaces: missed findings, wrong severities, hallucinated control ids and malformed JSON (truncation, single quotes, trailing commas, prose preambles). The harness parses each output through a strict two-layer validator (syntactic JSON, then schema shape), runs four scorers (exact match, per-field precision/recall/F1, severity-weighted accuracy so critical cases count most, strict-JSON validity), then compares the runs case by case into improved/regressed/unchanged verdicts with aggregate deltas. That comparison makes the core point: v2 lifts every aggregate yet breaks two cases v1 got right, which only the per-case gate catches. The benchmark is synthetic by design; the parsing, scoring and comparison machinery is real, fully deterministic and covered by 27 unit tests.
- TypeScript
- React
- Next.js
- Vitest
- Strict JSON validation
- Precision / Recall / F1
Architecture · gold labels → strict parse → multi-scorer → regression gate
Gold benchmark
24 hand-authored synthetic cases: an infrastructure snippet plus the expected {severity, control, evidence} finding.
Strict parsing
Each raw model output passes two layers: syntactic JSON, then schema shape. Trailing commas, prose preambles and missing keys all fail loudly.
Multi-scorer
Exact match, per-field precision/recall/F1, severity-weighted accuracy and strict-JSON validity, computed per run.
Regression gate
Prompt v1 vs v2 compared case by case into improved/regressed/unchanged verdicts, so a better aggregate cannot hide broken cases.
- Gold-labeled cases
- 24
- Scorers
- 4
- Mock outputs scored
- 48 (24 × 2 prompts)
- Unit tests
- 27 passing
What I'd improve
The mock outputs are static strings, so the harness scores hand-authored failure modes rather than a live model: the next step is wiring it to real model calls behind a rate-limited route, scored against a human-labeled gold set. Evidence matching is strict normalized string equality, which would punish valid paraphrases from a real model; a token-overlap scorer with a tuned threshold is the honest upgrade. And the regression gate belongs in CI, failing the build on any regressed case, not only in a browser demo.