Stop testing prompts by vibes
We gate every prompt and retrieval change behind an eval suite, the same way we gate code behind tests. Here's the setup, and what it caught.
A confession: for the first few months of running LLM features in production, our release process for prompt changes was “try a few inputs, looks better, ship it.”
This works right up until it doesn’t. Ours stopped working the day a one-line prompt tweak — adding a politeness instruction, of all things — quietly broke structured JSON output for an entire module. The model started wrapping its JSON in a friendly sentence. Parsers disagreed with the friendliness.
That was the day evals stopped being a research topic and became CI.
The shape of it
Nothing exotic. Three suites, run on every change to a prompt, a retrieval setting, or a model version:
Accuracy. A few hundred curated input/expected-output pairs per feature, drawn from real (anonymized) production traffic. Exact match where outputs are structured. LLM-as-judge with a rubric where they’re prose — and the judge gets audited quarterly against human ratings, because judges drift too.
Safety. Injection attempts, PII-leak probes, off-scope requests, the greatest hits from our own incident history. Every past incident becomes a permanent regression case. This suite is the only one where a single failure blocks release outright.
Latency and cost. p50/p95 per feature, plus tokens in and out. A prompt change that adds 40% more input tokens is a cost regression even if quality holds. We’ve rejected “improvements” on this basis alone.
Green suites → ship. Red accuracy → investigate. Red safety → no discussion.
What it caught (a sampler)
- A model version bump that improved fluency and halved accuracy on Urdu-language queries. Multilingual users would’ve eaten that silently for weeks.
- A retrieval change that lifted average relevance but tanked one department’s answers, because their documents were formatted differently. Averages lie; slice your evals.
- My own “obviously better” rewrite of a system prompt, which lost to the ugly original by eleven points. I keep that one around for humility.
The part people skip
Building the harness is a weekend. Curating the dataset is the job. Ours grew from 50 hand-written cases to a few thousand, and every single one that came from a real user failure is worth ten synthetic ones.
The habit that makes it compound: when production surprises you, the last step of the incident isn’t the fix. It’s adding the case to the suite. Do that for a year and you have something no framework can sell you — an executable definition of what “working” means for your system, with your users, in your domain.
Vibes don’t version. Evals do.