How we cut LLM inference cost by 35% without touching quality

At some point in 2024 our monthly inference bill crossed the line from “rounding error” to “line item with its own meeting.” What followed was a few months of unglamorous work that ended with costs down roughly 35% while our eval scores stayed flat.

Nothing here is clever. That’s rather the point.

Routing: stop using one model for everything

The single biggest win. We’d been sending everything to a frontier model because that’s what the prototype used, and nobody had questioned it since.

An audit of a week’s traffic found that around 60% of calls were tasks a much smaller model handles indistinguishably: classification, extraction into a fixed schema, reformatting, yes/no gating. We built a router — mostly static rules per task type, not a learned model — and sent each call to the cheapest model that passes that task’s eval suite.

That last clause is the safety net. Routing without evals is just praying with a spreadsheet. Every route has a quality bar; downgrades that fail it get reverted.

Caching: the request you don’t make

Three layers, in increasing order of subtlety:

1. Exact-match caching for identical requests. Embarrassingly effective for anything user-facing where many people ask the same question on the same day. (“What’s the leave policy?” spikes every December, who knew.)
2. Provider-side prompt caching for long shared prefixes. Our RAG system prompts plus tool definitions ran to thousands of tokens, resent on every call. Marking the stable prefix cacheable cut input cost on those routes dramatically for near-zero engineering.
3. Semantic caching, which I’ll be honest about: we tried it, tuned it, and eventually shrank its scope to a handful of high-traffic FAQ-style routes. False-positive matches — serving someone else’s almost-right answer — cost more trust than the tokens saved. Handle with care.

Context compression: send less, lose nothing

Our contexts were bloated in ways nobody noticed because nobody was looking. Retrieval stuffed in top-k documents regardless of length. Conversation history grew unbounded. Tool definitions for tools the current task couldn’t even use.

The fixes: re-rank retrieval down to what actually fits the question (six focused chunks beat fifteen loose ones on our evals — cheaper and better). Summarize conversation history past a window instead of replaying it verbatim. Prune tool schemas to the route’s actual tool set.

Average input tokens dropped by about a third, and answer quality went up slightly. Long noisy contexts weren’t just expensive — they were diluting attention from the chunks that mattered.

The dashboard is the deliverable

The lasting change wasn’t any single optimization. It was making cost per feature, per team, per model visible on the same dashboard as latency and eval scores. Once engineers could see that a feature cost X per thousand requests, they optimized it the way they optimize slow queries — without being asked.

Cost is a product metric. Treat it like one and it behaves.