Diff your context, not your prompts

A few months ago I wrote that context engineering is the job now — that when an agent misbehaves, the fix is almost never a better adjective in the system prompt, it’s discovering the context window had turned into a landfill. I ended that piece with a line I kept repeating to my team: we diff contexts, not just prompts.

Then I went looking for the tool that actually does that, and it wasn’t there.

The gap every observability tool leaves

The tracing platforms are good at one thing: showing you a list of calls with their payloads attached. LangSmith, Langfuse, Phoenix, AgentOps — all variations on “here is turn 8, here is its JSON.” Useful. But it’s the wrong shape for the question I actually have when an agent breaks at turn 8, which is never “what was sent” in isolation. It’s three questions at once:

• What exactly did the model see at that moment — every token, in order?
• What changed since turn 7 — what got injected, what got silently evicted?
• What did it cost, and how much of it was waste?

A multi-megabyte JSON blob answers none of these. You end up eyeballing two payloads side by side, counting braces, trying to spot the difference by hand. For a discipline whose whole premise is that the diff between contexts is where the bug lives, that’s absurd. We have git diff for code precisely because reading two versions in full and spotting the change is a job for a machine.

So I built the equivalent for context windows. It’s called ctxdiff, it’s open source, and it does one thing the tracing tools don’t: it treats your agent’s context as something you diff, profile, and replay — turn by turn.

What it looks like

One line — you wrap your client, run your agent as usual, and every call’s context is recorded locally. Then ctxdiff view opens a self-contained dashboard:

That’s a two-agent research pipeline. You can scrub turn by turn and watch the context evolve: a system rule quietly evicted, a RAG chunk injected, a schema that changed. Green added, red evicted, yellow modified — the git-diff grammar, applied to what the model saw.

The three questions, answered

What changed. Every message, content part, and tool schema becomes a content-hashed block. Diffing two turns is then a comparison of two ordered lists of hashes — added, evicted, modified, with a character-level inline diff on the modified ones. The turn a critical instruction fell out of the window becomes a single red line instead of a needle in a JSON haystack.

Where the budget went. A per-turn breakdown of tokens by role — system, RAG, history, tool schemas — plus something I find myself using constantly: schema-bloat detection. It cross-references the tools you registered against the tools you actually called, and flags the ones that are pure dead weight, taxing every single request. Half your context budget spent on schemas the model never invokes is a shockingly common finding.

What it cost. Both Anthropic and OpenAI bill a cached prefix at a fraction of full price — but only while that prefix stays byte-stable. One dynamic timestamp near the top of your system prompt and you’re paying full freight every turn. ctxdiff finds the exact character where the prefix breaks, attributes it to the block responsible, and counts the re-billed tokens. I refuse to print a dollar figure (provider prices drift, and a tool that fakes precision is worse than one that admits the gap), but it’ll tell you precisely what to move and why.

What it is, honestly

It’s early — v0.3. But it’s real: it wraps OpenAI, Azure, Anthropic, Gemini, Bedrock, any OpenAI-compatible OSS endpoint, and LangChain; it handles async clients and OpenAI’s Responses API; it attributes calls per-agent so a multi-agent run reads as separate stories on one timeline; and it’s local-first by design — your context payloads, the most sensitive data in your whole stack, never leave your machine. No SaaS, no telemetry, one dependency. The capture path is fail-open: a bug in the debugger can never take down the app it’s debugging.

You can see the whole thing without wiring up anything:

pip install ctxdiff
ctxdiff demo          # builds a sample multi-agent trace and opens the dashboard

No API key, no setup — it builds a realistic run and opens the dashboard above in about two seconds.

Why I bothered

The context-engineering post argued that context should be a build artifact — versioned, diffable, eval-gated like any other change. You can’t hold a discipline to that standard without the tooling to actually diff the artifact. Prompts got their debuggers years ago. Contexts didn’t have one. Now there’s a start.

If you build agents, try it and tell me where it’s wrong — that’s the fastest way it gets good. And if you’ve ever lost an afternoon to a JSON log trying to figure out what your model actually saw: I built this because I was tired of losing that afternoon too.