(KV) Cache Rules Everything Around Me

TL;DR: The biggest cost to devs of an agent is re-reading its own context, which costs the provider close to nothing to serve.

Background

Pop quiz: what’s the most important cost when running an agent?

Total amount billed broken down by %. Assuming Fable 5 + 100% cache hit rate. Only the additional cache write cost is separated from input cost.

If you’re like most AI engineers, your first two guesses are wrong. Output tokens? Per token outputs are the expensive ones, but they end up being a small slice of your final bill. Input tokens? Kinda, but not the ones you write in your prompts + messages. The answer is cache reads costs developers the most (or cache writes, if your cache hit rate isn’t great). Technically both are input tokens, just at very different prices.

An agent is a loop: the model reads the context then emits a tool call, the tool returns some tokens, and the whole thing goes back into the model. Every turn re-reads everything that came before it. With a context of N tokens and T tool calls, you process roughly N × T total input tokens (cached and not) to produce a comparatively tiny number of output tokens. And since N grows with every turn, the cost of a session grows quadratically with its length and even more with increasing tool calls.

The cost of an agent

Non-infra-engineer summary

• When generating an output token, the entire past context (KV cache) needs to be read.

• You aren’t charged for the KV cache, because… it is already cached.

• When an agent stops to call a tool then resumes, the entire past context needs to be read again.

• You are charged for the KV cache this time, because… fuck you.

• Also that’s the largest part of your bill now because agents have a LOT of tool calls.

Cache Cash

When Martin Alderson modeled a 100-turn session from 60k tokens of context, cache reads made up ~76% of his Opus 5 bill. I wanted real-world trajectories, so I wrote a small simulator that replays Nvidia’s Open-SWE-Traces (tokens counted with the gpt-oss tokenizer, cache writes at 1.25x input price) with Fable 5 pricing:

Output tokens, which most people associate with the big price tag, are just 9-18% of the total bill. The rest is the model re-reading its own history.

Here is the same dataset priced across various models, assuming a 100% cache hit rate:

Those are the costs with perfect caching. At Claude Code’s reported 89% hit rate, the same workload costs $1.97M instead of $1.07M: 84% more!!

So small changes in cache hit rate can make a big difference to the total bill. Here are the rates measured by the OpenCode team:

Note that others have measured Claude Code at 95%, so take the exact rates with a grain of salt. But even going from 95% to 98% saves roughly 17% of the bill on this workload.

Why cache reads are (almost) free (but not for devs)

Ingredient 1: Input tokens - prefill costs compute.

Prefill is the step where the model reads the prompt. All of the input tokens go through the forward pass at once, which is very efficient on a GPU, and the cost is proportional to the number of tokens. This is what you’re paying for with “input tokens”. It’s also the only step a cache can save: a cache write is a prefill where the provider keeps a preprocessed representation of the input tokens (“the activations”): this is what the KV cache is.

Cache reads are input tokens that do not need compute because the results were saved.

Ingredient 2: Output tokens - decode costs memory bandwidth.

Decode is the step where the model generates tokens, one at a time. To generate each token, the model has to attend over the entire KV cache, which means reading the whole context out of high bandwidth memory (HBM) for every single output token. There is very little compute involved but an enormous amount of memory traffic. This is why output tokens cost 5x more than input tokens. Decode is the real bottleneck of inference.

Every single output token needs to read the entirety of the input, but luckily they are already cached.

Ingredient 3: A tool call is just a pause.

When an agent stops to run a tool, the KV cache for the whole context is kept in memory. Nothing has to be recomputed. The only cost is keeping those bytes around while the tool runs, and keeping bytes around uses neither compute nor HBM bandwidth (the two things inference is actually constrained by).

So my claim is that the cost of cache reads is primarily already included in the price of output tokens, at the decoding step.

This is consistent with provider actions:

• Cache reads don’t count towards input token rate limits on the Claude API. Rate limits exist to protect capacity, so that’s Anthropic telling you what a cache read costs them.

• OpenAI ran automatic prefix caching for over a year without charging anything extra for cache writes; you can’t serve LLMs at scale without proper caching The 1.25x cache write premium only arrived with GPT-5.6, to match Anthropic.

How is inference so profitable all of a sudden?

Frontier models are more of a commodity than ever, open-weight models are on the rise, and yet the labs’ reported inference margins keep climbing. How?

Share of tokens to open-weight models from Vercel’s AI gateway

The answer of course is the KV cache (+ the agentic change of workload): more tool calls means more cache reads, which means more profit.

It also explains the “subsidized” subscription plans. SemiAnalysis found that the $200/month Claude plan can yield up to $8,000/month of API-equivalent tokens, and OpenAI’s up to $14,000/month. At list price that’s a 40-70x subsidy, which sounds insane, until you remember that 90%+ of a coding agent’s tokens are cache reads and the true cost of serving them is a small fraction of the sticker price.

Which is also why, if you run agents at scale, self-hosting makes way more sense than it does for chat. Self-hosting an open weight model is the only way to keep the cheapness of the KV cache for yourself rather than handing it to the lab. At least until cache read prices fall further. Fable 5.1 cut them from $1.00 to $0.25 per million tokens, which suggests they will keep dropping.

Why routing doesn’t work (for agents)

The pitch for routing is: use a cheap model for the easy steps and the expensive model for the hard ones, and save on output tokens. But output is the small slice of the bill. The moment a second model touches the context, you end up paying for it from scratch, since a KV cache isn’t shared between models.

Assumptions: 100k tokens in an Opus 5 session, next step generates 1k tokens and gets 3k back from a tool. Compare Opus[1] doing it all or routing to Sonnet then back.

The tiny amount saved on output tokens was dominated by cache costs, so the “easy” step cost almost 3x more, and it was done by a dumber model. 🙃

If a router picks a model per request, it pays for this cache thrash every time, especially on the longest priciest contexts. The only way to amortize the handoff is to leave the cheap model in charge for a while, which is what subagents do, but…

Subagents mostly don’t work either (if you want them to share the full context[2]). Because output tokens are a minority of the cost and you end up paying to write the whole context into both models’ cache, it’s rarely worth it.

Conclusion

The economics of agents come down to one question: how many times does the context get read, and who pays what for each read.

If you build agents, measure your cache hit rate and the number of tool calls. The number of tool calls matters way more than the number of tokens per call. And if you run agents at scale, consider self-hosting to keep those cache savings for yourself.

And if breaking free from the tyranny of the KV cache sounds interesting to you (if only there was a new kind of foundation model that could help 🤫), reach out! Would love to jam on designs, ideas, and sci-fi.

Big thanks to Ke Deng, Kevin Zhang, and Sasha Sheng for helping me write/review this post.

[1] AFAICT APIs don’t currently have a way to say “also cache my output” - this is perhaps a blindspot for agentic workloads

[2] The “mostly” is because subagents work great when they don’t need the parent’s context. If the parent says “search the codebase for X and report back in 200 tokens”, the brief is tiny, and more importantly the 50k tokens of grep output never enter the parent’s context, which makes N smaller for every turn after.