I discovered the same thing when analyzing the coding agent usage costs at my job (cache reads were even more dominant). At first I was surprised and thought it was just me, but as I dug deeper I saw it was consistent for all our users and across different models and frameworks.
There is surprisingly little research on cache-to-cache multi-agent communication, especially for real-world environments with tool calls, etc. I am exploring this...
Thanks for posting this! Clarifies a lot. The N*T → $*$+$ is wild...this explains my recent $900 runaway astra bill from hundreds of turns + tool use. Compelling case for tool use outside agents. Maybe free cache read during prefill is the one-shot answer? I think I need to go back to the drawing board.
This was a great article, thank you for sharing it. The tables and graphs really helped me understand your points, I really appreciate the extra effort you put into that. Love a good visual.
I guess the question is - why is it that the most competitive lab is pricing kv cache reads at 2% of input, vs 10% standard? You would expect some lab to make it 0 and take significant market share. There has to be some cost they are trying to cover.
Having said that, I am sure that part of what is going on is that they rely on users mainly reading the sticker input/output token costs as what they will realise on use, neglecting the most important cost as you mentioned above.
i always struggle with context window while using coding tools such as Claude Code. Compacting...also does not help much. Great article ! easy to understand.
I discovered the same thing when analyzing the coding agent usage costs at my job (cache reads were even more dominant). At first I was surprised and thought it was just me, but as I dug deeper I saw it was consistent for all our users and across different models and frameworks.
There is surprisingly little research on cache-to-cache multi-agent communication, especially for real-world environments with tool calls, etc. I am exploring this...
Thanks for posting this! Clarifies a lot. The N*T → $*$+$ is wild...this explains my recent $900 runaway astra bill from hundreds of turns + tool use. Compelling case for tool use outside agents. Maybe free cache read during prefill is the one-shot answer? I think I need to go back to the drawing board.
This was a great article, thank you for sharing it. The tables and graphs really helped me understand your points, I really appreciate the extra effort you put into that. Love a good visual.
I guess the question is - why is it that the most competitive lab is pricing kv cache reads at 2% of input, vs 10% standard? You would expect some lab to make it 0 and take significant market share. There has to be some cost they are trying to cover.
Having said that, I am sure that part of what is going on is that they rely on users mainly reading the sticker input/output token costs as what they will realise on use, neglecting the most important cost as you mentioned above.
i always struggle with context window while using coding tools such as Claude Code. Compacting...also does not help much. Great article ! easy to understand.