If you ever used an agent harness like Claude Code, Codex, OpenCode, or Pi, there are some things that are unintuitive. Why does your first request already consume thousands of tokens? What are these millions of "cached tokens" consumed in just a few minutes? Why does it feel like a session gets more expensive the longer it runs?
I think these can be explained by looking at what the agent is doing behind the scenes. Each request represents an API call to the model provider. You can click on the requests to see the inputs and outputs.
Loading... (JavaScript required)
You can see that the underlying behavior is very simple.
- It sends a system prompt, tool definitions, and your prompt as a request to be completed.
- The model responds with some text and a tool call.
- The harness then executes the tool call on your device, and the results are then added to the next request to be completed.
- This continues until the model stops responding with a tool call.
Notably, the entire context is being sent back as part of the request. This would have been very expensive if not for the prompt caching. Because of how LLMs work, you can reuse a lot of computation from past tokens, and most providers will persist this for a short while so it's available across multiple requests. This means those cached tokens don't take much processing power, and the price for them is correspondingly cheaper.
Plotting the Request Costs
We can plot the costs per request, split into the three categories: input, output, and cache read. The dashed lines represent the points where I sent a new prompt.
Just by chance, apparently my provider had a problem on one request that caused the cached data to be lost, so you can see what the request would have cost without prompt caching.
Loading... (JavaScript required)
It's worth noting that cheaper is not free. Here, I intentionally used one long session to add multiple mostly-unrelated features. By the end, the cached input cost takes up a significant part of each request's cost, and therefore the total session cost as well.
We can also plot the costs cumulatively over a session, and also for each new prompt:
Loading... (JavaScript required)
Loading... (JavaScript required)
This shows the cost of a large context. Near the end of the session, I made a few smaller prompts to the model. Because of the existing context, it doesn't need to explore the project as much to do the changes. However, the price of carrying those context makes them more expensive than the first prompt, even though it doesn't need a lot of requests to implement the change.
Conclusion
I've seen and heard a lot of tips about reducing costs when using AI. Use one session for a defined task, use sub-agents to explore the codebase, use the harness' file-referencing feature so it includes the file contents without an agent turn, that sort of thing.
But once you know how these tools work behind the scenes, you can understand where those tips are coming from, what they are trying to optimize for, and when they will be helpful or harmful. And maybe you don't even need to memorize such tricks, you can just use your understanding to decide what actions to take to reach the optimal result.
Personally, I feel this is a more efficient and enjoyable way to learn. It's similar to learning tools like git. Once you understand its structure, you can just chain simple operations to get the repository to the state you want, instead of memorizing specific commands to do the exact thing you want to do.
So I hope this post can help you in using these AI agent harnesses, which can be quite powerful when used correctly.
Source Code
You can check out the visualization in its GitHub repo here, and the Pi recorder extension here.