An agent loop's cache bill: explicit, automatic, and the one your gateway hides

An agent loop resends its entire history on every turn. The system prompt, the tool schemas, every previous tool call and its result — all of it, every iteration. By turn ten, the conversation dwarfs the static prefix in front of it, and without caching you re-prefill the whole thing at full price each time.

Prompt caching is supposed to fix this. What I learned this week is that “prompt caching” means three different things depending on who is serving your tokens, and only one of the three cares what your framework does. Here’s what I changed, what I measured, and where the numbers surprised me.

The change: cache the conversation, not just the prefix #

My Rust agent framework (harness-rs) already did the textbook thing for Anthropic’s API: one cache_control breakpoint at the end of the static prefix — system prompt plus tool schemas. Those bytes are identical on every call, so they cache once and read cheap forever after.

The conversation itself was unmarked. On Anthropic’s API, caching is strictly opt-in per block: no marker, no cache. So every iteration re-prefilled the entire growing history at full price, right behind a prefix we had carefully made cheap.

The fix is one more breakpoint, on the last cache-capable block of the final message:

/// An agent loop appends to its history and resends it whole; without this
/// breakpoint every iteration re-reads the entire conversation at full price.
/// Marking the final block makes this turn's request next turn's cache hit:
/// Anthropic matches the longest previously-cached prefix, so only the newly
/// appended blocks are paid at write price.
fn mark_history_breakpoint(messages: &mut [AnthropicMessage]) {
    for msg in messages.iter_mut().rev() {
        for block in msg.content.iter_mut().rev() {
            let slot = match block {
                AnthropicBlock::Text { cache_control, .. }
                | AnthropicBlock::ToolUse { cache_control, .. }
                | AnthropicBlock::ToolResult { cache_control, .. }
                | AnthropicBlock::Image { cache_control, .. } => cache_control,
                // Thinking blocks can't carry cache_control; walk back.
                _ => continue,
            };
            *slot = Some(CacheControl::ephemeral());
            return;
        }
    }
}

Each turn’s full request becomes the next turn’s cached prefix. Only the newly appended blocks — the model’s last reply and the new tool results — get paid at write price (1.25x on Anthropic). Everything older reads at 0.1x.

While I was in there I also promoted cache writes to a first-class usage field. Reads were already surfaced; the premium that bought them was only in a debug log. You can’t answer “what did caching actually save?” if half the ledger is invisible.

Measuring it: three providers, three semantics #

I built a scripted agent trace — fixed system prompt, three tool schemas, eight rounds of tool call and result, byte-identical across runs — and pointed it at every backend I could reach.

DeepSeek: automatic, and very good #

DeepSeek caches prefixes automatically. No markers, no configuration, hits reported in usage and billed at roughly a quarter of the miss price. The eight-round trace:

call0: prompt=1010  hit=0     miss=1010     ← cold
call1: prompt=1528  hit=896   miss=632
call2: prompt=2046  hit=1408  miss=638
...
call8: prompt=5154  hit=4608  miss=546
totals: 79.8% hit rate, input cost down 59.1%

Every call after the first hits on essentially the whole prior history (rounded down to 64-token blocks) and pays full price only for the newly appended round. This is exactly the behavior my Anthropic breakpoint buys explicitly — DeepSeek just does it for you.

Two gotchas worth knowing. The cache is written asynchronously: fire requests back-to-back and one will occasionally miss a beat, reading zero where the previous call should have seeded it. And DeepSeek also exposes an Anthropic-compatible endpoint (api.deepseek.com/anthropic) which reports usage with proper Anthropic semantics — input_tokens counts only the uncached remainder — but it ignores explicit cache_control markers. I ran my A/B there: with and without the history breakpoint, the results were byte-for-byte identical. The cache underneath is the same automatic one.

That endpoint had one more surprise: it strictly validates that assistant turns in thinking mode carry their thinking blocks. My first scripted trace — with fabricated assistant messages — got rejected with a 400. A real agent loop passes, because the framework echoes thinking blocks back verbatim. An accidental integration test, and my framework’s thinking round-trip passed it.

The pooled relay: caching you didn’t order #

I also ran the trace through the LLM gateway I use daily — one of those relay services that fronts a pool of upstream accounts. The results looked like caching, but wrong: hits appeared without any markers, cache_creation was always zero, and the hit sizes bounced around between calls — 2213, then 1809, then 2823 — on a monotonically growing conversation.

The explanation: the backend caches automatically, but the relay routes each request to whichever pool instance is free. Each instance has its own cache. Whether you hit depends on whether you landed on the instance that served your previous call. No session affinity, stochastic hit rate — my trace averaged about 60%, with zero control over it.

This is the strongest argument I’ve seen for cache-aware routing in inference gateways (what llm-d and the Kubernetes Gateway API Inference Extension are building): the cache is only as good as the router’s ability to send you back to it.

Anthropic direct: the only place your framework matters #

On Anthropic’s real API, none of this is automatic. No breakpoint on the history means zero cache hits on it, every turn, forever. Modeling my trace at Sonnet prices: the prefix-only strategy pays about $0.079 in input costs over nine calls; with the history breakpoint it’s $0.037 — a 54% cut even after the 1.25x write premium, growing with every additional round.

So the change matters exactly where caching is explicit, and is harmless everywhere else. That’s the right shape for a framework default.

The end-to-end check #

Synthetic traces are tidy; frameworks are not. I ran the real harness-rs adapter — new breakpoint, new usage plumbing — through a live tool loop against DeepSeek’s Anthropic endpoint: real model turns, real thinking blocks echoed back, fake file contents fed to its tool calls. Seven steps, 71.5% cache hit rate, every step’s read/write visible in the loop’s usage totals.

Then I did the same for my Go framework (agent-go), which had a different problem: it dropped usage entirely at the SDK boundary and estimated token counts with a tokenizer. Cached-token fields existed in the responses all along; nobody read them. After wiring usage through the provider (and opting streams into stream_options.include_usage — the final usage chunk arrives with empty choices, which the stream loop used to skip), the same growing-conversation test reports 71.2% hits end to end. Two frameworks, two languages, same trace shape, matching numbers.

What I’d tell you to do #

If you run an agent loop against Anthropic directly: mark the tail of your conversation, not just your prefix. It’s a few lines, and it’s the difference between 0% and ~80% history cache hits.

If you run against DeepSeek or OpenAI: you’re already getting cached — but only if your context is byte-stable. Append-only history, stable tool ordering, no timestamps in the system prompt. The cache doesn’t forgive rewrites.

If you run through a relay: measure before you assume. You may already have caching you didn’t know about, with a hit rate set by pool routing luck rather than anything you control. If the relay is yours, session affinity is the cheapest cache optimization you’ll ever ship.

And in all cases: surface the cache fields in your usage accounting. Reads, and writes. A number you don’t measure is a number you can’t defend in a cost review.