Prompt caching & context optimization for harnesses

Using AI powered applications has often made us realize the limitations of large language models, sometimes at the expense of continued exploitation of context windows. While building AI powered applications though, it’s important to understand what we can and cannot optimize and thus direct our efforts to what we can.

Prompt Caching In Agents is an important overview of the work regarding the technique which is used for avoiding excessive recomputing by inference providers and language model labs.

Prompt caching rewards stable prefixes.

Since all the tokens get processed computed by the transformer in order: to minimize cache misses it’s best to keep the part that changes frequently at the end. The app sends the tool definitions, system prompt, as well as project specific instructions sometimes such as AGENTS.md files for coding agents.

The ordered token prefix the model provider feeds into the transformer can be different (or undisclosed) depending on the provider. There is no industry convention for this.

Anthropic explicitly states that its cache prefix is stated as

tools → system → messages

It also documents that tool definitions and configuration are compiled into a special system prompt around the caller’s system prompt. That is an Anthropic convention inherited by prompt caching, not something standardized by OpenAI, Google, or some standards body.

Cache is the reusable transformer state. Most providers price cached tokens ~10 times lower than uncached ones. Cache lives are short (~5 min); pay more if you want them to live long (useful for long running tasks like coding where an agent can run a script for 30+ mins)

If you modify tokens in the system prompt or the tools, you’re literally changing the entire history, and all the tokens will be processed again.

I guess in retrospect this was something incorrectly assumed: the dynamism of the capability surface containing dozens to hundreds of tools appearing during a long agent run. Early function calling assumed something like “this chatbot has eight functions.”

Which is why, a deferred loading approach is in the right direction.

[tool-search kernel → system → history] ← explicitly cached
                                      → discovered tool schema

Discovering a tool appends information rather than rewriting the beginning, allowing the original cache to survive.

Anthropic excludes deferred schemas from Claude’s initial rendered context and expands them inline only after discovery, OpenAI injects tool search discoveries at the end of context. Gemini currently has no documentation around this. Google recommends keeping roughly 10–20 tools active and doing dynamic selection in the application: stupid.

Normally every request conceptually processes ~4 things:

tools → system instructions → history → new user message

Explicit caching lets the app construct the cache boundary, instead of relying on prefix detection:

[tools → system → old history] → new message
└──── cached prefix ──────────┘

The provider stores the transformer’s processed state for that prefix. Future requests reuse it rather than recomputing those tokens. Different model providers expose this differently:

  • for Anthropic: put cache_control at a breakpoint. Everything before and including it forms the cached prefix.

  • for Google: create a named CachedContent resource containing immutable system instructions, tools, and/or contents, then reference its ID later.

All modern models produce significantly worse outputs the moment the get to around ~160,000 tokens (irrespective of their total input capacity). Way better to compact/handoff conversations at this point and continue with a fresh context rather than keep going. The app is responsible to carry out compaction mid-conversation and it must be fast & accurate. More on that in the compaction section coming soon

Loading an integration like github should not load all its functions into context. If github exposes 10 functions

github_create_issue
github_get_issue
github_get_me
github_get_pr
github_list_commits
github_list_issues
github_search_code
github_search_repositories
github_update_issue
github_update_pr

There are levels to laziness.

Solely having lazy loading for integrations themselves will blow up if an integration has a dozen or more tools. We need this on the function level as well. The way Alfred does this right now, alongwith provider specific deferred loading:

  • A small kernel stays eager, including system.search_tools and system.load_tool.

  • The initial user prompt can deterministically preload up to four relevant exact tools

  • Later, the model searches the lightweight catalog and loads one exact qualified tool name. Its full schema appears on the next turn

  • The remaining functions from that integration stay unloaded.

  • Newly activated tools are sorted into the tool list, not simply appended at the end

The basic thing is to realize the harness chooses what to submit; while the provider chooses how the native fields become model context.

I will get more into squeezing more out of the context window as I keep testing how Alfred works (these things are hard on your wallet :))