Prompt caching and context optimisation for harnesses

Jul 29, 2026

Using AI powered applications has often made us realize the limitations of large language models, sometimes at the expense of continued exploitation of context windows. While building AI powered applications though, it’s important to understand what we can and cannot optimize and thus direct our efforts to what we can.

Prompt Caching In Agents is an important overview of the work regarding the technique which is used for avoiding excessive recomputing by inference providers and language model labs.

Prompt caching rewards stable prefixes.

Since all the tokens get processed computed by the transformer in order: to minimize cache misses it’s best to keep the part that changes frequently at the end. The app sends the tool definitions, system prompt, as well as project specific instructions sometimes such as AGENTS.md files for coding agents.

The ordered token prefix the model provider feeds into the transformer can be different (or undisclosed) depending on the provider. There is no industry convention for this.

Anthropic explicitly states that its cache prefix is stated as

tools → system → messages

It also documents that tool definitions and configuration are compiled into a special system prompt around the caller’s system prompt. That is an Anthropic convention inherited by prompt caching, not something standardized by OpenAI, Google, or some standards body.

Cache is the reusable transformer state. Most providers price cached tokens ~10 times lower than uncached ones. Cache lives are short (~5 min); pay more if you want them to live long (useful for long running tasks like coding where an agent can run a script for 30+ mins)

If you modify tokens in the system prompt or the tools, you’re literally changing the entire history, and all the tokens will be processed again.

I guess in retrospect this was something incorrectly assumed: the dynamism of the capability surface containing dozens to hundreds of tools appearing during a long agent run. Early function calling assumed something like “this chatbot has eight functions.”

Which is why, a deferred loading approach is in the right direction.

[tool-search kernel → system → history] ← explicitly cached
                                      → discovered tool schema

Discovering a tool appends information rather than rewriting the beginning, allowing the original cache to survive.

Anthropic excludes deferred schemas from Claude’s initial rendered context and expands them inline only after discovery, OpenAI injects tool search discoveries at the end of context. Gemini currently has no documentation around this. Google recommends keeping roughly 10–20 tools active and doing dynamic selection in the application: stupid.

Normally every request conceptually processes ~4 things:

tools → system instructions → history → new user message

Explicit caching lets the app construct the cache boundary, instead of relying on prefix detection:

[tools → system → old history] → new message
└──── cached prefix ──────────┘

The provider stores the transformer’s processed state for that prefix. Future requests reuse it rather than recomputing those tokens. Different model providers expose this differently:

  • for Anthropic: put cache_control at a breakpoint. Everything before and including it forms the cached prefix.

  • for Google: create a named CachedContent resource containing immutable system instructions, tools, and/or contents, then reference its ID later.

All modern models produce significantly worse outputs the moment the get to around ~160,000 tokens (irrespective of their total input capacity). Way better to compact/handoff conversations at this point and continue with a fresh context rather than keep going. The app is responsible to carry out compaction mid-conversation and it must be fast & accurate. More on that in the compaction section coming soon

Loading an integration like github should not load all its functions into context. If github exposes 10 functions

github_create_issue
github_get_issue
github_get_me
github_get_pr
github_list_commits
github_list_issues
github_search_code
github_search_repositories
github_update_issue
github_update_pr

There are levels to laziness.

Solely having lazy loading for integrations themselves will blow up if an integration has a dozen or more tools. We need this on the function level as well. The way Alfred does this right now, alongwith provider specific deferred loading:

  • A small kernel stays eager, including system.search_tools and system.load_tool.

  • The initial user prompt can deterministically preload up to four relevant exact tools

  • Later, the model searches the lightweight catalog and loads one exact qualified tool name. Its full schema appears on the next turn

  • The remaining functions from that integration stay unloaded.

  • Newly activated tools are sorted into the tool list, not simply appended at the end

The basic thing is to realize the harness chooses what to submit; while the provider chooses how the native fields become model context.

I will get more into squeezing more out of the context window as I keep testing how Alfred works (these things are hard on your wallet :))

Addendum: what MCP 2026-07-28 changes

I wrote the above against MCP 2025-06-18. The current revision is 2026-07-28, and it is a rewrite rather than a patch. Protocol sessions are gone. The initialize handshake is gone. Three things I treated as harness problems are now protocol rules, and one of my bullets above is wrong.

The correction

Newly activated tools are sorted into the tool list, not simply appended at the end

That is wrong as a single rule. The spec now ships a Client Best Practices page with a section on prompt caching, and it says the opposite:

Append newly discovered definitions after the cache breakpoint rather than re-sorting the tools array, or route every call through a single stable call_tool({name, args}) meta-tool so the array never changes.

Both rules are right. They apply at different moments, and I collapsed them into one:

  • Sort at run start. A canonical order makes the next run with the same tool set hit the cache. Order the array once, before the prefix is warm.

  • Append mid-run. Once a prefix is warm, never touch the array. An insert and an append cost the same, because tools sits first. That is exactly the cost you are trying to avoid.

The third option in that quote beats both. Expose one stable call_tool({ name, args }) and the array never changes at all. The prefix then survives every discovery, for the whole run. You pay for it by losing provider-side argument validation and constrained decoding, so the harness has to validate every call itself.

Three harness problems became protocol rules

The tool set can no longer drift per connection. SEP-2567 removed protocol sessions and the Mcp-Session-Id header. The tools page now requires that the set "MUST NOT vary per-connection or as a side effect of other requests on the connection". It may vary by the authorization on the request, because credentials are per-request input. A server that needs cross-call state mints an explicit handle and takes it as an ordinary tool argument. So the tool list is a function of (server, auth) and never of history. That is the cache-stability property, written into the protocol.

Deterministic order is now a SHOULD, and prompt caching is the reason given. Verbatim from the tools page:

Deterministic ordering enables clients to reliably cache the tool list and improves LLM prompt cache hit rates when tools are included in model context.

It is still a SHOULD, so the harness canonicalizes anyway. Sort by code point rather than localeCompare if you hash the result: ICU collation depends on the build and the locale, so the same tool set can otherwise hash differently on two machines.

Freshness hints are mandatory. SEP-2549 added a CacheableResult interface. ttlMs and cacheScope are now required on tools/list, prompts/list, the resources/* lists, resources/read, and server/discover. public is the recommended scope for a tool list. The cache key is the method plus the parameters that affect the result, so each page caches on its own clock and there is no cross-page consistency guarantee.

One more: invalidation is opt-in now. notifications/tools/list_changed reaches you only if you opened a subscriptions/listen stream with toolsListChanged: true. The old HTTP GET endpoint is gone. The harness decides whether it wants the interrupt at all.

The new failure mode: invalidation is lossy

SEP-2575 removed SSE resumability and the Last-Event-ID header. A broken stream loses the in-flight request. Notifications are not redelivered. The server holds no subscription state across reconnections, so the client has to re-send subscriptions/listen itself.

A missed list_changed is therefore undetectable, and the consequence inverts the intuition. A lossy invalidation channel makes your prompt cache live longer and your catalog wronger. You can hand the model a schema for a tool the server dropped an hour ago.

The spec plans for this. Clients may re-fetch before the TTL expires on "an unexpected error on a tool call indicating the method was not found or the parameters were invalid". So the trigger set is three-way, not one-way:

  1. the notification, when it arrives
  2. ttlMs expiry, which is the only guaranteed path back to the truth
  3. a tool call that fails with an unknown method or invalid parameters

Treat ttlMs as load-bearing. It is not a nicety on top of the notification any more.

The cost that now dominates the one this post optimizes

Multi round-trip requests (SEP-2322) let tools/call return resultType: "input_required". The client answers on a retry that carries inputResponses and requestState, under a new JSON-RPC id. Those results must not be cached. Tasks moved out of the core protocol into an extension that polls with tasks/get.

Both make a run park, and park in the middle of a call. A park longer than the provider cache lifetime costs a full cold write on resume, whatever your tool discipline is. That single event can cost more than every token the lazy catalog saves.

Tool laziness protects the prefix. Only park discipline protects the cache. I optimized the first above and never mentioned the second.

There is still no tools/get

Worth saying plainly, because the docs make it look otherwise: get_tool_details in the three-layer example is a harness meta-tool, not a protocol method. Nothing in MCP fetches one tool’s schema by name. Layer 2 is you, paging tools/list and filtering locally, which is why the docs also tell you to memoize the definitions host-side, separate from what currently sits in model context.

The same page puts a number on when to bother: switch to progressive discovery once tool definitions take 1% to 5% of the context window. Below that, load everything and stop thinking about it.

Where Alfred landed

Alfred took the strongest option before the doc recommended it. There is one mcp.call tool, so the native tools array never grows with the catalog. The catalog revision travels as a tool argument, in the tail, so a catalog change never edits a schema. The objection that normally kills a single meta-tool is risk: one registered tool cannot carry a static risk tier per remote tool. Alfred resolves the effective tier at the dispatch gate from (connection, remote name, descriptor hash) and fails closed on drift, so the caching win costs nothing in safety.