Multi agent harness architecture

Jul 25, 2026

Properly building AI agent harnesses requires getting familiar with some characteristics of how LLMs work and knowledge of conclusions around what their limitations are and what can hinder their responses. There are things that need to be tested out regardless of benchmarks, and testing can be hard given the variety of models and their effort levels. LLMs carry the entire conversation history right up to the last token (last word of the response) back into the consideration for spitting out the next token. This creates the problem of context pollution which exacerbates further responses and causes the so-called hallucinations.

Not one model for everything

We have to remember that we are thinking of building a harness to mitigate these issues. One way to go forward is not to make the same model perform all the tasks required to complete the request; since that bloats context. Since models come in various capabilities and trade-offs, it'd be in our best interest to realise places where we can dramatically improve results by using a swarm of smaller, cheaper models if we break down a complex task into simpler bits.

This philosophy will not only improve token usage and costs, it'd also result in faster and more efficient work as compared to conventionally dumping all information to a single model and letting it do all the work; big or small.

This would require experimenting with various pairs of cheap/expensive models to arrive at a combination which will give the best possible results with the most reasonable tradeoffs for your application's use case. The way I kind of envisioned this overall was to have a central language model sufficiently capable to keep track of a task, and given the ability to spin up a swarm of small models depending on the nature of the task. The small models should carry out independent, small scoped tasks and submit the result or ideally its summary in a shared state. The "boss" model can supervise this and make its conclusions from there on where to take the task next, or to mark it as complete. We'll get more into this soon.

The integration problem

This formula sounds simple and straightforward; the problem with an AI personal assistant is that it also needs access to your integrations and if they're present; can severely bloat the context because tools are loaded all at once. A github integration can have multiple functions associated with it to search PRs etc. A personal assistant has to support countless integrations and loading them into context every time would severely exacerbate the results, when most of the integrations won't be needed anyway.

This blog inspired me to actually realise something like lazy loading of integrations was possible which I had earlier dismissed due to my complete ignorance of explicit prompt caching.

The key here is that the boss should be able to load integrations and their function schemas on demand if it needs to. For this to be actualised we have to equip it with the ability to search through a catalog of integrations and update the catalog as we go on.

v1 of the harness

It started with a naive implementation that looked like:

All tool implementations + schemas are registered at server boot. Only some schemas are exposed to the model on a given turn. A chat run starts with: activeIntegrations: []. So the model initially gets: system.* tools. And the prompt includes a connected-summary block saying things like:

calendar.list_events, calendar.create_event — the user's calendar
gmail.search, gmail.read_message, gmail.send_draft — the user's email

So, for example: if the model needs Gmail, it calls:

system.load_integration { slug: "gmail" }

via the system.load_integration tool (exposed to the boss by default) does not import code; it adds a slug to run state. Next LLM turn rebuilds the model tool surface from that state.

Dispatcher runs that system tool. It checks the workflow allowlist and returns

{ ok: true, slug: "gmail" }

dispatch-tools sees that result and mutates run state:

activeIntegrations = ["gmail"]

On the next LLM turn, resolveSdkTools(["gmail"]) exposes

system.*            (this constituted 18 tools!)
gmail.search
gmail.read_message
gmail.send_draft

In a nutshell are 5 key parts:

  • registry = everything server knows how to run
  • activeIntegrations = non-system tools currently visible to the model
  • system.* = always visible bootstrap/control plane
  • load_integration = add integration slug to activeIntegrations for next turn
  • dispatcher = validates/runs/stages against global registry

Apart from many structural flaws this would fall apart if there are a lot of functions for a specific integration; which is crucial for any personal assistant. Loading each of their schemas and names in chat even when we'd need one or two of the defined functions is undesirable.

I, and most users will be willing to trade a few seconds of latency for better outputs.

v2 of the harness

system.load_integration
system.spawn_sub_agent
system.await_sub_agent
system.read_user_context
system.read_scratch
system.write_scratch
system.promote
system.remember
system.list_instructions
system.forget_instruction
system.edit_instruction
system.resolve_todo
system.suggest_todo
system.web_search
system.fetch_url
system.create_artifact
system.append_artifact_page
system.update_artifact

All these functions and their schemas were appended on each turn which was 21822B or ~5453 tokens per turn. v2 drops it down to <1.5k tokens. 14 system calls are kept in the dark only to be loaded on demand.

The architecture I moved toward comprises of a system.search_tools call and a function for loading the tool, rest all be lazy loaded by exact name.

Which is far better than

system.* always visible
load_integration("github") exposes all github.*

Finally we'll do something like:

search_tools("read issue 218 from github")
  -> github.search
  -> github.get_issue
  -> github.get_pull_request

load_tools(["github.get_issue"])
  -> next turn exposes only github.get_issue schema

The tool registry

If search_tools is weak, the model never sees the right schema.

{
  name: "github.get_issue",
  integration: "github",
  title: "Get GitHub issue",
  description: "...",
  aliases: ["read issue", "github issue", "issue details", "ticket"],
  entities: ["issue", "repo", "pull request"],
  verbs: ["get", "read", "summarize", "inspect"],
  requiresConnection: "github",
  riskTier: "low",
  relatedTools: ["github.search", "github.get_pull_request"]
}

I structured a registry like this. Most of this can be derived or required alongside the tool definition. For MCP/imported tools, derive the baseline automatically:

name + description + input schema fields + integration/server name

Then optionally allow overrides:

discovery: {
  aliases: ["open pr", "pull request details"],
  tags: ["code", "github", "pull_request"],
  relatedTools: [...]
}

So we don't have to maintain a giant separate list, and if we hide system.fetch_url, system.web_search, etc., the agent still needs to know those capabilities exist. That is much smaller than exposing every schema, but enough for the model to know it can fetch URLs.