Multi agent harness architecture
Most AI apps begin as a chat box.
You type. The model answers. The moment ends.
Alfred is trying to become something different: a personal assistant that keeps working when the window is closed. It reads new mail. It prepares briefings. It asks before taking risky actions. It can break a task into smaller investigations. It remembers enough state to resume after a deploy.
That required a different center of gravity.
Not a bigger prompt. A harness.
The shape
At the heart of Alfred is a durable run. A run is stored in Postgres. It has a workflow slug, state, transcript, current step, trigger, status, and cost attribution. A BullMQ worker claims one step at a time. After each step, the state is committed. If the process dies, another worker can resume from the last checkpoint.
For user-authored workflows, the shape is intentionally simple: boss-turn calls the model once. dispatch-tools executes the tool calls, records results, and either loops back to the boss or parks the run for human approval.
The model does not own the loop. The runtime does.
That means Alfred can checkpoint between turns, meter each model call, stage outbound effects, pause for approval, compact long transcripts, and recover after a worker crash without pretending the model is an operating system.
The boss
The boss is the planning agent. It sees the brief. It chooses tools. It decides whether to answer directly or delegate a focused investigation. It reads results and turns them into the final message.
Tools are not loose strings. They are registered actions with schemas, risk tiers, policy rules, and integration boundaries. A tool call is routed through the dispatcher, which validates input, enforces allowed integrations, resolves autonomy versus gated mode, writes an audit row, then executes or stages.
That gives Alfred a product property most prototypes lack: the same tool path can serve chat, workflows, approvals, smokes, and future agent-executable todos.
The worker
The worker is what makes the boss dependable. It heartbeats while a step runs. It re-enqueues immediately when a run advances. It sweeps for stale runs left behind by deploys or crashes. It snapshots scratchpad state when a run completes.
We saw that clearly during the chat latency investigation. A run that looked like "the model is slow" was not slow because of the model. The model calls totaled 8.5 seconds. The six-minute wait came from orchestration gaps: loop re-entry reused the same step attempt number, collided with the unique step key, and waited for stale-lease recovery.
The fix was small. The effect was product-level. Make step attempts monotonic, and the next step can run now instead of waiting for the sweep.
The scratchpad
Delegation needs memory, but not all memory should be trusted equally. Alfred uses a namespaced scratchpad. Sub-agents write to their own scratch.<subId>.* space. The boss can read those findings and promote selected pieces to shared.*.
That promotion is the validation moment.
Redis holds the live scratchpad during the run. Postgres receives a terminal snapshot for audit and replay. Fast while working. Durable when done.
The human gate
An assistant connected to email and calendar should be useful before it is fully trusted. So Alfred separates suggestion from action.
A todo suggestion can be created autonomously. It has no real-world side effect.
Sending an email, creating an event, or touching an external system goes through action staging when policy says it should. The run parks. The approval card appears. The user can approve, edit, reject, or reject and end the run.
The model does not get to nag its way past a boundary. The boundary is code.
Some boundaries the user cannot lower either. The autonomy toggle exists to stop Alfred asking permission to read things. But a few tools carry a blast radius that is no longer only the user's — a Railway or Vercel redeploy runs on a shared team's production, because a workspace token cannot be scoped down to one project. So high-risk tools always stage for approval, a floor the toggle cannot override. Policy can make a tool more gated. It can never make it less gated than its risk demands. The convenience control stops exactly where irreversible, cross-tenant action begins.
When a tool result is poison
The worker's whole promise is that no single step can take the run down with it. One bug tested that directly. A tool returned content with a NUL byte in it, and Postgres refuses to store a NUL byte inside a text value. The step could not commit its result, so it retried — and failed to commit again. The run did not crash and it did not finish. It wedged, looping forever on a result it could never persist.
The fix has two halves, and both are code, not prompt. Sanitize tool results at the dispatch boundary, so a poison byte never reaches persistence in the first place. And give the worker a backstop it owns: a step that fails to make progress a few times in a row dies instead of looping. A tool is allowed to return garbage. It is not allowed to hold the run hostage.
Durability is not only resuming after a crash. It is refusing to spin on a step that can never commit. A runtime that can be wedged by one bad string is not durable, however many checkpoints it writes.
The larger vision
The harness is the reason Alfred can treat the app as one assistant instead of several features.
- Email tagging is a background workflow.
- Todo suggestions are a passive materialization of open loops.
- Briefings are a timed render of the day.
- Producing a PDF, a deck, or a doc is a create-artifact run, streamed into a side panel and editable in place rather than exported as a dead file.
- Meeting prep can become a gather-and-compose run.
- Agent-executable todos can later become runs with approvals, progress, interruptions, and completion reflected back on the checkbox.
These do not need separate agent runtimes. They need different briefs, triggers, policies, and surfaces over the same durable substrate. The same dispatch path can now read an arbitrary URL on its own, pick a cheaper or more capable model depending on how hard the turn is, and record why it made each call — all without a second runtime to maintain.
The chat box is only one doorway. The worker is what keeps the lights on.