Reliable email labelling

Email looks simple from a distance.

A message arrives. Alfred reads it. A label appears.

But the closer we got to making that feel automatic, the less it looked like a classification problem. It became a trust problem.

An inbox is full of almosts. A calendar invite that is really a webinar. A GitHub review comment that sounds urgent, but only applies to code that has not shipped. A security alert that looks like a robot until it says a secret was exposed. A shareholder notice with a deadline, but no real action for you. A thread that used to be follow_up, then ended with a done reply.

The hard part was not naming ten labels. It was making the right label appear quickly, quietly, and consistently enough that you stop thinking about the system. What follows is the order in which that broke, and how each break got fixed.

The first hurdle: speed was not the whole story

At one point, tagging felt slow. The obvious guess was the model.

It was not the model. It was almost never the model.

Every "tagging is slow" report turned out to be plumbing. A newly connected Gmail account did not always have a fresh users.watchinstalled, so new mail waited for the five-minute poll. Gmail's history API is eventually consistent with its own Pub/Sub, so a webhook could fire with nothing to fetch. A single Google credential flipping to needs_reauth silently stalled all tagging. The classifier itself was sub-second the whole time.

The most embarrassing version of this: for one stretch, production had not run an email-triage workflow in days. The prod trigger row was frozen on an old event shape and matched nothing. Tagging still appeared to work, because a dev server left running on my laptop was quietly triaging the same real mailbox over the five-minute poll. Labels live on Google's side, so the laptop's work showed up in the real inbox. The product looked alive. It was on life support.

If a personal assistant is late, it does not matter how smart it is. The first promise is presence. Most of the time we were not fixing intelligence. We were fixing delivery.

The fix was infrastructural, not magical: install the watch at connect time, re-seed the builtin workflows for existing users at boot so prod stops drifting, keep polling as a real fallback, and route real-time and catch-up through the same persistence helper. One path for dedup. One path for triage. One path for label writes.

The second hurdle: Gmail threads do not behave like rows

Gmail has a quiet trap: thread view unions labels across every message in a thread.

If the first email in a thread was fyi, the second was follow_up, and the final reply closed the loop, Gmail could show all three labels at once. Technically true. Product-wise, unreadable. We saw threads carrying action_needed / action_needed / urgent stacked together.

The first mitigation was "latest message wins": strip the siblings, label the newest. It read clean and it had a race. The find-siblings query only saw labels that were already written, so two triage runs in the same thread could each look, find nothing to strip, and both apply. The lesson stuck: when a find-siblings query depends on data a parallel write has not committed yet, you have a race, not a feature.

So the model of the data changed. The triage table moved to one row per thread, keyed on (user_id, source_thread_id) instead of one row per message. When the same multi-tag bug resurfaced on production months later, the last gap closed: a Postgres advisory lock around the read-apply-strip sequence, plus a recency guard so a run for an older message cannot clobber a newer classification.

The honest framing is that the invariant — at most one Alfred label per thread — lives in Gmail, not in our database. Postgres can serialize the external writes. It cannot declare them true. So the label writer treats Gmail as the source of truth for thread shape: ask for the siblings, strip every old Alfred label, apply the new one to the latest message. The user sees one tag. The system does the fussy work underneath, and self-heals when an old deployment left a stray label behind.

There is a quieter failure in the same family. Gmail reassigns a message's id when a thread is sent to or merged, so the id Alfred ingested can go dead. The label writer then calls modify on a stale id, Gmail answers 404 — requested entity not found — and the tag silently never lands: the row keeps a null applied_label_id while the inbox shows nothing. The repair is the same posture as the strip. Re-list the live message ids on the thread, repoint the triage row at one that still exists, and re-apply. A stored message id is a cache of where a message used to be, not a fact about where it is.

And the thread keeps moving after Alfred is done with it. A thread tagged awaiting_replyis waiting on the user; the moment the user actually replies, that label is wrong. But triage only ever fired on inbound mail — a deliberate exclusion so Alfred does not re-triage its own sent fan-out — so the user closing the loop changed nothing, and the stale tag sat there until the next inbound message, if one ever came. The fix is to treat the user's own outbound reply as a triage trigger too. A reply is the strongest signal there is about a thread's state, and an assistant that ignores it is reading half the conversation.

The third hurdle: cheap models need better eyes, not a bigger brother

The cheap classifier read things too literally. A GitHub secret-scanning alert — a Redis URI exposed in a public repo — came through as fyi, because the prompt had filed "alerts" under passive awareness. Later, self-initiated sign-in and magic-link emails came through as urgent, because the prompt literally listed "sign-in verification" under urgent. Both were the model doing exactly what it was told.

The first instinct was to escalate. When the cheap model is uncertain, call the boss model, give it user context, let it think harder. That worked as a design and made the hot path heavier than it needed to be. Email tagging happens all day. It should not summon the most capable model every time a sender looks unfamiliar.

The next instinct was the opposite: cache a confident decision per sender and skip the model entirely on the easy hits. That got killed in review, and the reason is the whole thesis of triage v3:

Never skip the model. A sender who is 99% newsletter can still send the one email that matters. Cheap models are not smart, but smart models are slow and expensive and unsustainable per-email at inbox volume. The smartness we actually need only shows up in the edge cases.

So v3 kept the cheap model on every single email and made it smart by feeding it deterministic observations before it reads the prose:

• what kind of sender this is — service envelope, bot, or human
• what this sender usually becomes, as a fed histogram, never a bypass
• whether the account is work or personal
• whether the user already replied in this thread
• whether Gmail marked the message important, promotional, or sent
• whether the content carries a high-precision signal like exposed credentials

The model still reads the email and still makes the judgment. It just no longer has to infer the entire world from raw prose. Fast on every message, smarter without becoming heavy.

The fourth hurdle: rules can become clutter

Every inbox creates tempting patches. "This sender is usually a newsletter." "This bot is usually harmless." "This phrase often means urgency." Left unchecked, those patches become a wall of exceptions that pass today's example and fail tomorrow's neighbor.

Alfred uses a narrower pattern: observations, one cheap classification, one second cheap pass only on a hard deterministic conflict, and a tiny override floor for things we want to catch with high precision.

The override floor is where restraint had to be enforced against myself. The obvious move was "any security keyword forces urgent." That would have re-created the Dependabot-noise problem and re-tagged every CVE FYI as an emergency. So the floor was narrowed to a single, near-unambiguous signal: a secret or key paired with an exposure verb — exposed, leaked, committed, compromised. A self-initiated "sign in" or "your code is 123456" contains none of those verbs, so it never trips the floor. That was the exact bug that opened v3, now fenced off by construction.

Bad tags are tuned from logs, not vibes. The triage.sender_extractionevent records the sender context, the observations, both passes, the override flags, and the todo decision — enough to debug a mistake without turning the user's inbox into a raw log dump.

The fifth hurdle: a fix in triage is not a fix everywhere

The manufactured-urgency problem was supposed to be closed. Then an evening briefing led with this, as the first thing to check tonight:

The Visitors.now grace period ended today — if you haven't upgraded yet, tracking is now permanently disabled and incoming events are being dropped, so that's the first thing to check tonight.

We had already fixed this on the triage side. A free-tier upgrade ask — "upgrade your plan," "trial ending," "tracking ends soon," "upgrade before you lose the free tier" — is the vendor's conversion lever, not a bill and not a real consequence. Triage now reads it as marketing or fyi, never urgent or payment, and a deadline in that kind of email no longer earns immunity from the manufactured-stake test.

But the briefing does not just render triage tags. It has its own ranking, and its top rank is reserved for "something wrong or at risk — an incident, a fired alarm, a security signal." That ranker read the alarming wording as an incident and led with it, independent of triage — and here reinforced by a stale action_needed tag still sitting on the older threads in that conversation. The lie we had taught triage to ignore walked in through a different door and got believed by a different agent.

A correction lives in one component. The judgment it corrects can be re-derived in another. Every agent that ranks or re-scores has to carry the same discriminator, or the fix only looks complete.

So the same owed-versus-conversion discriminator was ported into the briefing's top rank, written as a principle rather than a Visitors.now-specific patch. A real incident carries a real stake: production is down, a credential is exposed, a bill is actually owed, a person is blocked. Vendor conversion pressure dressed as an incident is none of those, however alarming the wording. And because the briefing reads triage labels that can lag, the rule carries an explicit robustness clause — this holds even if the triage label looks urgent; the label can lag, so judge the content — which means the briefing gets it right without anyone re-triaging the stale threads first.

The sixth hurdle: "failed" was a fact for three minutes

The briefing had already taught us one lesson about itself. It taught us a second, and this one was about money — the worst place to be wrong.

A morning briefing led with: "Payment to Railway Corporation failed." It had not failed. The charge went through.

Behind one $6.79 charge were four emails sent over seven hours: a request to confirm the payment, two notices that it was unsuccessful, and then a receipt. Stripe issues a numbered receipt only when a charge actually settles, so the receipt was proof the money had moved. A person reading those four in order gets it right without thinking — they read the last one, and the earlier panic dissolves.

Alfred read them as a set.

The gather step that feeds the briefing collected every payment-tagged thread into one flat bucket with the timestamps stripped off. The receipt was even sitting first in the array. But "first in an unordered list" is not "most recent," and the composer — handed a bag of snippets with no sequence — latched onto the most alarming words it could see: unsuccessful, unsuccessful. It asserted the negative.

"Failed" is not a fact. It is a timestamped observation — true at 00:15, false at 00:18. The durable fact is the current state after you fold in everything that came later. We had engineered ourselves below the human baseline by running a set operation over what was really an ordered stream.

The instinct was to look for the hard link — the invoice id that proves this failure became thatreceipt. There was none. The confirm and the receipt share a hosted-invoice secret, but the failure email is an orphan: it carries the merchant's account id (on every Railway email this user will ever get) and an amount, and nothing that ties it to a specific charge. So the honest model is not key-matching. It is correlation — same merchant, same amount, narrow window — and, decisively, a later positive event that supersedes an earlier provisional negative.

That forced a clean split, because two different jobs were hiding inside one fix.

The first job is to stop being wrong in the briefing, for every kind of back-and-forth thread, not just payments. That is a render-time guard: cluster the likely-related messages, hand the composer a time-orderedcluster instead of a flat bucket, and tell it that a later success supersedes an earlier failure — and if the cluster is ambiguous, hedge, never assert. This guard never touches durable state. A fuzzy cluster that mis-groups two unrelated emails should, at worst, produce a cautious "couldn't reconcile," never a wrong action.

And because the original bug was the composer ignoring a rule it had been given, the rule cannot live only in the prompt. A deterministic preflight runs after compose and before send: if the headline claims a failure and a later success exists in that cluster, the headline is blocked. The prompt is the suspenders; the preflight is the belt.

The second job is allowed to change state — close the stale "update your payment method" todo the failure spawned — but only because it is deterministic. It matches on extracted fields (account, amount, currency, polarity, a single unambiguous candidate), not on a model's read of the prose. And even then it speaks carefully: "Railway sent failed-payment notices, but a matching receipt arrived minutes later — treating as resolved," not "that failed charge definitely became receipt #2654-5078," because the link is inferred, not proven.

Only deterministic matchers get to change the user's state. Everything fuzzy stays render-time, where the worst case is a hedge instead of a confident lie.

There was a quieter problem underneath the loud one. Even with perfect reconciliation, a resolved $6.79 charge should never be the most important thing in a morning briefing. The slot is allowed to say "quiet day" and didn't — it promoted the only item it had. Being wrong and being noisy are different failures, and a correct-but-trivial headline still spends trust. That one ships on its own track.

What it became

Alfred now classifies email across ten labels: urgent, action_needed, follow_up, awaiting_reply, meeting, fyi, done, payment, newsletter, and marketing.

In the dev database, all ten are in use across 307 thread-level triage rows. A taxonomy is not real until the product has to live with it.

What the system really earned was a memory of how sorting goes wrong:

• delivery can be mistaken for intelligence
• a Gmail thread is not one message, and the invariant lives in Gmail
• a bot is not always noise
• urgency can be manufactured, and so can a literal-minded prompt
• cheap models get better when code hands them the right facts
• corrections should come from observed misses, not one-off rules
• a fix in triage is not a fix in the briefing — every agent that ranks has to carry the same judgment
• a label is a fact about one message; a thread is a story, and "failed" is only true until the next message arrives — fold the stream, never flatten it

The result should feel almost boring. The email arrives. The label is there. The thread has one current state. And you move on. That is the product.