Agent Control

At a glance

A chat that hiccups is forgiven. A scheduled agent that fails silently once is distrusted for a month. Everything below is sequenced around that fact.

Agent Control at its density test: three failures, two runs waiting on input, outputs queued behind them. Fictional data, my working prototype.

• Defined the four-phase agent lifecycle and the run-level model every surface follows

• Wrote the card-boundary routing contract. It stayed the operative spec through handoff

• Designed HITL as a spectrum rather than a switch: block, choose, confirm, notify, with trust earned by track record

• Set the sequencing for the whole workstream: evals, then hardening, then UX polish

• Shipped Agent Control in a simplified form: the library, stopping an agent from chat, states grouped into anwhy is action hub. The full attention model stayed on the shelf

• By late July: 341 production runs in a 24-hour window across 74 graphs, at an 86.8 percent pass rate against the 90 percent gate. Publishing the miss is the point

The run was a black box

Analyze a 50-page lease into a clause-by-clause abstraction. Watch permit deadlines across hundreds of projects. Create risk records from climate data on a schedule, with nobody watching. Scout ran each of them as a compiled graph, its topology frozen before the first step, executing as a background job while only the card lived in the chat. What varied was whether the steps worked.

What it could not do was tell you what was happening. Which step is it on. What broke. Did it work at all. A scheduled run that quietly failed left no signal anywhere except the one agent page you would have to already suspect to open.

Two runs failed 33 days ago. Nothing told anyone.

The fix started with vocabulary, mapped in March working sessions with my design partner: Building, Triggering, Executing, Evaluating, one address system for every surface. Evaluating loops

back to Building because trust is re-earned, not granted once.

Everyday users do not want to inspect. They want output they can trust.

A team meeting surfaced something slightly uncomfortable: almost everyone arrives at a graph that already exists. Only the technical builder ever authors one. My contribution was the model that made the insight actionable: personas as role by proficiency, because role is assigned and proficiency is earned. That model inverted the builder.

Describe the job in conversation, the agent wires the graph, open the graph only to read it.

Inspection is what people fall back on when the system has already failed them. What an everyday user cares about is whether the output is accurate and consistent. So the target was a run legible enough that you rarely need to look, and unmistakable when you do.

Output got a standard, not a blank slate. I set the default contract for every agent's result: report a summary back into the chat, open the full output in the canvas rather than dumping it inline, save the artifact to the file vault. Builders could customize past those defaults; the point was that nobody had to decide from scratch. And the reasoning behind any step stayed one click away inside the card.

Oversight is a spectrum, not a switch, and trust moves rightward only on track record. Two rules kept my gates honest: hard gates only, and timing follows the trigger, because nobody is there at 2am to answer.

Every record an agent created got a link back to it, so verifying meant one click rather than a trip into Salesforce. The error card spec fixes the structure: what happened, impact, recovery, and whether production data was touched. The engine already tracks committed side effects per run, so it is a read, not a wish. For customers running operations on this data, that line is not negotiable.

The three fields came from a trace of 535 production runs across April: half graph and node issues, a third configuration. The user never sees that taxonomy. It decided which three questions the error card has to answer.

The shipped card, from a Jun 22 run. Three recovery paths, ranked, each naming who can act: you, your document, or your admin. The permit reference is redacted; everything else is as it ran.

The card is the boundary

Scout has a core agent, the one users talk to in every chat. The user’s mental model says, “I am talking to Scout.” The architecture says multiple systems may be listening: the core agent’s stream, a pending interrupt’s response path, the send-ahead queue, a mission card’s HITL resume, and the launch contract seeding inputs from history.

Three production bugs made that gap concrete: messages silently dropped during approval, sessions stuck after users typed mid-prompt, and cancel clicks misread as tool failures. Two of the three shared one root cause: user input had no routing rules.

My first answer was the opposite rule: lock the chat input while an agent runs. It survived eight days. Locking protected the run by breaking the conversation, which was backwards.

The contract that replaced it is simple: the card is the boundary. Agent execution lives inside a visually distinct card, the card owns every specialist interaction, and the chat input routes to the core agent always, even mid-run. Type while an agent works and Scout answers you, aware of its progress. Click inside the card and you are talking to the specialist.

Inside the card you talk to the specialist. The chat input reaches the core agent, always, even mid-run.

The card at a hard gate, from my mock-data demo: the run pauses, the card owns approve and reject, and the composer tells you where your words will go.

The rule generalized: any contained agent interaction is a card with its own color, and five states carry the run, folded down from the runtime's eleven statuses. Error, HITL interrupt, and cancelled branch from any step.

Working, step 4 of 7: the run names its step while it runs. The black box before this said only that the agent was running. Mock data, my own demo.

The scoping was a sequencing argument

I set the order for the workstream: evals, then hardening, then UX polish. Evals meant the mission pass-rate suite behind the gate we set in May: 90 percent, with fewer than two reported bugs per two-week period, before any new scope.

The hardest call was what we called principal agents, the multi-turn agents a user could switch to inside a session. I took them out of the July cycle: too large to rush, and nobody could yet say what one excluded. Strengthening the core agent bought more than bolting on a layer the infrastructure and UX could not yet carry. We all wanted principal agents. Wanting them was not an argument for building them in July. They are sequenced for Builder v2, where the shape turns genuinely meta: an agent that builds agent graphs.

The same question killed auto-categorization. Labels accumulate until none of them mean anything, which Jira already taught everyone, so cards filter by connector instead. The icons were already on the cards, which made the better answer the cheaper one. Automatic retries never shipped on purpose, because retrying a write-enabled agent duplicates records.

Every element proposed for a surface answered these five first. Most of the cuts in this section died here.

When our design director asked in Slack for agents in the main menu, referencing a mock, with pilot pressure behind the request, I pushed back on the franticness and asked for thirty minutes to map the real options first. My design manager agreed and timeboxed it. We reframed the problem as discoverability rather than one proposed UI, and the picker that answered it fits the longer-term menu vision instead of fighting it.

None of this happened in a calm quarter. The business was converting eight pilots, chasing thirty new logos, and answering "why not just use Claude?" from prospects.

That sequencing ran wider than my own backlog. I co-facilitated the May planning kickoff, introduced DMAIC as the quality frame, and split the work into three streams, leading the agents stream myself. The cycle moved from feature-driven to quality-first.

Then it broke, and that set the bar

Merging missions into the graph builder stripped three things from live agents at once: tool permissions fell back to read-only, node timeouts collapsed to the default, and input presets disappeared unless someone reset them on every node.

Nobody was alerted. A teammate noticed her own agents were broken, and finding out which others had gone down meant opening Grafana and auditing by hand.

We repaired the broken agents two ways: manually to unblock pilots, and structurally with engineering. That work surfaced a permissions bug and made the risk concrete: configuration that exists only in the right place can disappear in one merge.

I had named bulletproof scheduling as a requirement at PI planning eighteen days earlier. The merge still slipped through planning I co-owned. The regression did not teach me the requirement. It proved it.

Agent Control shipped lean: the library, stopping a run from chat, and states grouped into an action hub. Stopping a scheduled run from the management surface was in the design but not in the release. In July, an agent got scheduled dozens of times during a demo and, combined with a partition-pruning bug, took the environment down. No surface gathered the flood up or offered a stop.

By late July, production ran 341 agent runs in a 24-hour window across 74 active graphs, at 86.8 percent against the 90 percent gate set in May. The miss is published on purpose.

The miss is published on purpose. The gate is the honest comparison, set in May before any new scope.

What I’d Defend

Most designers working on AI are styling the conversation. This work was the layers underneath: what an agent is allowed to do, how a person supervises it, what earns it the right to act alone, and how it reports back.

The order of operations is the thing I would defend hardest: vocabulary before screens, execution before trust, supervised runs before autonomy.

What I would change is smaller and more specific. I handwrote deduplication rules into individual agent graphs because the platform could not show where a run stopped or why. That belongs in the platform, not in every graph. Builder v2 moves that work to the right altitude.