Failure discovery still depended on the people in the room

Two enterprise pilots were moving from demos into daily customer use, but failure discovery was still a support process. Customers reported what broke, engineers audited Grafana, and I found problems by re-running agents.

Across both pilots, I built nine agents. I edited graphs and prompts and re-ran until the outputs held consistently.

The builder UI didn’t get me there, so our senior director of product spun up a Scout MCP server as a workaround. Claude Code wrote each version of the graph, the MCP server published it into Scout, and Scout ran it against the pilot’s documents and sandbox records.

The agent turns a lease PDF into linked Salesforce records in minutes

One lease abstraction took the customer about an hour by hand. The agent drafts the linked project, parcel, and agreement records in minutes.

I made the approval gate fail closed. When a field is missing, the agent writes null instead of inventing a value. Ambiguity must never resolve toward action.

Published v13, mid-edit: chunking long leases into page-count batches. Clauses ignore page breaks.

I caught the failure that looks exactly like a correct answer

The second pilot was a field-ops customer, and testing there ran on live crew data. The agent reported that a crew member had logged zero work. Wrong: one Salesforce object stored the person as an ID, the other as a name; it matched nothing and reported zero. No error anywhere. My fix was a prompt step that resolves IDs to names before matching.

I moved the raw payload out of the user's message

Failures came to me first. Twelve platform issues in two days in March 2026, then a wave within a week of the beta users being onboarded in June. I triaged what came in.

A failure state has two readers: the person watching the run, and the technical person they hand it to. The agent is not a reader. When a run fails, the platform tells the next turn that something broke but not what, so a stray error string cannot steer the model. The message says what broke in plain language, and the raw output sits one click down, intact.

Customers ran the agents without me

This is replacing the snapshots. It will be requested every day. — renewable energy customer

Once customers were running the agents daily on their own, failures stopped coming to me first, and I had no customer-facing view of what was failing.

Monitoring can't run on the API it monitors

Grafana existed, but only engineers used it. So I mocked up an agent health dashboard in our admin app and argued for it. I lost, and the objection was right. That app runs on the same API the agents do, so it goes blind exactly when you need it. I took the requirements into Grafana instead, and that became how I caught regressions when I was not in the room.

The dashboard I got built after losing that argument. 86.8% across 341 runs in a day: the org-wide number, not my pilots’.

Fixes that needed a release became configuration

I drew the line between what I could change in a live customer org and what needed a release: company skills, keyword mappings, prompts, and agent settings were mine; runtime defects and schema were engineering's.

On engineering's eval infrastructure, I authored a 16-probe answer-correctness suite, tuned the knowledge graph and system prompt, and re-ran the identical suite. Passes moved from 9 of 16 before tuning to 14 of 16 after. The agent changed, not the test.

One run published a number nobody liked: precision 0.48 on one permit, from over-extraction rather than invention. It told the team to collapse nested conditions, gate non-obligation clauses, and dedupe by meaning.

Outcome

The pilot was extended into October, requested and granted.

Passes on the identical 16-probe suite moved from 9 of 16 to 14 of 16.

Twelve department heads and C-suite leaders onboarded during my time on the pilot.

Error states, fail-closed writes, and the eval groundwork shipped standard.