Piloting Scout with real customer data

At a glance

Scout, Sitetracker’s agentic AI platform, demoed well: read a document, write its data into Salesforce, pause for human approval. Sales sold do-anything, and two enterprise pilots were signed before the pilot agents existed.

Team: me as designer, with a pilot lead, agents and platform engineering, a product quality lead, later a second designer. Timeline: Feb to Jul 2026.

• One customer asked to extend the pilot to October.

• I built four agents: lease abstraction, risk extraction, permit conditions, meeting intelligence.

• A lease took the customer about an hour by hand; the agent drafts the records in minutes. Their baseline, my projection.

• 46 duplicate records in the customer’s sandbox. I got a duplicate-refusing constraint onto all seven objects the graph writes, not just the four that broke.

"95% of the execution and making it successful."

— the pilot lead, on the pilot I owned end to end

I built the agents, not the specs for the agents.

The graph builder existed and did not get me to a working agent. Our senior director of product had stood up an MCP server as a workaround, and my manager and I both built agents through it. I connected Claude Code to it, built these four, and tested each version inside Scout against the renewable-energy customer’s real leases and a full copy of their Salesforce org.

The build loop: knowing which half to automate is the design skill.

Published v13, routed by page count.

The design decisions live in the graph: pages read as images, because scanned, stamped, hand-annotated Italian leases defeat text parsing; vision at 150 dpi, low detail, capped at 20 pages, because I paid for vision and then made it cheap. I explored a text path for born-digital PDFs and cut it: not worth a second code path.

One lease fans out across eight Salesforce objects: seven the graph writes, one it only reads. The approval gate fails closed, anything but an explicit yes writes nothing, and missing fields go in null rather than invented. Ambiguity must never resolve toward action.

A teammate’s finance graph at the same gate: every record editable before the write commits.

A broken join and a missing report look identical to an agent.

Testing at the field-ops customer ran on live crew data. The agent reported a crew member had logged zero work. Wrong: one Salesforce object stored the person as an ID, the other as a name; it matched nothing and reported zero. No error anywhere.

The realization: an agent’s most dangerous failure isn’t the crash. It’s the confident empty result. My fix: a prompt step that resolves IDs to names before matching.

Field failures became product tickets.

Shipped:

• The approval card never appeared, so the customer missed approvals. Fixed in May: approval prompts rise above the fold.

• Executions died silently mid-run. Filed with the execution ID and step; fixed in 72 hours.

Specced, not shipped: agent-run failure states, safe re-runs, no premature error states.

The failure-state work is a prototype you can open: scout-agent-control-v5-mf.surge.sh/agents (fictional data). Building it real surfaced what a mock couldn’t: at narrow widths the runs table hid the controls it exists to show

Failed and needs-input runs triage first, each stating why.

The shipped agent page, failures and all: a run history that hides them cannot be triaged.

A pilot that only runs when the designer is on the call isn't a pilot.

The team’s own review: we had built forty-node pilot graphs before validating the basics. So I defined what ships standard — graph shape, fail-closed gate, duplicate-refusing writes — and what gets rebuilt per customer.

On engineering’s eval infrastructure I authored a 16-probe suite scoring whether answers are grounded and correct, ran it, tuned the knowledge graph and system prompt, then re-ran the identical suite: the agent changed, not the test. Another run published a number nobody liked: precision 0.48 on one permit, over-extraction rather than invention, and that number told the team which fix to build.

Same 16 probes: 9 of 16 (56%) before tuning, 14 of 16 (88%) after.

With my tech leads I ran weekly office hours for the sales engineers demoing Scout, so the people selling agents could build their own — and so I could watch people who were not me hit the parts that break. A teammate built the Finance Central graph; my probes on its data went into the quality lead’s customer demo, where one July run proposed 48 purchase-order updates, all held at the gate.

What nobody was measuring.

Run success read 82.4% on May 4 across a month of production traces, 86.8% on July 28 across a single platform-wide day. A month against a day, so I will not draw a line through them. No run-volume number ever came from the customer, so per-run saving is the only claim I make. The number I wanted, no dashboard could show: a run a customer started without us.

Chat adoption, instrumented after the fact. I was the heaviest user; the agent panel showed nothing.

I argued for this one and lost. I wanted agent health inside our admin app, where a non-engineer would already be, and I prototyped it there in the app’s own codebase. Engineering descoped it: the admin app runs on the same API the agents do, so it goes blind exactly when you need it. They were right. I learned Grafana, and a tech lead helped me build a view a non-engineer could read. Neither dashboard closed the gap. Slack alerts on failed runs did.

The instrument I lost the argument to, and then learned: 341 runs in 24 hours, platform-wide.

What shipped in chat: a generic apology over a parser trace, the real reason one disclosure away.