The Interface I Give an Agent

A lot of back-office software has no API, only the screen a human operator uses. The obvious agent design hands a model the browser for every request, which pays for model calls on every run and asks a reviewer to trust a new trajectory each time.

I wanted to know whether a model's one-time exploration of an interface could become automation that never needs the model again. I built Computer Use Runtime to answer that concretely. One model exploration of a synthetic banking screen became a seven-step capability that replayed on new inputs and a reversed layout with zero model events. Most of the work was in the interfaces around the model: the tools it may call, the artifact it leaves, and the checks replay runs.

The screen as the only API

The target is a synthetic banking workspace built to be awkward in the ways legacy software is awkward. It runs inside an iframe, lays data out in tables, has no test IDs, and can inject slow responses, expired sessions, dialogs, and application errors. The task is a lookup: verify a member ID and return the savings balance and currency, checked against independent fixture values.

Four tools instead of a browser

The model never receives raw browser control. It explores through four application tools: "observe", "act", "request_handoff", and "complete". Observations are redacted accessibility snapshots, and the model can ask for a masked screenshot. Actions are typed operations, not generated JavaScript or shell commands.

The worker binds input values and validates the outputs it reads, so the model never sees the balances it is helping to extract. A policy file limits the reachable origins and paths, the action types, six allowed click names, and the time and step budgets. A transfer button stays blocked even if the model asks for it, and a live-browser test confirms the block lands before the page changes.

Refuse the bad action, keep the run

Both final discoveries, one normal and one with an injected note on the page, needed corrections. Each made nine action attempts: seven executed and two were rejected before compilation. Without a correction path, both runs would have ended on their third action.

The first rejection came from my own redaction: the model copied the placeholder [input:memberId] from its snapshot as row text, and no cell on the page contained it. The second was a click that left the application unchanged.

The recorded runs judged that click with a single comparison, and the next observation confirmed the page had not moved. A single comparison can misread a late update, so discovery now waits within the action timeout without clicking twice; four regression tests cover that wait, but no new model run has. An uncertain click counts as executed, so the model observes or requests a handoff instead of retrying.

Compile what worked

When the model calls "complete", the compiler keeps only the actions that executed. It requires typed output reads and a final checkpoint that compares the result against the bound inputs. The output is a capability with typed inputs and outputs, ordered actions, declared outcomes, and model and run provenance, validated against a Zod schema. Each capability gets an immutable SHA-256 revision that replay can pin and verify before it touches the page; the digest detects changed content but is not an authorization signature.

Positions break, meaning survives

Targets are stored as roles, labels, text, or table row and header pairs, not as positions. To test that choice, I added a second layout that reverses both the rows and the columns of every table. The same discovered revision replayed correctly on both layouts.

Proving there is no model in replay

Making replay model-free was the easy part. Proving it took more work. The replay module has no dependency on a model provider, so the call path cannot reach one. Every run also writes an event stream, and a reviewer can count the model and provider events.

The exact-revision replays used member IDs other than the ones discovery used, returned the expected outputs, and emitted zero model or provider events. Each discovery took about 36 to 41 seconds and eight model calls; each replay finished in 441 to 792 milliseconds.

Failures are outcomes

Replay classifies the page before and after each step: a missing member returns as an answer, a transient error gets one bounded reload, an expired session pauses for an operator in the same headed browser, and an application error stops with a masked screenshot. In the recorded handoffs, Playwright simulated the operator's clicks.

The riskiest dependency was the model runtime

The hardest component to contain was not the browser but the model runtime. Plugins in my own Codex home kept MCP servers, including a Node REPL, attached even with overrides that emptied mcp_servers and plugins.

Discovery now runs from an isolated Codex home with minimal configuration, and a preflight check refuses to spend a model turn if any inherited MCP server is still active. The recorded preflight shows no inherited MCP servers and no native environment access. My first discovery failed before any UI step because that isolation had also disabled the tool host the model needed; the failed run stays in the evidence index. The recorded runs used gpt-6-astra through Codex CLI 0.153.4, whose sandboxed host still adds orchestration utilities, so four is the count of application tools, not of every tool.

The injected-note discovery confirmed that the note reached the model and that the task still completed without an unregistered application call. That is one observed case, not a general guarantee.

What this does not show

The evidence covers one synthetic application. Nothing has been deployed, and there are no real accounts, bank writes, or tenant isolation. The metered API adapter shares the same tools but has only mocked protocol tests. The policy allows six click names and the prompt names the outputs and final checkpoint, so discovery searched a narrow space; these runs are not a benchmark and say nothing about one model against another.

The local gates pass 67 tests: 39 core and provider, 9 browser, 15 runtime, and 4 console. The fixture defines 12 scenarios; prompt injection is covered by the one live discovery run, not by the local suites.

How I choose the interface now

1. I use a real API whenever one exists; a vendor-maintained contract beats any screen-level design.
2. When the screen is the only way in, I give the model typed operations over redacted semantic snapshots, never raw browser or shell code.
3. I check what my redaction shows the model, because a placeholder can become a target.
4. I compile only actions that executed, and I end every capability with a checkpoint bound to its inputs.
5. I make the absence of the model checkable from the event log instead of claiming it.
6. I version capabilities by content hash, and when a checkpoint fails I would rediscover through review instead of letting a model repair selectors in production.

The model earns its cost once, during discovery. After that, the interface I want is the one the model left behind: typed, versioned, inspectable, and boring to run.