Many agent tasks end at the boundary of a screen or device that offers no useful API. The Agent Interface Experiments are a collection of attempts to cross that boundary without pretending that general computer control is automatically safe or reliable.
The project takes the form of a collection of touch, window, camera, e-reader, and device-control experiments that give agents bounded ways to inspect and operate otherwise inaccessible interfaces. Its purpose is to explore practical bridges between agent reasoning and the physical or graphical surfaces where real work happens.
The question behind Agent Interface Experiments
The experiments include touch, window navigation, Kindle and camera workflows, and device-specific control. What unifies them is not one product interface, but a recurring question: what is the smallest bridge that lets an agent complete a blocked real-world task? Developers, accessibility researchers, automation users, and agent designers may benefit. Device owners and anyone visible on a screen or camera are affected.
How Agent Interface Experiments took shape
Each utility exposes a bounded surface—capture a view, send a controlled input, stream a phone camera, or wrap a device interaction—so the agent can use it through an explicit capability rather than improvising unrestricted control. Josiah identified blocked real-world workflows, designed the bridges and safety boundaries, and repeatedly directed agents to turn them into reusable capabilities.
What the evidence supports
Several individual utilities have worked for specific tasks, though the family has not been packaged as one product. The family suggests that the future of agent interfaces may be many narrow adapters rather than one omnipotent cursor. Scope, user visibility, consent, reversible actions, and inspection matter more as the agent moves closer to physical consequences.
Computer control is brittle and potentially destructive. Consent, scope, visual privacy, and reversible actions are essential. The experiments can mature into a public taxonomy of interface gaps, bounded control patterns, and utilities demonstrated in real workflows.