How to Teach AI Agents to Test Mobile Apps on Their Own

Familiar story: you ask an AI assistant like Cursor or Claude to write a feature for a mobile app, it spits out a bunch of code, you copy it, and then... you open the simulator and manually click through every screen to check if the layout broke. At that moment, it feels like automation took a wrong turn somewhere. AI can write code, but it's "blind" to the real interface outside the text editor.

The folks at Callstack solved this problem by releasing agent-device. It's a CLI tool that transforms an AI agent from a theorist into a practitioner. Now the agent can open the app on an iOS simulator or Android emulator, look at screen elements, and tap the right buttons.

Why bother when you have Appium or Maestro

It might seem like there are plenty of mobile automation tools already. But here's the catch: Appium and Maestro were designed for humans. An AI agent doesn't need to write complex YAML scenarios or mess with selectors in the XML tree. It needs a fast, cheap, and straightforward way to interact with the hardware.

agent-device works differently. It captures screenshots not just as images, but as structured accessibility trees. Rather than feeding heavy screenshots to a multimodal model and burning through tokens, the agent gets a text-based description of elements with short references like @e1, @e2. This cuts costs and speeds things up.

What this tool can do

The tool positions itself as the agent's "hands and eyes." Here are the main features that caught my attention:

1. Smart screenshots. The snapshot -i command returns only interactive elements. The agent sees a list: @e1 [button] "Sign In", @e2 [text-field] "Email". No guessing where to tap.
2. Cross-platform out of the box. The same workflow works for iOS, Android, TV (tvOS and Android TV), and even desktop apps on macOS and Linux.
3. Evidence gathering. If something goes wrong, the agent can start video recording, capture logs, or extract network traffic on its own. This is invaluable for debugging bugs that only reproduce at runtime.
4. React Native integration. Since Callstack is behind this project, there's deep RN support: you can inspect the component tree and profile renders.

How it looks in practice

Imagine you're setting up an MCP server (Model Context Protocol) for your AI agent. Now it can execute commands directly in the terminal.

First, we check the environment:

agent-device doctor

If everything checks out, the agent can launch the app:

agent-device open "MyApp" --platform ios

Then it takes a look around:

agent-device snapshot -i

Once it has the list of elements, it simply simulates user actions:

agent-device fill @e3 "[email protected]"
agent-device tap @e2

That's it. No waiting for compilation or manually switching windows.

Under the hood

The tool doesn't try to reinvent the wheel where standards already work. For iOS it uses XCTest, for Android it's ADB combined with a custom screenshot helper. For the web, Playwright runs under the hood (specifically logic from vercel/agent-browser).

Interestingly, agent-device can convert its sessions to Maestro format. This means the agent can draft a test while "exploring" the app, and you can then save it as a full E2E test for CI.

Who should try this

I see several scenarios where this will genuinely save time:

• React Native and Expo developers. If you're using Cursor or Windsurf, add the agent-device documentation to the context. The agent can verify its own changes without interrupting you.
• QA engineers. You can delegate writing basic smoke tests to AI. It will find the buttons itself and verify that transitions work.
• Teams working with TV platforms. TV automation is always a pain, and here it comes bundled.

The project is actively developing, and while the documentation is still being filled in places, the main CLI works stably. If you believe in the Agentic Workflows concept, this tool is definitely worth spending an evening on.

You can start with their official documentation, which details how to integrate the CLI with popular AI agents.