How to Wire an iPhone into a Hermes Agent with an MCP Bridge

I run a self-hosted Hermes agent runtime on my own hardware, and last week I gave it a physical iPhone to operate. Not a simulator. A real device on my desk, tapped and swiped by the same agent that already writes my code and drafts my emails.

Getting there took a small proxy server I call ios-mcp-bridge. Worth saying up front: this isn't a fully turnkey tutorial, because the iPhone side depends on a jailbroken device running its own MCP server, and that on-device piece isn't something everyone can drop in. What I can walk through is the pattern: how the bridge is built, how it plugs into an agent config, and what it actually exposes once it's wired up. If you're building anything similar, the shape of this should save you some dead ends.

If you haven't set up Hermes itself yet, my prerequisite guide covers installing it and minting the OAuth token calendar access needs. This post assumes that part's already done.

The problem: your agent needs a real device, not a simulator

The connective tissue for all of this is MCP, the Model Context Protocol. It's just a JSON-RPC contract for tools, so an agent can call anything that speaks it: a database, an API, or in this case, a phone.

Headless automation tools get detected. Anything running through XCTest, WebDriverAgent, or a virtual display leaves fingerprints that apps increasingly check for, especially on platforms that care about bot traffic. If you want an agent to genuinely operate a phone (tap through a UI, read a screen, launch an app) you want something closer to real touch input on real hardware.

That's a two-piece problem:

1. Something running on the device that can inject touch events and read the screen.
2. Something your agent can actually talk to: an MCP server it can add to its tool list.

The on-device half is jailbreak territory: real UIKit-level touch injection instead of an accessibility API. I'm not walking through that build here, both because it's device- and jailbreak-version-specific, and because most readers won't have a rooted device to run it on. What I want to show is the second half, because that part is portable: a stateless MCP proxy any agent runtime can add as a tool source.

The bridge: a stateless MCP-to-HTTP proxy

ios-mcp-bridge doesn't do any device automation itself. It's a stdio MCP server that forwards every tools/list and tools/call request to whatever's actually running on the device, over plain HTTP. The whole thing is under 40 lines:

import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";

const IOS_MCP_URL = process.env.IOS_MCP_URL || "http://<device-ip>:8090/mcp";
const PROTOCOL_VERSION = process.env.MCP_PROTOCOL_VERSION || "2025-11-25";

async function callRemote(method: string, params?: Record<string, unknown>) {
  const res = await fetch(IOS_MCP_URL, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "MCP-Protocol-Version": PROTOCOL_VERSION,
    },
    body: JSON.stringify({ jsonrpc: "2.0", method, params, id: 0 }),
  });

  if (!res.ok) throw new Error(`ios-mcp ${res.status}: ${await res.text()}`);
  const body = await res.json();
  if (body.error) throw new Error(`ios-mcp error: ${JSON.stringify(body.error)}`);
  return body.result;
}

const server = new Server(
  { name: "ios-mcp-bridge", version: "0.1.0" },
  { capabilities: { tools: {} } }
);

server.setRequestHandler(ListToolsRequestSchema, async () => callRemote("tools/list"));
server.setRequestHandler(CallToolRequestSchema, async (req) =>
  callRemote("tools/call", { name: req.params.name, arguments: req.params.arguments })
);

await server.connect(new StdioServerTransport());

That's the entire pattern worth stealing: your agent runtime spawns this as a normal stdio MCP server (same as any local tool), and the bridge relays every call to the device's own MCP server over the network. The agent doesn't know or care that the "tool" it's calling lives on a phone three rooms away.

Building and wiring it in

Standard MCP server build:

npm install
node build.mjs   # esbuild bundle, dist/index.js

Bundling with esbuild rather than a plain tsc transpile matters if you ever deploy this into a container or a remote host that doesn't ship node_modules alongside it. A tsc build will crash with ERR_MODULE_NOT_FOUND the moment it's copied somewhere without the dependency tree intact.

Then point IOS_MCP_URL at wherever the on-device server is actually listening, and register it like any other MCP server in your agent's config:

mcp_servers:
  ios-bridge:
    command: node
    args: ["/path/to/ios-mcp-bridge/dist/index.js"]
    env:
      IOS_MCP_URL: "http://<device-ip>:8090/mcp"

Once that's live, the agent's tool list picks up whatever the on-device server exposes. There's no bridge-side schema to maintain, because it's just forwarding tools/list verbatim.

What ends up in the tool list

On my setup, the on-device server exposes real UIKit-level control: tap, swipe, scroll, drag-and-drop, screenshot, launch/kill app, read the frontmost app, OCR a screen, read/write files on the device, get device info. All of it goes through actual touch injection, not a simulator or an accessibility bridge. The agent taps and swipes the phone the way I would, not scripting around it from the outside.

The practical lesson, if you're building your own version of this: keep the bridge dumb. The moment you start putting device logic in the proxy layer, you've coupled two things that should stay independent. The proxy's only job is transport. Let the on-device server own everything about what "tap" or "screenshot" actually means, and the bridge stays a 40-line file you never have to touch again.

What a dumb proxy doesn't solve for you

A stateless bridge means every reliability problem lands on the calling agent, not the transport layer. Three came up enough on my setup that I built them into how I drive the tool, not into the bridge itself.

Screenshots lag under load. A screenshot can arrive a step behind the live UI, especially with a heavy page rendering. Tap off the image and you'll sometimes hit whatever actually occupies that spot now, not what you saw. The fix is treating screenshots as verification, not targeting: pull live element geometry to decide where to tap, then screenshot afterward to confirm the tap landed.

Pixel coordinates and tap coordinates aren't the same scale. Screenshots come back at device resolution (2x on most modern iPhones), but tap coordinates are specified in logical points. Read a coordinate straight off a screenshot without halving it and the tap goes somewhere else entirely.

There's no built-in wait. The tool surface has taps, swipes, and reads, but no primitive for "pause until this loads." Pacing for page loads and animations is the agent's job, not the bridge's.

None of this is a knock on the pattern. It's the tradeoff you're signing up for: a thin, honest proxy versus a fatter server that tries to guess what you meant. I'd take the thin one again.

If you're building agent tooling generally, the Claude Code MCP docs are a solid reference for how a client-side agent actually consumes servers like this one.

What would you want an agent to be able to do on a physical device that it can't do today?