Skip to content

Agent browser: mpx bridge, mpx browser, and the multiplex skill - #2

Draft
jhen0409 wants to merge 2 commits into
mainfrom
agent-browser
Draft

jhen0409 wants to merge 2 commits into
mainfrom
agent-browser

Conversation

@jhen0409

@jhen0409 jhen0409 commented Oct 8, 2026

Copy link
Copy Markdown
Member

Summary

Lets an agent on this machine open and operate web pages in the Multiplex app on the user's device (Vision Pro / iPad / iPhone). Pairs with the app-side PR in multiplex-term/Multiplex (branch agent-browser).

  • mpx bridge (hidden) — the app starts it over its own SSH connection (a PTY-less login shell, so the host's PATH applies). It serves a unix socket ~/.mpx/run/browser-<device>-<install id>.sock; no TCP port, no sshd forwarding.
    • The directory is 0700. Every ancestor must be owned by you or root, and any other-writable ancestor must be root-owned and sticky. Both ends check the peer's uid.
    • Framing is bounded JSON lines, with at most 16 clients.
    • Requests carry t/ttl and a cancel is sent on timeout, so an app that was suspended never replays stale clicks.
    • The socket is removed on EOF or on SIGTERM/SIGINT/SIGHUP.
  • mpx browser … — open, snapshot (outline with [ref=eN]), click, fill, type, press, select, scroll, wait, eval, console, screenshot, show, close, tabs, status, forward <port> [--open].
    • Screenshots are written 0600 under ~/.mpx/shots.
    • --device / MPX_BROWSER_DEVICE picks among connected devices; an ambiguous match is an error. Without it, the newest device wins and stderr says so.
    • In tmux, the agent's session is sent along so the tab docks beside it.
  • Skill + Claude Code plugin — mpx skill install [--agent claude|codex] / mpx skill print, and /plugin marketplace add multiplex-term/mpx-cli → /plugin install multiplex@multiplex-term. One plugin/skills/multiplex/SKILL.md serves both; a test pins plugin.json's version to the crate's.

Release note: the app copy says "mpx 0.2 or later", but Cargo.toml is still 0.1.2. Bump it, together with plugin/.claude-plugin/plugin.json, at release.

Testing

  • cargo test: 47 unit tests and 3 integration tests (tests/browser_bridge.rs: a request round-trip through a real bridge process, the actionable no-device error, same-named devices on separate sockets, SIGTERM socket cleanup).
  • cargo clippy --all-targets is clean.
  • claude plugin validate passes for the marketplace and for the plugin (--strict); installed into a throwaway CLAUDE_CONFIG_DIR.
  • End to end against the app on an iPhone Air and the visionOS/iPad simulators. Details are in the app PR.

Lets an agent on this machine drive web pages in the Multiplex app on the
user's device.

- `mpx bridge` (hidden): started by the app over its own SSH connection;
  serves `~/.mpx/run/browser-<device>-<install id>.sock` (0700 dir,
  owner/sticky ancestry checked, peer uid checked on both ends), JSON
  lines both ways, request `t`/`ttl` stamps and `cancel` for the app's
  stale-request guard, socket removed on EOF or signal.
- `mpx browser …`: open, snapshot (outline with [ref=eN]), click, fill,
  type, press, select, scroll, wait, eval, console, screenshot (0600 in
  ~/.mpx/shots), show, close, tabs, status, forward [--open]. `--device`
  picks among several connected devices; an ambiguous match is an error.
- `mpx skill install|print` and a Claude Code plugin
  (`.claude-plugin/marketplace.json` → `plugin/`), one SKILL.md for both;
  a test pins the plugin version to the crate's.
CI's newer clippy flags unnecessary_sort_by (live_bridges) and chunks_exact_to_as_chunks (hex_decode, pre-existing on main).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant