Skip to content

feat(opencode): add browser tool for visual UI verification - #48755

Open
maskjelly wants to merge 2 commits into
anomalyco:devfrom
maskjelly:browser-tool
Open

maskjelly wants to merge 2 commits into
anomalyco:devfrom
maskjelly:browser-tool

Conversation

@maskjelly

Copy link
Copy Markdown

Issue for this PR

Closes #48377
Related: #40782

Supersedes #48505, which was closed by the compliance bot before the description was updated to match the template. Same branch and commits.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Adds a built-in browser tool so the agent can see and interact with web pages.

  • Actions: open, screenshot, click (selector or coordinates), type, press, scroll, read, back, forward, reload, close.
  • Screenshots come back as image attachments, so the model can actually see the page.
  • Drives a local Chrome / Chromium / Edge over CDP with a small WebSocket client launched via Bun.spawn — no new dependencies.
  • Discovers browsers on macOS / Linux / Windows; OPENCODE_BROWSER_PATH overrides.
  • Temporary profile, idle shutdown after 10 minutes, cleanup on exit.
  • Registered as a regular built-in tool, behind a new browser permission key; docs/tools.mdx and docs/permissions.mdx updated.

Scoped to the browser; OS-level desktop control is out of scope for this PR. Happy to adjust the tool surface (naming, action granularity, gating) based on maintainer feedback.

How did you verify your code works?

  • bun typecheck in packages/opencode — clean.
  • bun test test/tool — 350 pass, including new tests for URL normalization, key parsing, and platform browser discovery.
  • Integration test opens a real page in headless Chrome, captures a screenshot attachment, reads element text, clicks a link, and closes; it skips automatically when no Chrome-family browser is installed.

Screenshots / recordings

N/A — no UI changes. The tool returns page screenshots to the model as attachments.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

Add a built-in browser tool that drives headless Chrome, Chromium, or Edge
over the Chrome DevTools Protocol. It opens pages, returns screenshots as
image attachments, and supports clicking, typing, pressing keys, scrolling,
and reading page text so the agent can verify front-end work end to end.

- no new dependencies: raw CDP over a WebSocket connection
- cross-platform browser discovery with OPENCODE_BROWSER_PATH override
- permission key "browser" with per-action patterns
- idle shutdown plus process exit cleanup
- unit tests and a Chrome-gated integration test
Run the browser tool with a visible window when a display is available so
the user can watch the agent work. Pass `headless: true` or set
`OPENCODE_BROWSER_HEADLESS=1` to hide it, and fall back to headless on
Linux hosts without a display.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE]:Screen vision (screenshots) + browser control tools for the agent

1 participant