Skip to content

feat(opencode): add browser tool for visual UI verification - #48505

Closed
maskjelly wants to merge 2 commits into
anomalyco:devfrom
maskjelly:browser-tool
Closed

maskjelly wants to merge 2 commits into
anomalyco:devfrom
maskjelly:browser-tool

Conversation

@maskjelly

@maskjelly maskjelly commented Sep 11, 2026

Copy link
Copy Markdown

Issue for this PR

Closes #48377
Related: #40782

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Adds a built-in browser tool so the agent can see and interact with web pages.

  • Actions: open, screenshot, click (selector or coordinates), type, press, scroll, read, back, forward, reload, close.
  • Screenshots come back as image attachments, so the model can actually see the page.
  • Drives a local Chrome / Chromium / Edge over CDP with a small WebSocket client launched via Bun.spawn — no new dependencies.
  • Discovers browsers on macOS / Linux / Windows; OPENCODE_BROWSER_PATH overrides.
  • Temporary profile, idle shutdown after 10 minutes, cleanup on exit.
  • Registered as a regular built-in tool, behind a new browser permission key; docs/tools.mdx and docs/permissions.mdx updated.

Scoped to the browser; OS-level desktop control is out of scope for this PR. Happy to adjust the tool surface (naming, action granularity, gating) based on maintainer feedback.

How did you verify your code works?

  • bun typecheck in packages/opencode — clean.
  • bun test test/tool — 350 pass, including new tests for URL normalization, key parsing, and platform browser discovery.
  • Integration test opens a real page in headless Chrome, captures a screenshot attachment, reads element text, clicks a link, and closes; it skips automatically when no Chrome-family browser is installed.

Screenshots / recordings

N/A — no UI changes. The tool returns page screenshots to the model as attachments.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

Add a built-in browser tool that drives headless Chrome, Chromium, or Edge
over the Chrome DevTools Protocol. It opens pages, returns screenshots as
image attachments, and supports clicking, typing, pressing keys, scrolling,
and reading page text so the agent can verify front-end work end to end.

- no new dependencies: raw CDP over a WebSocket connection
- cross-platform browser discovery with OPENCODE_BROWSER_PATH override
- permission key "browser" with per-action patterns
- idle shutdown plus process exit cleanup
- unit tests and a Chrome-gated integration test
@github-actions github-actions Bot added the needs:compliance This means the issue will auto-close after 2 hours. label Sep 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR doesn't fully meet our contributing guidelines and PR template.

What needs to be fixed:

  • PR description is missing required template sections. Please use the PR template.

Please edit this PR description to address the above within 2 hours, or it will be automatically closed.

If you believe this was flagged incorrectly, please let a maintainer know.

@github-actions

Copy link
Copy Markdown
Contributor

The following comment was made by an LLM, it may be inaccurate:

I found one potentially related PR:

PR #7302 — "Added in-built browser tools using playwright and a parallel playwright node process using spawn for bun-playwright issues"
#7302

This PR appears to be an earlier attempt at adding browser automation capabilities to the agent, but uses Playwright instead of Chrome DevTools Protocol. This may have been closed or superseded, but it's worth noting as related work in the same feature area.

The current PR (#48505) takes a different implementation approach (CDP-based) and adds more comprehensive features like cross-platform browser discovery and proper screenshot integration.

Run the browser tool with a visible window when a display is available so
the user can watch the agent work. Pass `headless: true` or set
`OPENCODE_BROWSER_HEADLESS=1` to hide it, and fall back to headless on
Linux hosts without a display.
@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been automatically closed because it was not updated to meet our contributing guidelines within the 2-hour window.

Feel free to open a new pull request that follows our guidelines.

@github-actions github-actions Bot removed the needs:compliance This means the issue will auto-close after 2 hours. label Sep 11, 2026
@github-actions github-actions Bot closed this Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE]:Screen vision (screenshots) + browser control tools for the agent

1 participant