Browser Use 2.0

[hannesrudolph]

Summary
This upgrades the in‑chat browsing experience with persistent sessions, clearer feedback, a dedicated browser panel, and more natural action descriptions.

Extension.Development.Host.rc10.2025-11-05.13-47-15.mp4

What's new

• Persistent Browser Sessions
  • The browser stays open across steps so you can send follow‑ups without relaunching.
  • You’ll see a "Browser Session" header and a "Session started" note when active.
• Dedicated Browser Session panel
  • Open a full‑size view when you need more space, while keeping the chat context in view.
• Live, readable action feed
  • Actions are presented in plain language: Launch, Click, Type, Press, Hover, Scroll.
  • Keyboard events now appear as "Press Enter" or "Press Esc" for easier scanning.
  • Broader keyboard coverage: navigation keys and common shortcuts are supported for more natural control.
• Inline console logs
  • Console output is surfaced inline with a clear "No new logs" state.
  • Noise-reduced by default: only new entries since the previous step are shown to cut repeat noise.
  • Filter by type (Errors, Warnings, Logs) so you can focus on what matters.
• Clear session controls
  • A prominent Disconnect/Close control makes it easy to end a session when you’re done.
• Interactive in-session controls
  • Follow-ups attach to the active session so you can guide the assistant mid-flow without restarting.
  • Suggested follow-ups appear inline to keep momentum.
• More accurate interactions
  • Improved click, scroll, and hover reliability across screen sizes with a consistent preview aspect ratio.
• Seamless follow‑ups
  • Keep chatting while the session is open; the assistant continues from the same context.
• Fully localized
  • New labels and action text are translated across all supported languages.

What you'll notice in the UI

• "Browser Session" appears in chat when a session is active.
• A "Session started" status line confirms the start.
• Follow-up suggestions appear inside the Browser Session row when active.
• Keyboard actions are summarized clearly (e.g., "Press Tab", "Shift+Tab", "Arrow keys").
• New action wording like "Press Enter" or "Hover (x, y)".
• Console Logs are visible inline, with a "No new logs" indicator and a noise‑reduced view that shows only new entries since the last step.
• Type filters (All, Errors, Warnings, Logs) above the log list to quickly narrow the feed.
• A quick Disconnect button to end the session.

---

Important

Enhances in-chat browsing with persistent sessions, a dedicated browser panel, and improved UI feedback, including new components and tests.

• Behavior:
  • Introduces persistent browser sessions in presentAssistantMessage() in presentAssistantMessage.ts.
  • Adds a dedicated browser session panel in BrowserSessionPanelManager.ts.
  • Updates browserActionTool() in browserActionTool.ts to handle new action types and coordinate scaling.
• UI Components:
  • Adds BrowserSessionPanel and BrowserActionRow components for improved session management.
  • Updates ChatView.tsx and TaskHeader.tsx to integrate browser session status and controls.
  • Modifies ChatTextArea.tsx to include browser session toggle.
• Testing:
  • Adds tests for browser session UI elements in BrowserSessionRow.aspect-ratio.spec.tsx and BrowserSessionRow.disconnect-button.spec.tsx.
  • Updates ChatView.followup-in-session.spec.tsx to test follow-up behavior during sessions.
• Configuration:
  • Updates vite.config.ts to include new entry points for browser panel.

^{This description was created by for f430445. You can customize this summary. It will automatically update as commits are pushed.}

[roomote]

Rooviewer   See task on Roo Cloud

Reviewed commit f430445. Changes look good - no new issues found.

Issues to Address

• Race condition in browser session state callback - Fixed by deferring panel operations to the microtask queue and verifying session is still active before opening the panel.

• Missing size parameter in launch action - Fixed by including the viewport size in the launch action message.

• ResizeObserver cleanup missing mounted check - Fixed by adding a mounted flag to prevent state updates after component unmount.

• Coordinate format mismatch - Fixed by storing scaled viewport coordinates in the browser_action message.

Previous reviews

• 99069c0: Review #1
• 46578e0: Review #2
• d9df0df: Review #3
• 9311f10: Review #4
• 622f43a: Review #5

_{Mention @roomote to ask your PR Fixer agent to address the feedback.}

[Copilot]

Pull Request Overview

This PR adds a new Browser Session panel feature that provides a dedicated UI for viewing and controlling browser automation sessions. Key improvements include:

• New standalone browser session panel with navigation controls
• Enhanced coordinate scaling for accurate click/hover actions on downscaled screenshots
• New keyboard press action support
• Improved browser session lifecycle management
• Real-time browser session status tracking

Reviewed Changes

Copilot reviewed 54 out of 54 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
webview-ui/vite.config.ts	Adds browser-panel.html as a new build entry point
webview-ui/src/i18n/locales/*/chat.json	Adds translations for new browser session UI labels (session, press, hover actions)
webview-ui/src/components/chat/BrowserSessionRow.tsx	Major refactor: adds full browser-like UI with navigation, toolbar, and improved screenshot display
webview-ui/src/components/chat/BrowserActionRow.tsx	New component to display browser actions inline in chat with auto-panel-opening logic
webview-ui/src/components/browser-session/*	New components for standalone browser session panel
src/services/browser/BrowserSession.ts	Adds press() method, cursor visualization, viewport tracking, and state change callbacks
src/core/tools/browserActionTool.ts	Implements coordinate scaling from screenshot to viewport dimensions
src/core/webview/BrowserSessionPanelManager.ts	New manager for browser panel lifecycle and communication
src/shared/*Message.ts	Adds new message types for browser panel communication

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[roomote]

Review complete. I found 3 issues that should be addressed before approval. Please see the inline comments and checklist above.