Skip to content

browser-computer-use-security.md

Latest commit

 

History

History
250 lines (169 loc) · 9.63 KB

browser-computer-use-security.md

File metadata and controls

250 lines (169 loc) · 9.63 KB
Raw
title Browser & Computer-Use Security
slug browser-computer-use-security
category ai
depth 2
audit_level
2
3
4
last_reviewed 2026-04-21
sources
OpenAI Operator System Card — https://openai.com/safety
OpenAI Computer-Using Agent / Preparedness Framework — https://openai.com/safety
Anthropic Claude 4 System Cards + Responsible Scaling Policy — https://www.anthropic.com/research (2025-2026)
Google AI security frontier strategy
USENIX Security / IEEE S&P agent misalignment & tool poisoning papers (2024-2026)
triggers_strong
browser use
computer use
web agent
operator
GUI agent
desktop agent
triggers_weak
web automation
agent browser
screen control
related
llm-agent-security
ai-cli-hardening
ai-agent-incident-response
agent-evals-red-teaming

Browser & Computer-Use Security

Last reviewed: 2026-04-18 | Next review: 2026-10-18 | Priority: Recommended | Audit Level: 2-4 | Automation: Partial (sandboxing, domain allowlists, and confirmation hooks automatable; task scoping, sensitive-action review, and failure adjudication manual)

Use this guide when an AI system can interact with a browser, desktop, or GUI through screenshots, mouse/keyboard actions, or a general computer-use loop.

This surface is riskier than plain tool calling because the agent can encounter third-party content, follow multi-step UI flows, and sometimes operate in environments where a prompt injection can lead to real side effects on the local OS or in authenticated web sessions.

1. Core risk model

Browser/computer-use systems combine three hazards:

  1. They read untrusted visual or web content.
  2. They plan across multiple steps.
  3. They can trigger external side effects with general-purpose actions.

This means a single failure can move from:

  • prompt injection,
  • to goal drift,
  • to unsafe click/type behavior,
  • to real-world impact such as data leakage, destructive edits, fraud, or credential misuse.

OpenAI explicitly notes that for API-based computer use, prompt-injection impact is higher in non-browser environments and that the model currently performs best in browser-sandboxed contexts. Anthropic likewise requires a sandboxed computing environment for computer use and recommends extra caution when logins are involved.

2. What makes this surface different

Surface Typical problem Why it is different
Browser-only agent Hidden instructions in websites, emails, docs, or forms The attacker controls visible and invisible page content
Desktop / full computer use File-system side effects, uploads/downloads, clipboard abuse A successful prompt injection may spill beyond the browser
Authenticated sessions Account actions happen in a trusted user context The agent inherits the blast radius of the signed-in user
Multi-tool loop Browser + bash + file tools amplify each other One compromised step can pivot into a broader execution chain

If the agent can browse, download, upload, copy, paste, type credentials, or submit forms, treat it as a higher-risk workflow.

3. Main failure modes

Indirect prompt injection from the interface

Examples:

  • malicious instructions hidden in page text, HTML comments, PDFs, or emails,
  • UI hints that attempt to override the user goal,
  • documents that tell the agent to navigate elsewhere or reveal credentials.

Google says Project Mariner is being trained to prioritize user instructions over third-party prompt injections hidden in websites, documents, or emails. That is a useful model goal, but not a sufficient production control on its own.

Over-broad autonomy

Examples:

  • clicking through a purchase, approval, delete, or publish flow with no human boundary,
  • navigating away from the intended site,
  • retrying unsafe actions until something "works."

Local OS spillover

Examples:

  • opening or modifying local files that were never in scope,
  • pasting secrets into the wrong field,
  • uploading sensitive files because the agent misread the UI,
  • following a prompt injection into a local shell or file tool.

Weak observability

Examples:

  • no record of which page content changed the plan,
  • no screenshot or trace tied to the action sequence,
  • no operator-visible stop control for a runaway session.

4. Minimum architecture controls

Sandbox the environment

Start here:

  • run browser/computer-use inside a VM or container,
  • use an isolated browser profile,
  • mount only the minimum working directory,
  • keep the environment disposable and resettable,
  • separate this environment from the operator's real desktop.

Anthropic's docs explicitly say computer use requires a sandboxed computing environment, and their reference implementation runs inside Docker. OpenAI likewise recommends isolated/containerized setups for CUA usage in the API.

Narrow the action surface

  • Prefer browser-only over full desktop when possible.
  • Disable clipboard, shared folders, and file sync if not required.
  • Keep uploads/downloads off by default.
  • Do not combine browser use with shell or file-write tools unless the task clearly requires it.

Constrain navigation

  • Use domain allowlists for destinations.
  • Keep the agent on the expected website or app unless the operator explicitly broadens scope.
  • Block access to admin panels, billing, credentials, or identity settings unless the task is explicitly about them.

Require confirmation for sensitive actions

Examples:

  • login submission,
  • payment or purchase confirmation,
  • external message send,
  • account changes,
  • publish/deploy/delete flows,
  • file upload from local disk.

Google's current agent-security principles are a good baseline here: well-defined human controllers, carefully limited powers, and observable actions/planning.

5. Credentials and authenticated sessions

Do not treat a signed-in session as a convenience feature. It is a privilege escalation.

Rules:

  • Prefer pre-authenticated low-privilege test accounts instead of production accounts.
  • Never let the agent discover credentials by browsing around.
  • Avoid giving reusable secrets directly to the model when a broker or injected session can be used instead.
  • Scope accounts to the narrowest role needed for the task.
  • Reset the environment after credential entry or after the task completes.

Anthropic explicitly warns that using computer use in applications that require login increases the risk of bad outcomes from prompt injection.

6. Downloads, uploads, and local files

Treat every file transfer as a trust-boundary crossing.

For downloads:

  • quarantine files before reuse,
  • scan and classify before handing them to other tools,
  • do not auto-open downloaded content in a more privileged workflow.

For uploads:

  • require explicit operator approval,
  • expose the exact file path and destination before upload,
  • block secret-bearing or out-of-scope paths by policy.

For local files:

  • keep the mounted workspace narrow,
  • avoid home-directory or desktop-wide access,
  • log every file open, read, write, upload, and delete action.

7. Session hygiene

  • Use ephemeral sessions when possible.
  • Clear cookies, history, and tokens between unrelated tasks.
  • Do not reuse a session across users, cases, or trust tiers.
  • Limit how long a browser or desktop session can remain active without operator supervision.

Long-lived sessions quietly increase the chance that an indirect prompt injection or misclick becomes a real incident.

8. Logging and evidence

Keep enough telemetry to reconstruct:

  • user goal,
  • system/task scope,
  • screenshots or page state transitions,
  • tool or action sequence,
  • confirmations shown and operator responses,
  • final side effects.

If an agent can take visible actions but you cannot reconstruct what it saw and did, the safety boundary is mostly performative.

9. Release gate checklist

Before enabling browser/computer-use in a real workflow:

Check Expected
Disposable sandboxed environment Yes
Domain or app allowlist Yes
Confirmation on login, submit, send, publish, upload, delete Yes
Prompt-injection test cases using hostile web/doc content Yes
File-transfer quarantine path Yes
Operator-visible stop / reset path Yes
Session reset between tasks Yes
Action log tied to screenshots or state transitions Yes

Pair this guide with agent-evals-red-teaming.md before widening scope.

10. Incident indicators

Investigate immediately if you see:

  • navigation to domains outside the expected task,
  • the agent following text that looks like instructions from a page or email rather than from the operator,
  • local file access unrelated to the task,
  • credentials typed or pasted into unexpected fields,
  • unexpected downloads/uploads,
  • repeated retries on sensitive flows,
  • missing confirmations for actions that should have paused.

If any of these escaped a test environment, route to ai-agent-incident-response.md.

11. Official references