Mock-service tasks can fall back to operator-hosted account connectors

[kywch]

Summary

A BenchFlow rollout provided local mock Google Workspace services but omitted sandbox.network_mode, inheriting BenchFlow's public default. During one codex-acp subscription-auth trial, the intended mock GWS path failed. Codex then invoked the operator's real connected Google Drive through the hosted mcp.codex_apps.google_drive.search tool seven times.

No writes occurred. Real Drive metadata/content was returned to the agent and retained in local trajectory artifacts. The task evaluator did not surface the unexpected real-service access.

Codex supplies confirmed evidence, but exposure class is cross-harness: Claude Code can derive connectors from claude.ai subscription auth, Gemini receives host auth plus settings that may declare MCP, OpenClaw enables provider plugins, and external agents have heterogeneous credential/config behavior. Each requires separate version-pinned validation; only Codex Drive access is confirmed here.

This report intentionally excludes private trajectory content, account identifiers, queries, and returned Drive data.

Impact

This incident was not a bypass of an active no-network policy: the task inherited public networking. It exposed a separate missing boundary between task-provided tools and operator/account connectors. A provider-hosted connector is also outside the Docker/Daytona process boundary, so container egress controls alone cannot enforce that boundary.

Consequences:

• benchmark agents may read or mutate operator-connected services that were never declared by the task;
• real account data may enter provider context and local run artifacts;
• task reward can look valid while the agent used tools outside the benchmark's intended mock-service boundary;
• future Docker and Daytona process-egress tests could pass while hosted connector access remains available.

This is especially risky for subscription/OAuth runs because the host account may already have connectors enabled.

Cross-harness threat model

This is not only a Codex Apps concern. BenchFlow launches several full coding-agent CLIs, supplies provider or subscription credentials, and sometimes copies agent settings. Those CLIs can derive tools from an account, user config, project config, extensions, plugins, or MCP in addition to task-declared tools. Personal email, calendar, documents, chat/Slack, and Drive are representative data classes; only Drive access is confirmed in this incident.

Harness	Auth/config entering sandbox	Current restriction in BenchFlow	Potential personal/account-data path	Evidence status
codex-acp	Copies host ~/.codex/auth.json for native subscription auth; task skills are installed separately.	Public-default runs apply no no-web suffix. At incident revision, restricted runs added tools.web_search=false only. Proposed narrow fix also adds features.apps=false.	Subscription identity can expose hosted Codex Apps connected to operator services. Real Drive search/read occurred through mcp.codex_apps.google_drive.search. Similar account Apps could represent mail, calendar, docs, chat, or Drive.	Confirmed: seven real Drive reads. Other Codex Apps/data classes untested.
claude-agent-acp	Copies host ~/.claude/.credentials.json. Current BenchFlow path does not copy host ~/.claude.json, host user settings, or host project-local settings. Task/workspace files such as .mcp.json or .claude/settings*.json may still be present if included in the task workspace.	Restricted runs merge deny rules for WebSearch and WebFetch; no current disableClaudeAiConnectors, ENABLE_CLAUDEAI_MCP_SERVERS=false, or complete MCP deny policy.	Claude Code documents that claude.ai subscription sessions automatically fetch account connectors. Project MCP/settings can add more servers; user/local MCP settings are a known Claude surface but are not copied from host by BenchFlow today. Such connectors may expose mail, calendar, docs, Slack/chat, Drive, databases, or other account data.	Code/documentation-supported risk: subscription credential is copied and current restriction does not disable claude.ai connectors. No ClawsBench live personal-data access reproduced.
gemini	Subscription path copies ~/.gemini/oauth_creds.json, full host ~/.gemini/settings.json, and ~/.gemini/google_accounts.json. Host ~/.gemini/extensions/ is not copied by this path; task workspace config may still contribute settings.	Restricted runs add google_web_search and web_fetch to tools.exclude. They do not set admin.mcp.enabled=false or admin.extensions.enabled=false, and copied settings may contain mcpServers.	Copied user settings or workspace settings can register MCP servers. Gemini extensions can also provide MCP servers when extension code is present. Those servers may use account/service credentials to expose mail, calendar, docs, chat, or Drive; Google login alone is not evidence that these services are exposed.	Code/documentation-supported risk: complete host settings are copied and current policy leaves MCP enabled. No live personal-data access reproduced.
OpenClaw	Shim writes provider auth into a fresh OpenClaw home; OpenAI key profiles or Google ADC may be written. Vertex setup explicitly enables OpenClaw's Google provider plugin. Host personal OpenClaw config is not copied by the examined path.	No verified generic no-web, hosted-connector, plugin, or MCP restriction is declared through the external manifest contract.	Provider plugins, task/workspace configuration, or future copied plugin state could expose resources reachable by injected credentials. Google provider plugin enablement proves plugin activation, not access to personal Workspace data.	Code-supported capability surface; personal-data exposure untested.
Other built-in/external agents	Varies by registry entry/manifest: provider env, API keys, auth files, task MCP, skills, and workspace config may be supplied. External manifest contract does not currently express disallow_web_tools_* or a connector-isolation capability.	No cross-agent connector policy. Restrictions exist only where BenchFlow has a harness-specific core AgentConfig mechanism.	Any agent that imports user/project config, discovers MCP/extensions/plugins, or derives hosted tools from account auth may reach operator or organization data outside task-declared services.	Threat-model coverage only: must be tested per pinned agent/version/auth mode; do not assume exposure or containment.

Relevant upstream behavior is documented for Claude Code account connectors and MCP scopes and Gemini CLI settings, MCP, and extensions. BenchFlow-specific statements above come from current credential-copy and no-web policy code; rows explicitly distinguish copied state from possible CLI inputs.

Confirmed incident

• Harness: codex-acp using subscription auth.
• Task environment: local mock productivity services; no public internet needed by task tools.
• Declared task policy: no sandbox.network_mode; BenchFlow defaulted to public.
• Trigger: expected mock GWS tool path failed.
• Observed fallback: seven mcp.codex_apps.google_drive.search calls against the operator's connected Drive.
• Effect: reads only; no external writes observed.
• Detection: raw ACP capture from fix(acp): keep tool_call content, rawInput, and rawOutput in captured trajectories #1100 preserved enough tool-call input/output to prove the access.
• Evaluation: task evaluator did not flag the unexpected external connector calls.

Minimal reproduction

1. Use codex-acp with ChatGPT subscription auth on an account that has a Google Drive app connected.
2. Run a task that omits sandbox.network_mode, provides local mock Drive/Docs services, and requires document lookup.
3. Make the expected mock client unavailable or return a recoverable tool error.
4. Observe the agent tool list and ACP trajectory.

Current result: task inherits public networking, and Codex can discover and invoke mcp.codex_apps.google_drive.*. The connector belongs to the operator account rather than the task's mock service set.

Expected result: BenchFlow provides a policy that makes undeclared hosted connectors absent or rejects their use before execution, independently of whether the task also permits process networking. If BenchFlow cannot enforce the requested policy for the selected harness/version/auth mode, the rollout fails before sandbox provisioning.

Current behavior and likely boundary gap

BenchFlow currently derives a generic "disallow web tools" decision from task networking policy, then applies harness-specific setup commands or launch suffixes. The incident task inherited public networking, so this restriction was not activated. The available Codex restriction at that revision disabled tools.web_search but did not disable Codex Apps.

Strict no-network with native Codex subscription auth currently fails before the prompt because the no-web path requires a sandbox-local loopback provider endpoint. Therefore this incident does not prove that a strict no-network subscription rollout can reach Apps; that path never reached agent execution in follow-up testing.

Process-level isolation and agent-level tool isolation are different controls:

• sandbox.network_mode and sandbox backends govern container/process egress when that policy is active;
• model/provider traffic may still require a controlled outbound lane;
• hosted Apps/connectors execute outside that process boundary;
• task-declared MCP servers and operator/account connectors currently lack one explicit effective allowlist;
• reward/evaluator output does not independently enforce forbidden tool namespaces.

Confirmed versus suspected exposure

Confirmed:

• Codex Apps remained available in a subscription run whose task omitted network policy and inherited public.
• A real connected Drive was read seven times.
• The task's mock-service set did not constrain the hosted tool catalog.
• raw ACP capture exposed the calls; normal task evaluation did not.

Needs harness-specific reproduction:

• Claude Code currently denies WebSearch/WebFetch, but account connectors or undeclared MCP servers may remain reachable.
• Gemini currently excludes built-in web search/fetch while copying host auth/settings; extensions or MCP configured there may survive.
• OpenClaw and external manifest agents do not have a verified generic hosted-connector restriction contract.
• connector mutation paths may permit writes; this incident exercised reads only.
• after native subscription support is added to strict no-network mode, hosted connectors may remain outside process-egress controls unless separately disabled; this has not been reproduced because current strict mode fails before prompt.

Short-term mitigations

1. Require mock-only benchmark tasks to declare sandbox.network_mode: no-network; do not rely on BenchFlow's public default. Native subscription runs currently need a safe provider relay or must fail clearly before prompt.

2. For Codex no-network/no-search runs, add both launch overrides:

   -c tools.web_search=false -c features.apps=false
   

3. Consider disabling hosted account Apps by default for benchmark runs, with explicit task opt-in, rather than coupling this only to process networking.

4. Add equivalent fail-closed connector/MCP restrictions for each supported subscription harness. Candidate controls to verify include Claude Code's disableClaudeAiConnectors/ENABLE_CLAUDEAI_MCP_SERVERS=false plus an MCP allowlist, and Gemini's admin.mcp.enabled=false/admin.extensions.enabled=false with sanitized settings. Do not claim support until a live negative test proves operator connectors are absent.

5. Start from sanitized harness config for evals. Copy only auth material required for model access; do not copy user extensions, MCP config, or connector-enabling settings.

6. Keep task-declared MCP servers as an explicit allowlist. Reject any effective tool server or namespace outside it.

7. Scan captured trajectories for forbidden provider/connector namespaces and report an infrastructure-policy failure independent of task reward.

8. Record effective restrictions, harness/CLI version, auth mode, agent-definition provenance, and manifest digest in run artifacts.

The Codex launch override is a narrow mitigation for the confirmed path. It is not a substitute for a cross-harness capability and enforcement model.

Proposed quick-fix ownership and lifecycle

Use BenchFlow's existing conditional policy hook rather than embedding restrictions in agent installation or the task image:

1. BenchFlow registry entry declares each harness-specific disallow_web_tools_setup_cmd and/or disallow_web_tools_launch_suffix.
2. BenchFlow installs the agent/package normally.
3. BenchFlow writes provider credentials and uploads subscription auth into that sandbox.
4. src/benchflow/agents/install.py::apply_web_tool_policy runs the selected setup command as sandbox root when task policy requests restriction.
5. BenchFlow launches the agent with any restricted launch suffix.

Current rollout ordering already places apply_web_tool_policy after package installation and credential/config upload, and before agent launch. This lets the policy override copied user settings without making agent installation itself policy-dependent. Setup failure already raises instead of silently continuing.

Proposed system-policy recipes, not implemented yet:

Harness	Ephemeral root-owned policy target	Purpose
Codex	/etc/codex/requirements.toml	Enforce Apps/web/MCP restrictions through Codex's system requirements layer; retain launch overrides where required by pinned CLI behavior.
Claude Code	/etc/claude-code/managed-settings.json	Disable claude.ai connectors and deny undeclared MCP through managed settings that user/project config cannot loosen.
Gemini CLI	/etc/gemini-cli/settings.json	Disable undeclared MCP/extensions through system settings while preserving explicitly task-authorized tools.

Create these files inside each trial sandbox only when effective task/agent policy requests them. Keep them root-owned and non-writable by the agent user. Remove them with the ephemeral sandbox at trial end. Do not put this logic in normal install_cmd: installation is shared agent setup, while restriction is conditional per task/run. Do not bake it into env0: env0 is a reusable task environment, and unconditional system policy would also affect tasks that explicitly allow those capabilities.

Each recipe must be tested against the pinned harness/CLI version. If a version does not honor its system policy or BenchFlow cannot verify the resulting effective tool surface, restricted execution should fail before prompt.

BenchFlow surfaces to inspect

• src/benchflow/task/config.py: network_mode, deprecated allow_internet, and policy semantics.
• src/benchflow/rollout/_setup.py: task-policy detection and launch-policy construction.
• src/benchflow/rollout/__init__.py: policy activation, auth/config setup, MCP delivery, ACP connection, and ordering relative to lockdown.
• src/benchflow/agents/registry.py: harness-specific disallow_web_tools_*, subscription auth, home/config paths, and task MCP transport.
• src/benchflow/agents/install.py: application and failure handling for harness policy setup.
• src/benchflow/agents/manifest.py and the benchflow-ai/agents manifest schema: policy fields are currently shim/core-only rather than declarative manifest capabilities.
• src/benchflow/agents/credentials.py: minimum auth state copied into sandbox; sanitization boundaries.
• src/benchflow/sandbox/setup.py: preserve_agent_network changes no-network backend handling so model/install traffic can remain possible.
• src/benchflow/sandbox/lockdown.py: sandbox-user UID firewall; useful for process egress, unable to contain provider-hosted tools.
• src/benchflow/sandbox/docker.py, daytona.py, and daytona_strategies.py: backend enforcement and parity.
• src/benchflow/acp/client.py: permission auto-approval and whether undeclared connector calls can be rejected at protocol boundary.
• src/benchflow/acp/runtime.py: task MCP attachment and agent session setup.
• src/benchflow/acp/session.py and src/benchflow/trajectories/_capture.py: complete tool-call capture.
• trajectory exporters/results/evaluation: forbidden-tool audit, policy outcome, and artifact redaction.

Relationship to registry refactor

#1089 is relevant infrastructure: deterministic agent resolution, immutable agent-definition provenance, version/manifest compatibility, and pre-sandbox failure make security capability checks auditable. It does not itself define or enforce hosted-connector isolation.

This issue should remain separate and reference #1089. Long-term ownership should follow that boundary:

• benchflow-ai/agents owns version-specific policy templates/mechanisms and exposes them through a structured manifest capability;
• BenchFlow owns task-policy trigger, application timing, capability/version validation, fail-closed behavior, and recorded evidence.

A registry refactor can provide the place to resolve those manifest capabilities, but BenchFlow still needs runtime policy semantics, live conformance tests, and trajectory auditing.

#1090/#1093 move OpenClaw-specific adapter ownership to benchflow-ai/agents and explicitly avoid a generic MCP capability model. Connector isolation should not expand or block that extraction. Follow-up manifest-contract work can place harness-specific mechanisms in benchflow-ai/agents, while BenchFlow owns task policy, enforcement decisions, fail-closed preflight, and evidence.

No current benchflow-ai/agents issue appears to track connector isolation. agents#73 concerns OpenClaw streaming/cancellation, and agents#74 pins the Codex CLI used for Astra support. They are useful version/parity context, not substitutes for this security work.

Architecture questions

• Should policy distinguish process egress, provider/model egress, built-in web tools, hosted account connectors, task MCP, user MCP/extensions, and startup/install traffic?
• Where should versioned harness mechanisms live: agent manifests, tested adapter code, or BenchFlow built-ins?
• What capability declaration proves a harness can disable hosted connectors for a given CLI version and auth mode?
• Can BenchFlow query the effective tool catalog before the prompt and compare it with a task allowlist?
• Should unknown tool namespaces fail execution, or only calls to them?
• How should native subscription traffic reach providers without reopening arbitrary agent process egress?
• Which artifacts may retain connector input/output, and what redaction or quarantine is required after a policy violation?

Acceptance criteria

• A task-level connector policy can prohibit operator-hosted Apps independently of network_mode.
• A public-network task with hosted connectors prohibited cannot list or invoke Codex Apps.
• A strict no-network Codex subscription trial either reaches model inference through a permitted provider path without exposing Apps, or fails preflight with a clear unsupported-path error.
• Equivalent Docker and Daytona tests prove separately: strict no-network permits localhost task mocks while blocking public process egress; connector policy removes hosted connectors even when process networking is public.
• Claude, Gemini, OpenClaw, and external agents either pass version-pinned connector-isolation conformance tests or fail preflight with an actionable unsupported-policy error.
• Only task-declared MCP servers reach session/new or native agent config.
• Host/user MCP, extension, and connector config is not copied unless explicitly declared by task policy.
• A synthetic forbidden connector call yields a distinct infrastructure-policy failure even if verifier reward is otherwise successful.
• Raw ACP capture retains tool name, arguments, result/status, and enough provenance for audit without exposing secrets in summary artifacts.
• Run artifacts record task network policy, effective agent restrictions, auth mode, agent/CLI version, resolved agent source SHA, and manifest digest.
• Policy setup occurs before any untrusted agent initialization capable of tool use.
• Restricted trials create the selected /etc policy file after install/auth setup and before launch; it is root-owned, agent-non-writable, and recorded without secret content.
• Unrestricted trials do not create those conditional /etc policy files.
• Agent install_cmd and env0 remain policy-neutral; changing task policy does not require rebuilding either.
• Regression test covers actual trigger: public-default task, mock-client failure, then attempted fallback to an operator connector.
• Separate future test covers native subscription plus strict no-network after that provider path is supported; the incident is not used as evidence that this combination currently reaches prompt.

Related work

• Sandbox isolation leaks host state and weakens no-internet guarantees #363 — sandbox isolation and no-internet guarantees.
• feat(sandbox): Enforce network_mode allowlist on docker #785 — Docker network_mode allowlist enforcement.
• fix(sandbox): resolve the no-web loopback endpoint per adapter #940 — sandbox-local no-web provider endpoints.
• Agent pre-launch policy assumes undeclared runtimes in task images #1047 / fix(agents): run the no-web JSON policy on the provisioned Node runtime #1048 — agent pre-launch no-web policy runtime assumptions and fix.
• Capture LLM trajectories across OAuth and API-key runs #1057 — trajectory capture across OAuth and API-key runs.
• Harden agent catalog resolution, provenance, and discovery #1089 — agent catalog resolution, provenance, and discovery.
• Extract OpenClaw shim from BenchFlow core into benchflow-ai/agents #1090 / refactor(agents): extract OpenClaw from BenchFlow core #1093 — OpenClaw extraction and adapter ownership boundary.
• fix(acp): keep tool_call content, rawInput, and rawOutput in captured trajectories #1100 — preservation of ACP tool-call content, rawInput, and rawOutput; enabled incident confirmation.
• fix(openclaw): stream ACP events during prompts agents#73 — OpenClaw ACP event streaming; adjacent adapter queue only.
• fix(codex-acp): pin Codex 0.153.4 for Astra support agents#74 — Codex CLI pin; relevant to version-pinned policy validation.