docs: add A21 nested-mountpoint EROFS failure mode to runner doctor catalogs (#7706)

* Initial plan

* docs: add runner doctor A21 failure mode updates

* docs: clarify A21 mountpoint diagnostics

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

Author: Copilot

.github/agents/self-hosted-runner-doctor.md

@@ -113,6 +113,7 @@ Prefer the narrowest match. Examples:
 - `EACCES` / write failures from XDG-respecting tools (Flutter, etc.) writing directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind` → A18 (`XDG_CONFIG_HOME` captured stale root-owned home before `HOME` updated to writable arc-dind path; fixed in github/gh-aw#48658)
 - `create_pull_request` fails with `No patch file found` on ARC/DinD despite safeoutputs reporting a successful write → A19 (safeoutputs `/tmp/gh-aw` mount not `docker-host-path-prefix`-translated; fixed in github/gh-aw-firewall#6959)
 - `awf-agent` fails to start under `runner.topology: arc-dind` (runc cannot create the `/dev/null` credential-hiding overlay mountpoints under `/host$HOME`), or the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` → A20 (sysroot filter dropped every mount targeting `/host$HOME`, including a caller-supplied writable home; fixed in github/gh-aw-firewall#7244)
+- `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21
 
 ### 4. Check for known gaps and notable fixes
 
@@ -130,6 +131,8 @@ A19 / github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958
 
 A20 / github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 — Under `runner.topology: arc-dind`, `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount. This caused either `awf-agent` startup failure (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`) or, once worked around, an entrypoint abort at `mkdir -p /host$HOME/.m2` under `set -e`. **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup. `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`.
 
+A21 / github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681 — When a `filesystem.allowWrite` policy narrows `/tmp` to read-only, `awf-agent` startup can fail with `runc create failed: ... mkdirat ... read-only file system` for nested AWF control-plane mountpoints such as `/tmp/awf-init` or `/tmp/awf-runner-bin`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers that destination; nested mountpoints under a now-read-only `/tmp` therefore fail with `EROFS`. On ARC/DinD split-filesystem runs with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF's own workDir-derived binds as daemon-only and failed closed. **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`, helper staging moved to `/run/awf-lib` as related hardening, nested mountpoints are planned/created ahead of time via `planNestedMountpoints()`/`ensureNestedMountpoints()`, and `isSharedDockerHostPathPrefix` now treats only literal `/tmp` as shared in ARC/DinD detection. Legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images. Upgrade AWF to include github/gh-aw-firewall#7679.
+
 B8 / github/gh-aw-firewall#5983 — Pre-flight EACCES on persistent runners from stale root-owned `/tmp/gh-aw/` dirs is **fixed** in AWF version including github/gh-aw-firewall#5983 (`preflight-reclaim.ts`). Workaround: `sudo rm -rf /tmp/gh-aw/sandbox`.
 
 B9 / github/gh-aw-firewall#5783 — RHEL/Amazon Linux CA bundle not accessible in chroot is **fixed** in AWF version including github/gh-aw-firewall#5783. Workaround: copy `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` to a chroot-visible path and set `SSL_CERT_FILE`/`NODE_EXTRA_CA_CERTS`/`REQUESTS_CA_BUNDLE`/`CURL_CA_BUNDLE`/`GIT_SSL_CAINFO`.
@@ -251,6 +254,7 @@ Establish these facts before matching a failure mode:
 | A18 | XDG-respecting tools (Flutter, etc.) fail with `EACCES` / permission errors; writes land directly under `/home/runner` (for example `/home/runner/tool_state`) under `runner.topology: arc-dind`; the actual write target is under `/home/runner` (root-owned) rather than the writable `${RUNNER_TEMP}/gh-aw/home` | The copilot engine entrypoint (`gh-aw` v0.79.8+) emits `export XDG_CONFIG_HOME="$HOME"` before reassigning `HOME` to the writable arc-dind path `${RUNNER_TEMP}/gh-aw/home`. Any XDG-respecting tool sees the stale, unwritable value. `engine.env` is sourced before this export, so `XDG_CONFIG_HOME` set there is silently overwritten by the later shell export. | **Fixed in gh-aw (PR github/gh-aw#48658, merged 2026-07-28):** `XDG_CONFIG_HOME` is now exported after `HOME` is reassigned to the writable arc-dind path. Upgrade gh-aw to the version including github/gh-aw#48658. **Workaround (older gh-aw):** Setting `XDG_CONFIG_HOME` in `engine.env` is ineffective because the later shell export overwrites it; override `HOME` to the writable path instead (e.g. add `HOME=${RUNNER_TEMP}/gh-aw/home` to `engine.env`). | Inside the arc-dind agent container: `echo "$XDG_CONFIG_HOME"` — if it shows `/home/runner` rather than a path under `$RUNNER_TEMP`, the ordering bug is present; `ls -la /home/runner` — root ownership confirms the mode | github/gh-aw-firewall#6684, github/gh-aw#48658 |
 | A19 | `create_pull_request` fails with `No patch file found` / `No patch or bundle files found in: /tmp/gh-aw` on ARC/DinD even though the safeoutputs MCP server inside the agent container reports it successfully wrote `aw-.patch`/`aw-.bundle` | The `/tmp/gh-aw:/tmp/gh-aw:rw` bind mount used for the safeoutputs patch/bundle handoff was not passed through AWF's existing `translateBindMountHostPath()` / `--docker-host-path-prefix` normalization (`src/services/agent-volumes.ts`). In DinD split-filesystem topologies the Docker daemon resolves the bind source against its own filesystem, not the runner's staged path, so writes made inside the container land somewhere the runner-side ingestion step never sees. | **Fixed in AWF (PR github/gh-aw-firewall#6959, merged 2026-08-05)**: the safeoutputs exchange mount source is now built through the same `docker-host-path-prefix` translation path as other agent bind mounts (the generated Compose target remains `/host/tmp/gh-aw`, which is `/tmp/gh-aw` inside the chroot; only the host-side bind source changes when a prefix is configured). Upgrade AWF to include github/gh-aw-firewall#6959. | After a `create_pull_request` failure on ARC/DinD, check whether `--docker-host-path-prefix` is set and inspect the generated Compose bind mount for `/host/tmp/gh-aw` — on unpatched AWF the host source is untranslated (e.g. `/tmp/gh-aw:/host/tmp/gh-aw:rw` instead of `<prefix>/tmp/gh-aw:/host/tmp/gh-aw:rw`) | github/gh-aw#50217, github/gh-aw-firewall#6948, github/gh-aw-firewall#6958, github/gh-aw-firewall#6959 |
 | A20 | Under `runner.topology: arc-dind`, `awf-agent` fails to start (runc cannot create the ~30 credential-hiding `/dev/null` overlay mountpoints under `/host$HOME`), or once worked around, the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`, including the compiler-supplied writable home (`${RUNNER_TEMP}/gh-aw/home`), because it could not distinguish AWF's own unshared `${workDir}-chroot-home` mount (correctly dropped) from a caller-supplied, daemon-visible home mount | **Fixed in AWF (PR github/gh-aw-firewall#7244, merged 2026-08-11):** home mounts whose target matches an explicitly supplied `--mount`/`config.volumeMounts` spec now survive the sysroot filter (the caller vouches for daemon-visibility); AWF's own workDir-based chroot-home mount is still dropped. If no writable `/host$HOME` survives, `/dev/null` credential overlays under that path are skipped with a warning instead of failing runc startup (overlays at the un-prefixed `$HOME` path are unaffected). `containers/agent/entrypoint.sh`'s JVM proxy pre-seeding now guards its `mkdir -p .../.m2` call and logs+skips instead of aborting under `set -e` when the chroot home is read-only. `buildCustomVolumeMounts()` (`src/services/agent-volumes/workspace-mounts.ts`) also stops re-prefixing targets that already start with `/host`, fixing a related double-`/host` bug for `--mount src:/host/path:ro` specs. Upgrade AWF to include github/gh-aw-firewall#7244. **Not addressed:** gh-aw emitting `${RUNNER_TEMP}/gh-aw` read-only over the chroot home, and its `DOCKER_HOST` gate on the chroot config patch — both require changes in `github/gh-aw`. | Inspect `docker-compose.redacted.yml` for a writable `/host$HOME` (or its explicit-mount target) under `runner.topology: arc-dind`; check agent startup logs for the "no writable home survived, skipping overlays" warning vs. a runc mountpoint-creation failure; check entrypoint logs for the "Cannot create .../.m2 (read-only home)" skip message | github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 |
+| A21 | `awf-agent` fails to start with `runc create failed: ... mkdirat /var/lib/docker/overlay2/<layer-id>/merged/tmp/awf-init: read-only file system` (or equivalent for `/tmp/awf-runner-bin`) when a `filesystem.allowWrite` policy narrows `/tmp` to read-only; most reliably reproduced on ARC/DinD split-filesystem topologies using `--docker-host-path-prefix` | runc creates missing bind mountpoints with `mkdirat` against whichever bind already covers the destination. AWF control-plane mountpoints (`/tmp/awf-init`, `/tmp/awf-runner-bin`) were nested under the user-narrowable `/tmp` bind, so narrowing `/tmp` to `ro` blocked nested mountpoint creation and failed startup with `EROFS`. `/tmp/awf-lib` was helper-copy staging rather than a nested mountpoint; narrowing `/tmp` could silently prevent those copies. On ARC/DinD with a `/tmp`-rooted `--docker-host-path-prefix`, shared-prefix detection also misclassified AWF workDir-derived binds as daemon-only and failed closed. | **Fixed in AWF (PR github/gh-aw-firewall#7679, merged 2026-08-24):** init-signal moved to `/run/awf-init`; helper staging moved to `/run/awf-lib` as related hardening; a new `planNestedMountpoints()`/`ensureNestedMountpoints()` pass pre-creates mountpoints that would land inside read-only covers (or fails closed); `isSharedDockerHostPathPrefix` now treats only the literal `/tmp` prefix as shared for ARC/DinD detection; legacy `/tmp/awf-init` compatibility binds remain for older pinned agent images. Upgrade AWF to include github/gh-aw-firewall#7679. | Check `awf --version` for github/gh-aw-firewall#7679; inspect startup logs for `mkdirat ... read-only file system` with active `filesystem.allowWrite`; on ARC/DinD verify whether `--docker-host-path-prefix` is exactly `/tmp` (shared) vs. daemon-only (for example `/host`) | github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681 |
 
 ## Category B — Self-hosted runners
 
@@ -373,6 +377,7 @@ Establish these facts before matching a failure mode:
 | `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; AWF-side fixed via `ensure_usr_local_bin_shims()`/`prepare_usr_local_bin_overlay()` in github/gh-aw-firewall#7245; durable upstream fix still tracked in github/gh-aw-firewall#7130, open) |
 | `runc` mountpoint creation failure for `/dev/null` credential overlays under `/host$HOME` on `runner.topology: arc-dind` | A20 |
 | `mkdir -p .../.m2` failing under `set -e` in agent entrypoint on `arc-dind` | A20 |
+| `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) | A21 |
 
 ## Known unresolved items