Fix public-beta local reinstall blockers - #4215
Conversation
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
…ic-beta-local-reinstall # Conflicts: # ods/tests/pixel_inference/test_advice_setup_process.py
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
|
Fleet checkpoint after the terminal paired-WSL run and a current-public-beta forward merge. Terminal frozen run: ODS
|
|
Follow-up: exact merged head |
|
Fresh exact-pair qualification is live: product |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
|
Tower3 exact-head canary found and this branch now fixes a real Ubuntu 26.04 Pixel bootstrap failure.
Tower3 was restored to its exact Qwen workload after the failed disposable install, and a new exact-head canary is running with this commit. |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
|
Exact fleet candidate advanced to Tower3's real Ubuntu 26.04 install reached Pixel Operations Broker activation and exposed a Pixel access-probe defect: Rust coreutils Evidence: the isolated Tower3 bridge reproduction showed external test false, actual read true, Bash read true, and Bash write denied; a clean Tower2 exact checkout passed |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
|
Fleet update — exact candidate
No fleet-green claim yet. The strict five-machine chain remains gated on Tower3 release confidence, installed Pixel journeys, model switching, distro lanes, and restored pre-state. |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
|
Fleet update — exact candidate fe4e53b Tower3 retry run r6 reproduced an interrupted ODS-managed Pixel installation: the exact release, active link, candidate sandbox image, OpenClaw configuration, and gateway unit existed, but runtime attestation had not yet been written. ODS correctly detected the state, then failed closed because uninstall required fields that are only persisted after successful verification. This commit adds a narrowly bounded recovery path:
Evidence on the clean exact Tower2 checkout matching the pushed diff:
The next acceptance step is an exact-head Tower3 retry against the untouched interrupted state, followed by Pixel capability and model-switch qualification. No fleet-green claim yet. |
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
|
Tower3 retry update — exact candidate abfe807 Canary r7 safely reached the preserved interrupted Pixel state and exposed one narrower lifecycle boundary: the freshly cloned ODS tree already ships extension_manager.py and workspace_preview.py, while their generated owner-unit contracts are not created until a later apply step. The retry validator treated that source-only installing state as an incomplete deployed service. This commit permits source-only lifecycle files only for the already bounded inactive-installing or exact minimal unattested-install recovery states. A generated unit without its source still fails closed, and ready or partially bound markers remain rejected. Regression evidence from the clean Tower2 checkout:
An exact-head Tower3 r8 retry is next. No fleet-green claim yet. |
Summary This integration PR collects ODS product fixes found by repeatedly installing and exercising
public-betaon the Dream Fleet. It remains based onpublic-beta, notmain, and is not a fleet-green claim. ## Exact candidate0642ad018f5c91c861a1666011045f705ab1400epublic-betabase:d49ad4bbc760be319de7f4c9b9a93b1f18fe392265a0516c92b5c0339f63df7e92b6059b14e2747fbbd1d2d62c7260f822ba1e727728a0a02f78895fProduct fixes accumulated here - reconcile optional-service state during reinstall without pruning user-selected services - recover stale or recycled Pixel admission state while retaining strict source, ownership, and catalog-custody checks - support isolated native-Docker WSL Pixel installs with validated gateway/preview ports - repair Windows/WSL bootstrap, model-download, host-agent, Hermes-config, model-promotion, and restart behavior - surface and safely answer authenticated Dream Talk approvals - pull pinned MinIO images from Quay after the Docker Hub paths stopped resolving - keep model-router activation and terminal failure reporting fail closed - allow the GPU-independent Aider CLI extension to resolve on supported CPU-only installs - bind the privileged Pixel access bridge to the root-recorded gateway port instead of guessing the default when owner config omits a custom port - re-run the existing owner-authenticated access proof ceremony after ODS-controlled gateway restarts, while refusing busy, pending, ambiguous, or differently configured states The latest product deltas address a live Strixy/laptop Pixel control-plane failure. ODS now persists the validated Pixel gateway port in the root-only coordinator configuration and performs a narrowly admitted same-mode proof through the existing access controller after an ODS-owned gateway restart. It never writes a verified state directly; the existing controller still acquires both admission gates, checks the configured service boundary, executes its runtime probe, and records proof atomically. Any unavailable, busy, pending, unknown-mode, stale-revision, or otherwise ambiguous state remains fail closed. ## Current verification At predecessor
35d7c697: - all GitHub checks are green, including Linux integration, dashboard API/frontends, Pixel inference contracts, environment-schema tiers, PowerShell lint, and distro smoke - Aider dashboard config regression: 60 passed - extension manifest validation: 33/33 library manifests and 30 compatible core manifests, with zero errors - the predecessor dashboard model-activation module: 298 passed, 1 skipped; dedicated Hermes restart regression passed At predecessor product00c25411, all GitHub checks are green, includingintegration-smoke; focused Linux verification is also green: 280/280 Pixel host-install contract checks, 16/16 access-bridge tests, and 5/5 post-restart recovery tests. The live fresh-install failure below demonstrates why those source checks were necessary but insufficient. At predecessor harness5cb52c1, the Pixel/model-switch contract passes on both the laptop checkout and a clean detached Tower2 checkout. It now requiresavailable=true,runtime_verified=true, matching configured/effective modes, and bothbusy=falseandpending=falsebefore chat evidence can count. Harness #40 merged at0303687; the follow-up gate is now carried by the still-open integration PR #39. All GitHub checks on that predecessor are green. Current harness 2a694d adds target-scoped llama-slot quiescence for the Strixy CPU lane while retaining the strict 90-second runtime usability gate; its new GitHub checks are running. The predecessorcc4100afportability candidate passes the full Docker multi-distro matrix (10/10 lanes) and the clean Incus VM matrix on Ubuntu 24.04, Fedora 42, Rocky 9, Arch, and openSUSE Tumbleweed. Because the product head advanced, those live matrices are predecessor evidence, not exact-head acceptance. The terminal predecessor laptop run (f71d14b3/420f9b5) proved clean install, Product Green, Pixel Green, Capability Green, and Lifecycle Green. Model UI remained red: - cycles 1, 3, and 4 passed - cycle 2 returned an observed 502 when Hermes became unhealthy during Qwen restoration; cleanup restored Qwen and deleted the test model - cycle 5 stopped before UI work whenods restartleft Hermes unhealthy - cycle 6 proved Pixel, Open WebUI, OpenCode, Perplexica, and route evidence, but a lost harness credential read caused an unauthenticated LiteLLM 401; direct installed-system probes proved unauthenticated 401 and authenticated 200 The Hermes failures are addressed by the latest product commit. The activation-response and credential-attribution defects are addressed in harness #40. No predecessor result is counted as acceptance. The first exactcc4100afpaired-WSL attempt stopped before product installation because of two harness preflight defects: a lost laptop WSL privilege probe was mislabeled as missing sudo, and Strixy's absent Docker Desktop engine was treated as a fatal inventory error. Both are fixed and covered at harnessad4f934; that failed attempt makes no product claim. A replacement exactcc4100af/ad4f934diagnostic run installed successfully on both isolated WSL lanes. The laptop passed UI, Pixel, broad capabilities, and lifecycle. Strixy exposed three fail-closed results: the CPU-only no-GPU telemetry response was treated as a generic 503; Aider was accepted by the dashboard but excluded by its CPU-incompatible manifest; and Pixel chat was blocked after model reconciliation restarted the gateway. The first two defects were fixed at0303687/35d7c697. The third is now root-caused: the restart invalidated the live access proof and left the native runtimeinterrupted; the custom laptop port also proved the bridge could target an unused default port. Both product defects are fixed at00c25411, and the harness now detects this class before counting chat semantics. Strixy's CPU-bound broad capability probe remains actively progressing, so this predecessor run is diagnostic only. The clean exact00c25411/5cb52c1paired-WSL run is terminal red: the laptop lost its WSL transport before install, and Strixy reached Pixel installation but failed the final access proof. It is diagnostic evidence only and cannot validate the newer product or harness heads. ### Latest exact-pair diagnostic (f50ac59 / e73f1c55; superseded)The exact predecessor laptop-and-Strixy WSL run installed both targets from ODS
f50ac591fe31af1c15677a1648acd59389463697, harnesse73f1c55, and Pixelbbd1d2d6. It is diagnostic red and is not acceptance evidence.Host:prefix, so Caddy returned the hostless HTTP 200 catch-all and the POST never reached Talk. Harness docs: improve international accessibility of README #39 now sends the detected host header correctly; the corrected installed replay observed busy state, a real session frame, client abort, and return to idle.d809ebf1also stops immediately when a terminal failed bootstrap has no live updater, reports post-install bootstrap deferral honestly, and carries complete regression fixtures. Its full clean Tower2 suite and GitHub checks are green.A fresh exact
c9d0b48b/d809ebf1run is required; no predecessor result is promoted to fleet acceptance.Ubuntu 26.04 physical-fleet qualification (96b5ca0)
Live preflight established that Tower1 and Tower3 run Ubuntu 26.04 with kernel 7.0.0-29, Python 3.14.4, systemd 259, and Docker 29.7.2; Tower2 remains Ubuntu 24.04. The prior exact allowlist rejected both 26.04 hosts before Pixel installation even though the Docker
ubuntu:26.04manifest and Incusimages:ubuntu/26.04VM image are available.This commit admits only exact Ubuntu 26.04 while retaining Linux, PID1 systemd, WSL2, license, model-route, source-custody, and all other fail-closed gates. It adds native and WSL2 26.04 qualification regressions plus CI Docker and private Incus lanes, and updates the support/install documentation.
Evidence before push:
git diff --checkpassedThis is source and host-preflight evidence only. Installed Pixel qualification on Tower1/Tower3 still requires the declared noninteractive system privilege; both currently return
sudo -nfailure. The active c9d0b48 WSL run is now predecessor diagnostic evidence because this product head advanced, and a fresh exact-head five-host aggregate remains required.Still required before fleet acceptance - a fresh exact-head release run on the laptop and Strixy through Pixel-first post-install checks and six restart-separated model cycles - Tower2, Tower1, and Tower3 installs after their required noninteractive Pixel system privilege is provisioned - one exact candidate/harness evidence set covering all five physical hosts with no missing, skipped, stale, or red required receipts - human review; this integration PR is intentionally not merged by the qualification loop The five named test installations have explicit Pixel-license authorization. Fleet invocations pass both
PIXEL_LICENSE_ACCEPTED=trueandDREAM_FLEET_PIXEL_LICENSE_ACCEPTED=trueexplicitly; the harness neither infers acceptance nor applies Pixel to targets that do not declare Pixel support. ### Fresh-install access-edge ordering follow-up The exact00c25411/5cb52c1Strixy run exposed a new installation-order defect. The root-protected access coordinator correctly failed closed withavailable=false,configured_mode=unknown, andreason=host-command-failed. Direct inspection showedods-pixel-edgedid not exist yet. The coordinator's proof ceremony requires that container's durable transition gate, but the installer was calling the final proof before the later whole-stack Compose launch. On the same failed disposable install, starting only the exactpixel-edgeCompose service and re-running the protected reconciliation helper immediately returned{"result":"reproved","mode":"sandboxed"}. This isolates the cause: the proof code was correct; its required edge service was started too late. Commit53e2795c5990708294633f2b6c7465f0f52cf1f6addspixel-edgeto the bounded Pixel prerequisite launch alongside LiteLLM and dashboard-api. The transition endpoint can start before host-ingress chat readiness; the final proof still occurs only after ingress health, plugin loading, and the ready marker. A contract assertion fixes that ordering. Verification so far: -bash -n ods/installers/lib/pixel-host-install.sh-git diff --check- live failed-install reproduction and exact one-service recovery on Strixy - the local focused suite could not complete because this laptop's WSL service is intermittently returningWsl/Service/0x8007274c; GitHub CI and a clean Tower2 checkout are the next test authorities At current product53e2795c, all GitHub checks are green, including Linux integration, dashboard, Pixel inference contracts, PowerShell lint, the distro matrix, andintegration-smoke. A clean Tower2 exact checkout passed 280/280 Pixel host-install contracts and all five access-recovery tests. Exact53e2795c/283cc5blaptop-and-Strixy release qualification is now live with both Pixel license variables explicitly set totrue. ### Fresh access-coordinator activation-race follow-up The exact53e2795c/283cc5brerun provedods-pixel-edgepresent, running, and healthy, but a fresh Strixy install still failed the mandatory final access proof. Post-state was the narrowly saferuntime-proof-requiredprojection: available owner-host coordinator, sandboxed configured mode, no pending transition, no busy runtime, valid revision, and no runtime proof. Re-running the protected same-mode change returned HTTP 200 witheffective_mode=sandboxedandruntime_verified=true. The remaining defect was activation order:systemctl restart ods-pixel-access.serviceis asynchronous, while the installer called its protected client immediately. Commit334e8961fb9fbf03402171407b146f6eebc1cd09gives only the mandatory fresh-install proof a bounded 30-attempt, one-second retry window. The helper remains fail closed on busy, pending, ambiguous, stale-revision, mismatched, or differently configured state; no verified state is written directly. Tower2 exact-head evidence is 284/284 Pixel host-install checks plus 5/5 access-reconcile tests. The next exact334e8961/283cc5brun made Strixy installation green, then passed installed verification, cloud-mode contracts, dashboard/model-management, Hermes agentic, and Playwright UI. Pixel authentication, runtime identity, verified sandboxed access mode, and active cancellation passed. Its two chat streams reached HTTP 200 but outlived the harness fixed 300-second cap on the CPU target, so this is diagnostic rather than Pixel Green. Harness #39 atf12dc0anow applies Strixy''s 5x behavioral budget and an idle-slot gate to Pixel chats. The laptop remains independently blocked before install by its WSL transport failure (exit 82). ### Exact-head Pixel bootstrap-promotion diagnosis (7ad5d61) The terminal laptop release run at predecessor product334e8961/ harnessf12dc0apassed install, verification, cloud-mode contracts, dashboard/model-management, Hermes, Playwright UI, full-model capability reprobe, lifecycle, and all 22 regression fixtures. Pixel alone failed during the initial bootstrap window, and the harness correctly blocked model-UI acceptance. Retained SQLite bytes and the system journal isolate the failure. Pixel entered an OpenClaw provider fetch on the bootstrap Qwen 2B route; background bootstrap promotion then changed the route to Qwen 9B and restarted the gateway while that turn was active. Pixel Edge emitted the sanitized terminal stream{"error":"upstream error"}followed by[DONE]. The dashboard retained-result producer incorrectly treated[DONE]as success and marked the result complete. A stable reprobe after Qwen 9B was fully ready passed authenticated status/access, nonce-bound non-retained chat, retained replay/hash equality, and active cancellation. Commit7ad5d617634b2c52c9ebf9849b59d3a17c1b61effixes the false-success state: retained SSE containing an error frame remains replayable but is terminalinterrupted, nevercomplete. Exact clean Tower2 qualification at this head passed: - 284/284 Pixel host-install contract checks - 82/82 dashboard Pixel/access/retained-result tests in a clean test virtualenv - all current GitHub checks green, including Linux integration, dashboard, Pixel inference, lint, schema, and the distro smoke matrix This is not yet a fleet-green claim. The patch correctly reports the interrupted turn but does not by itself prevent background model promotion from overlapping an active whole Pixel turn. A durable transition exclusion spanning route mutation and gateway/ingress reconciliation remains required, followed by a fresh exact-head installed run. ### Paired harness observation hardening (939ead8) The live predecessor laptop loop completed the cycle-2 Granite H-Tiny UI, model, Talk, Pixel and broad-consumer work, restored Qwen, deleted the test model, and reached a clean final inventory before the native WSL service became unavailable. The old paired harness suppressedWsl/Service/0x8007274cand falsely labeled the later router observation as a container identity change. Paired harness commit939ead84a6f21263faede3b80e44e868fc42871anow preserves the host execution failure asobservationStatus=unavailable, keeps the cycle red and fail closed, and reservesmissingfor a successful observation with no container. Its complete clean Tower2 suite and all four GitHub jobs are green. This is evidence-attribution hardening only; it does not make the predecessor installed run exact-head or fleet green. ### Active-turn-safe model promotion (3f3a3d0) The predecessor laptop evidence showed a real race: bootstrap model promotion changed the routed model and restarted the Pixel gateway while a Pixel turn was still active. Reporting the resulting SSE error asinterruptedfixed false success, but did not prevent the interruption. Commit3f3a3d0347fab9b9aaa86b9f719cf3dfb8474283adds a root-custodied model-promotion transaction. Before the first route mutation it records durable intent, acquires and drains both native and externally reachable Pixel admission gates, and holds them across route mutation, gateway restart, ingress checks, and the final authenticated runtime/configuration proof. Rollback restores the exact begin-time route while the gates remain held. Release is native first and edge last, so any partial-release failure keeps external admission closed. A bounded root-only completion receipt makes an exact finish replayable after a lost Unix-socket response without exposing the admission token or accepting a different outcome. Exact-head source verification at3f3a3d0347fab9b9aaa86b9f719cf3dfb8474283: - clean detached Tower2 checkout: 57/57 host qualification/integration checks and 284/284 Pixel host-install contracts - 13/13 model-transition tests and 17/17 access-bridge tests - complete Pixel agent Node suite: 1,044 passed, 1 skipped, 0 failed -git diff --check, installerbash -n, native-search, compose-wiring, timeout, and artifact-promoter suites all passed - independent Tower2 GLM-5.3-Flash and Tower3 Qwen3.6 no-think reviews found no remaining actionable defect after two earlier review findings were repaired All current-head GitHub checks are green, including integration-smoke, API/frontends, Pixel inference, PowerShell lint, environment-schema tiers, security review, and the ten-distro smoke matrix. This remains source-level evidence, not installed fleet acceptance. A fresh exact3f3a3d03/ harnessfb1d8584run must prove installation, Pixel chat retention/cancellation, repeated model switching, broad consumers, restoration, cleanup, and restart-separated stability on all five named physical hosts. ### Paired clean-checkout harness unblock (e923a6f) The first exact-pair laptop attempt at ODS3f3a3d03and harness939ead84stopped in two seconds before installation because the clean harness checkout required interactive sudo for Playwright OS dependencies on Tower2. Harness PR #39 commite923a6f2e305869785820ef272232d9cdd685d69adds the already-established browser-only fallback and a behavioral regression test. The complete Tower2 harness suite passes. This was a harness preparation defect, not an ODS result; exact installed qualification is restarting against the updated pair. Harness follow-upfb1d85846f17e83db56f8724a3479770d74eecfamoves the Playwright coordination lock outside the Git checkout so subsequent install runs remain clean-attested. The dirty preflight was stopped and makes no product claim. ### Isolated model-promotion client repair (5e0d740) The first clean bundle-backed laptop install at product3f3a3d03, harnessfb1d8584, and Pixelbbd1d2d6completed installation and post-install verification, but the downloaded and SHA-verified Qwen 3.5 9B promotion rolled back to the 2B bootstrap model. The root-owned promotion log isolated the failure:python3 -I /usr/local/libexec/ods-pixel-access/pixel_model_transition.pycould not import its siblingpixel_access_client. Python isolated mode excludes the script directory fromsys.path; the transition helper had not implemented the root-custody-checked runtime import already used by the reconciliation helper. Commit5e0d7404933d7df7962ccbdf7728ff0039cd83d5validates the installed helper directory, all parents, and the sibling client as root-owned, nonsymlinked, and not group/other writable before narrowly adding that protected directory to the isolated interpreter path. Dependency injection remains available for unit tests; the privileged client is loaded only at runtime. Exact clean Tower2 source evidence at this head: - 14/14 model-transition tests - 284/284 Pixel host-install contract checks - full harness suite green at paired harness02bdcdd6531b879849afa519f0955d8fce4d01bcThis directly repairs the observed public-beta promotion rollback, but it is not yet installed acceptance. The next required result is a fresh exact-pair laptop install proving bootstrap status complete, active/model-router/Pixel runtime identity on Qwen 3.5 9B, then Pixel chat retention, cancellation, model switching, broad capabilities, cleanup, and lifecycle. The other four physical hosts remain required for fleet green. ### Owner-ready access socket activation (c41c6ed) The exact5e0d7404/02bdcddbundle-backed laptop reinstall made the isolated transition import succeed, then exposed the next activation race. Both the 9B promotion and verified rollback invoked the owner client immediately aftersystemctl restart ods-pixel-access.service; the simple service process had spawned, but its Unix socket was still in the root-only bind window, so both attempts failed withPermissionError: [Errno 13]. The installer itself returned green whilebootstrap-status.jsoncorrectly recorded failed promotion, proving again that install exit alone is not beta acceptance. Commitc41c6ede6d4f29033dba66a54558ec0574157779makes access-service installation wait at most 30 seconds for the exact root directory mode, root-owned socket, admitted owner group, mode0660, and a live read-only protocol response. The response check prevents a correctly owned but stale socket inode from satisfying readiness. It does not retrymodel-begin, whose lost-response semantics must remain fail closed. Clean Tower2 evidence is 14/14 model-transition tests,bash -n, and 284/284 Pixel host-install contract checks. A fresh exact install remains required to prove full-model promotion and Pixel runtime behavior. ### Preserve the installed Pixel gateway port during model activation (82ff7f5) The exactc41c6ede/02bdcddlaptop reinstall is terminal and diagnostic. It installed successfully, completed bootstrap promotion to the SHA-verified Qwen 3.5 9B model, passed installed verification, cloud contracts, Hermes, Playwright UI, all broad capability probes, and all 22 replayed regressions. A standalone Pixel phase before model switching proved authenticated runtime identity, verified sandboxed access, nonce-bound non-retained chat, retained replay/hash equality, and active cancellation. Dashboard model activation then returned HTTP 502, and the later Pixel phase failed only the protected access-mode status. The installed runtime remained safely on Qwen 3.5 9B. Host-agent and system journals isolated the failure: the installation selectedPIXEL_GATEWAY_PORT=18790, but_reconcile_ods_managed_pixel_modelconstructed a minimal child environment without that value. The sourced shell reconciliation therefore fell back to 18789 and rewrote/etc/ods/pixel-access.jsonto the wrong native gateway port; both activation and rollback correctly failed closed asruntime-unavailable-or-busy. Commit82ff7f5dd702aa76db790d99f49254d206818ff6loads the root-installed.envvalue, validates it as a canonical TCP port before any privileged subprocess runs, and passes it through the existing minimal environment. Legacy installs with no explicit port retain the 18789 default. Tests cover the live custom-port case, the default, and malformed/zero/out-of-range inputs. Clean detached Tower2 evidence at this head: - dashboard model-activation module: 304 passed, 1 skipped - Pixel host-install contracts: 284/284 passed - Pixel model-transition tests: 14/14 passed - Python compile andgit diff --check: passed Current-head GitHub checks are running. This is still not installed acceptance: a fresh exact-head laptop reinstall must prove the same dashboard switch plus a second Pixel access/chat/cancel pass, the six-cycle browser model matrix, lifecycle, and cleanup. Strixy and the three tower installs remain required for fleet green. ### Fail closed on an explicitly empty Pixel gateway port (2161359) The exact predecessor82ff7f5d/02bdcddlaptop run supplied the missing installed proof for the custom-port fix. Dashboard model activation returned HTTP 200activatedwith Pixelreconciled, and the subsequent Pixel phase passed authenticated runtime identity,available=true,runtime_verified=true, matching sandboxed configured/effective modes, nonce-bound non-retained chat, retained replay/hash equality, and active cancellation. The installed Qwen 3.5 9B runtime stayed healthy. An independent Tower2 GLM review found no critical or exploitable defect, but identified one fail-closed edge: an explicitly present but emptyPIXEL_GATEWAY_PORT=would have been treated like an absent key and defaulted to 18789. Commit851563f6066f05c85eb83ff72e3d29789db29701defaults only when the key is absent and rejects explicit empty/whitespace values before subprocess execution. Coverage also pins quoted valid input, port 65535, overlong numeric input, zero, leading-zero, negative, and nonnumeric cases. Clean detached Tower2 evidence at this head: - dashboard model-activation module: 309 passed, 1 skipped - focused managed-Pixel reconciliation cases: 14/14 passed - Pixel host-install contracts: 284/284 passed - Pixel model-transition tests: 14/14 passed - Python compile andgit diff --check: passed The predecessor run later recorded a Windows-to-WSL transport failure while invoking the unrelated Aider extension request, so it is diagnostic rather than Product Green. A fresh exact21613597install remains required for release acceptance.Exact predecessor live result and paired harness hardening
The terminal exact predecessor laptop run at product
82ff7f5dd702aa76db790d99f49254d206818ff6and harness02bdcdd6531b879849afa519f0955d8fce4d01bcproves the product-side custom gateway-port repair end to end. The isolated install and full Qwen 3.5 9B promotion passed. Dashboard activation returned HTTP 200, all configured consumers reported successful reconciliation, and the response explicitly recordedpixel: reconciled. The immediately following Pixel phase passed authentication enforcement, exact Qwen runtime identity, verified sandboxed access mode on the installed custom port, nonce-bound non-retained SSE, retained replay with hash equality, and active cancellation.That run is still diagnostic red. After the product proof, native Windows-to-WSL execution degraded to
Wsl/Service/0x8007274c; Aider/API UI, broad capabilities, and seven regression fixtures then failed through the same transport boundary. Those are not evidence that the seven fixed product bugs returned. Paired harness PR #39 now advances tod268d3ab2f1a14bac693691528075c45548ea5e7, which stops later mutations after transport loss, never retries mutating WSL requests, collapses the cascade into one environment signature, preserves regression evidence through triage, and triages before reporting.Current ODS head
851563f6066f05c85eb83ff72e3d29789db29701is fully GitHub-CI green. Its explicit-empty/quoted/boundary port hardening has passed the 309-test dashboard module, 284 Pixel host-install contracts, and 14 model-transition tests on Tower2. A fresh exact21613597/d7b7a01install remains required before acceptance; neither predecessor result nor source CI is a fleet-green claim.Remote installer custody and host-agent classification (851563f / d268d3a)
The first exact
21613597/d7b7a01laptop attempt was not a valid product result. The staged WSL installer kept running after stop cleanup refused its identity: the old cleanup required the run token inps command=, while large WSL payloads execute as a stagedbash <path>whose argv does not contain that token. A second run then began uninstall at 23:08:38Z and killed/removed the first run's healthyods-host-agent.serviceat 23:08:41Z while the first install was still finishing Pixel access reproof.ODS commit
851563f6066f05c85eb83ff72e3d29789db29701keeps that overlap attribution separate from product behavior and hardens the observed edge.SystemdAccessBridge.discover()now checks the host-agent unit's load state, active root process, positive PID, and process-read race before touching/proc. Missing, stopped, malformed, empty-cmdline, or raced root agents return the stable fail-closedhost-agent-unavailablereason; root execution still must prove isolated-Iexecution and root code custody.Paired harness commit
d268d3ab2f1a14bac693691528075c45548ea5e7adds target-side install exclusion and exact staged-process custody. It holds a targetflockfor the destructive run, validates PID start time, PGID, and command hash rather than searching argv for the token, refuses a second install before native cleanup/product mutation, retains markers when termination is unproven, and refuses to release the orchestrator lock after incomplete remote cleanup.Clean Tower2 exact-pair evidence:
GitHub CI is running. No installed host is counted green at this new pair; the next valid result must be a non-overlapping fresh laptop reinstall with both Pixel license variables explicitly set to
true.Candidate-forward reinstall and dashboard draft preservation (9e98a52)
The current exact ODS candidate is
9e98a52a0898319a66656fbde50d017c3f3b4f66, paired with fleet harness579b124d94b3435f25848c9a494fc3c391540f97.Commit
126a461ccfd8603330dc0d1c787e4cd0170868b4fixes the Ubuntu-reproduced Remote Provider form race: a delayed status response can no longer overwrite an in-progress user draft. The previously failing focused test passed 100/100 repetitions on Tower2, and a clean exact-base checkout with the patch passed all 898 dashboard tests.Commit
9e98a52a0898319a66656fbde50d017c3f3b4f66adds candidate-forward forced reinstall. The bootstrap clones and verifies the requested source before it invokes that candidate's uninstaller against a fingerprinted existing install. It refuses unsafe, symlinked, root-level, and bootstrap-root targets; a failed candidate uninstall prevents source overlay. This specifically targets the current laptop state where the installed predecessor uninstaller cannot recover a marker-bound interrupted Pixel release, while the newer candidate can. The clean Tower2 installer-hardening suite passes end to end, including exact-SHA, candidate-uninstaller, failure, symlink, and root-target contracts.GitHub CI on this exact head is running. This is source/candidate evidence only, not installed acceptance. The next decisive proof is candidate-forward recovery and a fresh exact-pair release run on the authorized disposable laptop, followed by the other four named hosts.
Real-uninstaller forward-recovery fidelity (5da6a15)
The first live candidate-forward laptop attempt at predecessor
9e98a52afailed safely before replacement. The bootstrap cloned and selected the requested candidate, but the realods-uninstall.shauto-detected its temporary candidate directory and overrode the target environment accepted by the earlier contract double. Pixel then correctly refused because the old install marker did not bind the temporary checkout. The existing install was not overlaid.Commit
5da6a15664a7163c3d1c11d03e3a0ab8f4ae9045replaces that implicit interface with explicit--install-dir. The candidate uninstaller independently canonicalizes and fingerprints the requested ODS tree, rejecting relative paths, symlinks, filesystem root, the user home, its own source directory, and missing or symlinked ODS/compose fingerprints before mutation. The bootstrap and behavioral double now use the same argument interface.Clean Tower2 verification passes: installer-hardening contracts, Pixel uninstall 68/68, uninstall/compose safety,
bash -n, andgit diff --check. GitHub CI is running. A fresh exact5da6a156live recovery is required before the laptop can start full release qualification; the predecessor failure is diagnostic evidence only.Non-interactive candidate-uninstall hang follow-up (34a7939)
The first live candidate-forward recovery at product
5da6a156, harness579b124d, and Pixelbbd1d2d6reached the requested candidate uninstaller and removed the interrupted Pixel release, then hung indefinitely at an interactivesudo -vwhile the bootstrap itself was running with--non-interactive --force. The captured process tree was the candidateods-uninstall.sh --install-dir /home/odsbeta/ods --forcewaiting on itssudo -vchild. The run was stopped, the exact leftover processes were retired, and the target install lock is free. No candidate overlay occurred; the prior ODS tree remains, while the interrupted Pixel state is absent.Commit
34a7939f3263d4b7414818fe611fff30d5c825a0makes the candidate bootstrap propagate--non-interactiveto the uninstaller. The uninstaller now validates privilege withsudo -n -vbefore Pixel, container, or install-tree mutation, fails promptly with a recovery instruction when no cached/passwordless credential exists, and returns nonzero if complete install-directory cleanup was not achieved. Interactive uninstall keeps its terminal-attachedsudo -vbehavior.Clean exact-head Tower2 evidence at
34a7939f:A live exact-head rerun on the laptop test distro exited through the intended failure path in 4 seconds (not the 120-second timeout), emitted the cached/passwordless-sudo recovery instruction, and preserved the SHA-256 fingerprints of
.env,ods-cli,ods-uninstall.sh, anddocker-compose.base.yml; Pixel marker/current remained absent before and after. All current-head GitHub checks are terminal green: 33 succeeded, 3 intentionally skipped, 0 pending, and 0 failed. This is a live-derived fail-fast repair, not installed acceptance. A fresh exact-pair run remains required after the laptop test target has a valid noninteractive privilege path.Installed laptop Pixel and model-promotion follow-up (481cacf)
A fresh disposable laptop installation completed at ODS 34a7939, harness 579b124d94b3435f25848c9a494fc3c391540f97, and Pixel bbd1d2d62c7260f822ba1e727728a0a02f78895f with both Pixel license acceptance variables set true. Install, core verification, dashboard, cloud mode, full-model capabilities, lifecycle, and all 22 installed regression fixtures passed. Six zero-prerequisite lanes, the 10-distribution Docker matrix, and the five-distribution Incus matrix also passed.
The first full-model promotion reached a healthy gateway but one strict Pixel verification reported that pixel-operations-broker was not loaded and correctly rolled back. A later supported lifecycle retry succeeded. On the stabilized Qwen 3.5 9B runtime, the exact Pixel harness passed authentication, schema rejection, runtime identity, verified sandbox access mode, nonce-bound non-retained and retained SSE, exact replay hash equality, and active cancellation. Six immediate restart/health/plugin-inspect cycles then found the expected broker version and active release root every time.
The independent model-management run reproduced the product impact through the real UI: Granite 4.0 H-Micro downloaded, the Run action issued POST /api/models/granite4.0-h-micro-q4/load, and the product returned HTTP 502 with ODS-managed Pixel model reconciliation failed; rollback could not be proved. Harness recovery restored the Qwen 9B baseline and deleted the harness-owned test model.
Commit 481cacf retries only the same complete strict Pixel verification, at most three times with two-second settling intervals. Persistent failure still aborts and invokes the existing rollback path. Clean Tower2 validation: 286 Pixel installer tests passed, Bash syntax and diff checks passed. An independent Tower1 read-only review found no actionable correctness, security, timeout, side-effect, or fail-closed issue.
This is a live-derived repair, not a five-host-green claim. A fresh exact-head installation and model-management rerun at 481cacf remains required.
Hermes activation health-window follow-up (02a2e9b)
The exact predecessor laptop run at ODS
481cacf56f06ffcd60852261bcb92d1b31a2a6d1and harnessd47a1a3bb08820c38640d43f69c61707b61ee696passed destructive reinstall, installed verification, cloud-mode contracts, dashboard, Hermes, UI, the complete Pixel phase, broad chat/search/files/code/skills/Talk/context/grounding capabilities, lifecycle, and all 22 regression fixtures before the model loop.Model cycle 1 activated Granite H-Micro and passed Talk, every required switched-model consumer, and both retained and non-retained Pixel SSE. Restoring Qwen then returned an observed HTTP 502 because
ods-hermeswas still in Docker'sstartingstate when the generic 120-second health budget expired. The product journal showed the gateway became ready shortly afterward, and cycle 3 completed the same switch/restore/cleanup path successfully. This is timing-sensitive installed evidence, not a consistently broken route.Commit
02a2e9bb47d1a558de796cbe8a29a50d9acfec69gives only Hermes a 90-attempt/180-second bounded health window during model activation. Other containers retain the 60-attempt default, explicit overrides still win, explicit unhealthy/invalid/exited states still fail immediately, and persistentstartingstill fails closed at the new bound.Exact clean Tower2 verification:
313 passed, 1 skippedfortest_model_activate.py. Independent Tower1 Qwen3.6 no-think review and Codex diff review found no actionable defect. The paired harness head isafb1cb00b02943e0b7dd19076355db61fc45eeab, which separately makes the observed 502 fail immediately with preserved response evidence. Fresh exact-pair installed qualification is required; neither PR claims fleet green.Fresh exact-pair patched laptop rerun
A fresh destructive laptop qualification started at 2026-09-13T04:40:51Z against exact clean ODS
02a2e9bb47d1a558de796cbe8a29a50d9acfec69, exact clean harnessabebafe2af292f074bcb7164fbc39346f3043a85, and immutable Pixelbbd1d2d62c7260f822ba1e727728a0a02f78895f. BothPIXEL_LICENSE_ACCEPTED=trueandDREAM_FLEET_PIXEL_LICENSE_ACCEPTED=trueare explicit. The predecessor red run released the heavy lock; only the disposableODS-Public-Beta-TestWSL distro was restarted after its service enteredWsl/Service/0x8007274c.Run directory:
/home/michael/.local/state/dream-fleet/tasks/ods-public-beta-fleet-green-20260911/exact-02a2e9bb-abebafe2/release/20260913T044051Z-windows-laptop. Zero-prerequisite distro lanes are progressing green. Installed Pixel/model-switch acceptance remains pending.Exact installed full-model evidence and paired lifecycle fix
The exact laptop run at this ODS head and harness predecessor
abebafe2af292f074bcb7164fbc39346f3043a85completed a clean install, installed Pixel reproving, core verification, and cloud-mode contracts. Its dashboard phase then failed only after a fixed 360-by-five-second observation budget expired while the authenticated lifecycle response still reported a healthy active Qwen 3.5 9B download. The full 5,680,522,464-byte model subsequently completed and promoted successfully.On the installed full model, ODS
02a2e9bb47d1a558de796cbe8a29a50d9acfec69passed Hermes and the complete Pixel phase: required authentication, exact Qwen 3.5 9B runtime identity, verified sandboxed access with no busy/pending state, nonce-bound non-retained SSE, retained completion and byte-identical replay hash, and real active cancellation. Broad chat, search, files, code, skills, Dream Talk status/pool/SSE/context/grounding, model identity, and strict 90-second runtime probes passed, as did all 22 regression fixtures.This is strong installed product evidence but not acceptance: the target-blind harness timeout made dashboard and lifecycle red, UI Aider transition also remained red, and the six-cycle model matrix was correctly blocked. Paired harness PR #39 commit
827159afaa46d6abfd021ff0df309fe67fb9cd50now gives only the two disposable Windows WSL lanes their declared four-hour lifecycle observation budget while preserving every fail-closed terminal, transport, exhaustion, and no-follow-on-mutation rule. Its focused behavior, complete Tower2 suite, Windows-native contract, and all four GitHub jobs are green.A fresh exact
02a2e9bb/827159aflaptop release run is active with both Pixel license acknowledgements set totrue; the matching Strixy run is queued behind it. No ODS code changed for this harness defect, and no five-host green claim is made.Talk disconnect abort propagation (fb69ff0)
The exact
02a2e9bb/49065336Strixy release run passed zero-prerequisite bootstrap, install, verify, cloud-mode contracts, dashboard/model management, Hermes agentic, Playwright UI, the complete Pixel phase, and lifecycle. Pixel proved authenticated full-model runtime identity, verified sandboxed access, nonce-bound non-retained SSE, retained byte-identical replay, and observed-active cancellation.Broad capabilities proved runtime performance, chat, search, files, code, skills, exact model identity, Talk status frames, pooled-session reuse, and persona/context. Talk SSE and grounding then failed because the first Talk client disconnected while Hermes continued the detached weather-agent turn for more than twelve minutes. The installed Hermes log recorded
detached_sessions=1; the abandoned turn made three local model calls, ran two web searches, and occupied the single CPU llama slot long enough for two 900-second idle gates to expire.Commit
fb69ff0bsends Hermes's documented, session-scopedsession.interruptabort before closing and evicting the pooled Talk connection. It preserves the existing approval denial ordering, bounds all cleanup writes/closes, never opens a new connection or interrupts an idle session, and prevents late frames from an abandoned turn contaminating a later request.Verification:
git diff --checksession.interruptis the gateway'sabortequivalent and clears only the named sessionThe predecessor Strixy run is diagnostic red and does not count as acceptance. A fresh exact product/harness pair must prove a deliberately aborted Talk stream releases the llama slot promptly, then complete Pixel, broad capabilities, lifecycle, and all six model cycles.
Terminal c9 WSL diagnostic and live model-reload follow-up
The terminal predecessor run at ODS
c9d0b48b4287a8d62f41bfbfdecd88005e9e24ea, harnessd809ebf16d2c6bf6fdbb67ff5291abbbdbc8b742, and Pixelbbd1d2d62c7260f822ba1e727728a0a02f78895fis diagnostic only.65a0516now refreshes inventory after the lifecycle-idle gate and selects the current loadable model. The complete clean Tower2 suite and all four GitHub jobs are green at that harness head.000failures coincided with nativeWsl/Service/0x8007274c; direct same-distro commands reproduced that transport failure while later semantic Pixel checks passed. These results are environment evidence, not a Pixel product regression.After a targeted WSL test-distro restart restored command execution, a live harness-
65a0516replay correctly selectedqwen3.5-9b-q4, proving the stale-bootstrap fix. The same-model/loadthen returned HTTP 502 because the root Pixel transition controller reportedruntime-proof-requiredafter the WSL/gateway restart, and rollback could not be proved. Host-agent journal evidence shows the target model itself became ready before Pixel reconciliation failed. This was a current-candidate product defect at96b5ca04; the follow-up fix and validation are recorded below. No predecessor run is promoted to acceptance.Stale Pixel runtime reproof under model hold (0642ad0)
A targeted live replay after laptop WSL recovery proved a model lifecycle deadlock at predecessor
96b5ca04: Qwen 3.5 9B became ready, but both forward Pixel reconciliation and rollback were rejected because the gateway restart had invalidatedverified.jsonand the root controller reported the exact fail-closed stateavailable=true,configured_mode=sandboxed,effective_mode=unknown,runtime_verified=false,busy=false,pending=false,reason=runtime-proof-required.Commit
0642ad018f5c91c861a1666011045f705ab1400elets a model transaction admit only that exact owner-host stale-proof projection. The root controller journals first, closes and drains the external and native admission gates, then re-proves the configured access mode while both gates remain held. Ambiguous, unavailable, busy, pending, invalid-revision, or mismatched states still fail closed. A failed proof retains the error journal and both holds; a successful transition still re-proves the post-change route before releasing native admission and then edge admission.Verification before push:
runtime_verified=truewith no pending transition; the full authenticated dashboard replay then selected Qwen 9B and changed the former HTTP 502 into HTTP 200 withpixel=reconciled, while Aider remained installedThe installed replay used a narrowly patched predecessor installation and is diagnostic proof of the fix, not exact-head fleet acceptance. A fresh exact
0642ad01/ harness65a0516/ Pixelbbd1d2d6release run is required.