Skip to content

Fix public-beta local reinstall blockers - #4215

Open
Lightheartdevs wants to merge 83 commits into
public-betafrom
fix/public-beta-local-reinstall
Open

Fix public-beta local reinstall blockers#4215
Lightheartdevs wants to merge 83 commits into
public-betafrom
fix/public-beta-local-reinstall

Conversation

@Lightheartdevs

@Lightheartdevs Lightheartdevs commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Summary This integration PR collects ODS product fixes found by repeatedly installing and exercising public-beta on the Dream Fleet. It remains based on public-beta, not main, and is not a fleet-green claim. ## Exact candidate

  • ODS candidate: 0642ad018f5c91c861a1666011045f705ab1400e
  • current public-beta base: d49ad4bbc760be319de7f4c9b9a93b1f18fe3922
  • relationship: the current base is an ancestor of the candidate (candidate is 75 commits ahead, 0 behind)
  • fleet harness: Lightheartdevs/DS-Fleet-Test#39 at 65a0516c92b5c0339f63df7e92b6059b14e2747f
  • Pixel source used by the authorized fleet installs: bbd1d2d62c7260f822ba1e727728a0a02f78895f

Product fixes accumulated here - reconcile optional-service state during reinstall without pruning user-selected services - recover stale or recycled Pixel admission state while retaining strict source, ownership, and catalog-custody checks - support isolated native-Docker WSL Pixel installs with validated gateway/preview ports - repair Windows/WSL bootstrap, model-download, host-agent, Hermes-config, model-promotion, and restart behavior - surface and safely answer authenticated Dream Talk approvals - pull pinned MinIO images from Quay after the Docker Hub paths stopped resolving - keep model-router activation and terminal failure reporting fail closed - allow the GPU-independent Aider CLI extension to resolve on supported CPU-only installs - bind the privileged Pixel access bridge to the root-recorded gateway port instead of guessing the default when owner config omits a custom port - re-run the existing owner-authenticated access proof ceremony after ODS-controlled gateway restarts, while refusing busy, pending, ambiguous, or differently configured states The latest product deltas address a live Strixy/laptop Pixel control-plane failure. ODS now persists the validated Pixel gateway port in the root-only coordinator configuration and performs a narrowly admitted same-mode proof through the existing access controller after an ODS-owned gateway restart. It never writes a verified state directly; the existing controller still acquires both admission gates, checks the configured service boundary, executes its runtime probe, and records proof atomically. Any unavailable, busy, pending, unknown-mode, stale-revision, or otherwise ambiguous state remains fail closed. ## Current verification At predecessor 35d7c697: - all GitHub checks are green, including Linux integration, dashboard API/frontends, Pixel inference contracts, environment-schema tiers, PowerShell lint, and distro smoke - Aider dashboard config regression: 60 passed - extension manifest validation: 33/33 library manifests and 30 compatible core manifests, with zero errors - the predecessor dashboard model-activation module: 298 passed, 1 skipped; dedicated Hermes restart regression passed At predecessor product 00c25411, all GitHub checks are green, including integration-smoke; focused Linux verification is also green: 280/280 Pixel host-install contract checks, 16/16 access-bridge tests, and 5/5 post-restart recovery tests. The live fresh-install failure below demonstrates why those source checks were necessary but insufficient. At predecessor harness 5cb52c1, the Pixel/model-switch contract passes on both the laptop checkout and a clean detached Tower2 checkout. It now requires available=true, runtime_verified=true, matching configured/effective modes, and both busy=false and pending=false before chat evidence can count. Harness #40 merged at 0303687; the follow-up gate is now carried by the still-open integration PR #39. All GitHub checks on that predecessor are green. Current harness 2a694d adds target-scoped llama-slot quiescence for the Strixy CPU lane while retaining the strict 90-second runtime usability gate; its new GitHub checks are running. The predecessor cc4100af portability candidate passes the full Docker multi-distro matrix (10/10 lanes) and the clean Incus VM matrix on Ubuntu 24.04, Fedora 42, Rocky 9, Arch, and openSUSE Tumbleweed. Because the product head advanced, those live matrices are predecessor evidence, not exact-head acceptance. The terminal predecessor laptop run (f71d14b3 / 420f9b5) proved clean install, Product Green, Pixel Green, Capability Green, and Lifecycle Green. Model UI remained red: - cycles 1, 3, and 4 passed - cycle 2 returned an observed 502 when Hermes became unhealthy during Qwen restoration; cleanup restored Qwen and deleted the test model - cycle 5 stopped before UI work when ods restart left Hermes unhealthy - cycle 6 proved Pixel, Open WebUI, OpenCode, Perplexica, and route evidence, but a lost harness credential read caused an unauthenticated LiteLLM 401; direct installed-system probes proved unauthenticated 401 and authenticated 200 The Hermes failures are addressed by the latest product commit. The activation-response and credential-attribution defects are addressed in harness #40. No predecessor result is counted as acceptance. The first exact cc4100af paired-WSL attempt stopped before product installation because of two harness preflight defects: a lost laptop WSL privilege probe was mislabeled as missing sudo, and Strixy's absent Docker Desktop engine was treated as a fatal inventory error. Both are fixed and covered at harness ad4f934; that failed attempt makes no product claim. A replacement exact cc4100af / ad4f934 diagnostic run installed successfully on both isolated WSL lanes. The laptop passed UI, Pixel, broad capabilities, and lifecycle. Strixy exposed three fail-closed results: the CPU-only no-GPU telemetry response was treated as a generic 503; Aider was accepted by the dashboard but excluded by its CPU-incompatible manifest; and Pixel chat was blocked after model reconciliation restarted the gateway. The first two defects were fixed at 0303687/35d7c697. The third is now root-caused: the restart invalidated the live access proof and left the native runtime interrupted; the custom laptop port also proved the bridge could target an unused default port. Both product defects are fixed at 00c25411, and the harness now detects this class before counting chat semantics. Strixy's CPU-bound broad capability probe remains actively progressing, so this predecessor run is diagnostic only. The clean exact 00c25411 / 5cb52c1 paired-WSL run is terminal red: the laptop lost its WSL transport before install, and Strixy reached Pixel installation but failed the final access proof. It is diagnostic evidence only and cannot validate the newer product or harness heads. ### Latest exact-pair diagnostic (f50ac59 / e73f1c55; superseded)

The exact predecessor laptop-and-Strixy WSL run installed both targets from ODS f50ac591fe31af1c15677a1648acd59389463697, harness e73f1c55, and Pixel bbd1d2d6. It is diagnostic red and is not acceptance evidence.

  • Strixy passed installation, installed verification, cloud-mode contracts, dashboard/model management, Hermes, Playwright UI, authenticated Pixel status, verified sandboxed access, retained and non-retained replay/hash checks, cancellation, and the broad chat/search/files/code/skills/model-identity/Talk/SSE/pool/context/grounding matrix. Its only red was Talk abort. The exact installed replay proved this was a harness routing defect: the abort curl sent the hostname without the required Host: prefix, so Caddy returned the hostless HTTP 200 catch-all and the POST never reached Talk. Harness docs: improve international accessibility of README #39 now sends the detected host header correctly; the corrected installed replay observed busy state, a real session frame, client abort, and return to idle.
  • The laptop downloaded and verified Phi-4-mini but llama-server restart-looped because the selected container profile exceeded the WSL VM's actual available RAM after KV-cache demand. Verified rollback to Qwen 2B succeeded and Pixel was correctly deferred. This candidate now bases runtime-profile eligibility and persisted system RAM on actual VM RAM, while retaining reserved RAM only for coarse tier sizing. Focused selector/profile tests are green; installed proof remains required.
  • Paired harness d809ebf1 also stops immediately when a terminal failed bootstrap has no live updater, reports post-install bootstrap deferral honestly, and carries complete regression fixtures. Its full clean Tower2 suite and GitHub checks are green.

A fresh exact c9d0b48b / d809ebf1 run is required; no predecessor result is promoted to fleet acceptance.

Ubuntu 26.04 physical-fleet qualification (96b5ca0)

Live preflight established that Tower1 and Tower3 run Ubuntu 26.04 with kernel 7.0.0-29, Python 3.14.4, systemd 259, and Docker 29.7.2; Tower2 remains Ubuntu 24.04. The prior exact allowlist rejected both 26.04 hosts before Pixel installation even though the Docker ubuntu:26.04 manifest and Incus images:ubuntu/26.04 VM image are available.

This commit admits only exact Ubuntu 26.04 while retaining Linux, PID1 systemd, WSL2, license, model-route, source-custody, and all other fail-closed gates. It adds native and WSL2 26.04 qualification regressions plus CI Docker and private Incus lanes, and updates the support/install documentation.

Evidence before push:

  • local and clean Tower2 qualification suites: 59/59 passed
  • actual Tower3 Ubuntu 26.04 host: 59/59 passed, live default-path host qualification passed, and all installer Python sources compiled under Python 3.14.4
  • Docker and Incus Ubuntu 26.04 image discovery passed
  • shell syntax and git diff --check passed

This is source and host-preflight evidence only. Installed Pixel qualification on Tower1/Tower3 still requires the declared noninteractive system privilege; both currently return sudo -n failure. The active c9d0b48 WSL run is now predecessor diagnostic evidence because this product head advanced, and a fresh exact-head five-host aggregate remains required.

Still required before fleet acceptance - a fresh exact-head release run on the laptop and Strixy through Pixel-first post-install checks and six restart-separated model cycles - Tower2, Tower1, and Tower3 installs after their required noninteractive Pixel system privilege is provisioned - one exact candidate/harness evidence set covering all five physical hosts with no missing, skipped, stale, or red required receipts - human review; this integration PR is intentionally not merged by the qualification loop The five named test installations have explicit Pixel-license authorization. Fleet invocations pass both PIXEL_LICENSE_ACCEPTED=true and DREAM_FLEET_PIXEL_LICENSE_ACCEPTED=true explicitly; the harness neither infers acceptance nor applies Pixel to targets that do not declare Pixel support. ### Fresh-install access-edge ordering follow-up The exact 00c25411 / 5cb52c1 Strixy run exposed a new installation-order defect. The root-protected access coordinator correctly failed closed with available=false, configured_mode=unknown, and reason=host-command-failed. Direct inspection showed ods-pixel-edge did not exist yet. The coordinator's proof ceremony requires that container's durable transition gate, but the installer was calling the final proof before the later whole-stack Compose launch. On the same failed disposable install, starting only the exact pixel-edge Compose service and re-running the protected reconciliation helper immediately returned {"result":"reproved","mode":"sandboxed"}. This isolates the cause: the proof code was correct; its required edge service was started too late. Commit 53e2795c5990708294633f2b6c7465f0f52cf1f6 adds pixel-edge to the bounded Pixel prerequisite launch alongside LiteLLM and dashboard-api. The transition endpoint can start before host-ingress chat readiness; the final proof still occurs only after ingress health, plugin loading, and the ready marker. A contract assertion fixes that ordering. Verification so far: - bash -n ods/installers/lib/pixel-host-install.sh - git diff --check - live failed-install reproduction and exact one-service recovery on Strixy - the local focused suite could not complete because this laptop's WSL service is intermittently returning Wsl/Service/0x8007274c; GitHub CI and a clean Tower2 checkout are the next test authorities At current product 53e2795c, all GitHub checks are green, including Linux integration, dashboard, Pixel inference contracts, PowerShell lint, the distro matrix, and integration-smoke. A clean Tower2 exact checkout passed 280/280 Pixel host-install contracts and all five access-recovery tests. Exact 53e2795c / 283cc5b laptop-and-Strixy release qualification is now live with both Pixel license variables explicitly set to true. ### Fresh access-coordinator activation-race follow-up The exact 53e2795c / 283cc5b rerun proved ods-pixel-edge present, running, and healthy, but a fresh Strixy install still failed the mandatory final access proof. Post-state was the narrowly safe runtime-proof-required projection: available owner-host coordinator, sandboxed configured mode, no pending transition, no busy runtime, valid revision, and no runtime proof. Re-running the protected same-mode change returned HTTP 200 with effective_mode=sandboxed and runtime_verified=true. The remaining defect was activation order: systemctl restart ods-pixel-access.service is asynchronous, while the installer called its protected client immediately. Commit 334e8961fb9fbf03402171407b146f6eebc1cd09 gives only the mandatory fresh-install proof a bounded 30-attempt, one-second retry window. The helper remains fail closed on busy, pending, ambiguous, stale-revision, mismatched, or differently configured state; no verified state is written directly. Tower2 exact-head evidence is 284/284 Pixel host-install checks plus 5/5 access-reconcile tests. The next exact 334e8961 / 283cc5b run made Strixy installation green, then passed installed verification, cloud-mode contracts, dashboard/model-management, Hermes agentic, and Playwright UI. Pixel authentication, runtime identity, verified sandboxed access mode, and active cancellation passed. Its two chat streams reached HTTP 200 but outlived the harness fixed 300-second cap on the CPU target, so this is diagnostic rather than Pixel Green. Harness #39 at f12dc0a now applies Strixy''s 5x behavioral budget and an idle-slot gate to Pixel chats. The laptop remains independently blocked before install by its WSL transport failure (exit 82). ### Exact-head Pixel bootstrap-promotion diagnosis (7ad5d61) The terminal laptop release run at predecessor product 334e8961 / harness f12dc0a passed install, verification, cloud-mode contracts, dashboard/model-management, Hermes, Playwright UI, full-model capability reprobe, lifecycle, and all 22 regression fixtures. Pixel alone failed during the initial bootstrap window, and the harness correctly blocked model-UI acceptance. Retained SQLite bytes and the system journal isolate the failure. Pixel entered an OpenClaw provider fetch on the bootstrap Qwen 2B route; background bootstrap promotion then changed the route to Qwen 9B and restarted the gateway while that turn was active. Pixel Edge emitted the sanitized terminal stream {"error":"upstream error"} followed by [DONE]. The dashboard retained-result producer incorrectly treated [DONE] as success and marked the result complete. A stable reprobe after Qwen 9B was fully ready passed authenticated status/access, nonce-bound non-retained chat, retained replay/hash equality, and active cancellation. Commit 7ad5d617634b2c52c9ebf9849b59d3a17c1b61ef fixes the false-success state: retained SSE containing an error frame remains replayable but is terminal interrupted, never complete. Exact clean Tower2 qualification at this head passed: - 284/284 Pixel host-install contract checks - 82/82 dashboard Pixel/access/retained-result tests in a clean test virtualenv - all current GitHub checks green, including Linux integration, dashboard, Pixel inference, lint, schema, and the distro smoke matrix This is not yet a fleet-green claim. The patch correctly reports the interrupted turn but does not by itself prevent background model promotion from overlapping an active whole Pixel turn. A durable transition exclusion spanning route mutation and gateway/ingress reconciliation remains required, followed by a fresh exact-head installed run. ### Paired harness observation hardening (939ead8) The live predecessor laptop loop completed the cycle-2 Granite H-Tiny UI, model, Talk, Pixel and broad-consumer work, restored Qwen, deleted the test model, and reached a clean final inventory before the native WSL service became unavailable. The old paired harness suppressed Wsl/Service/0x8007274c and falsely labeled the later router observation as a container identity change. Paired harness commit 939ead84a6f21263faede3b80e44e868fc42871a now preserves the host execution failure as observationStatus=unavailable, keeps the cycle red and fail closed, and reserves missing for a successful observation with no container. Its complete clean Tower2 suite and all four GitHub jobs are green. This is evidence-attribution hardening only; it does not make the predecessor installed run exact-head or fleet green. ### Active-turn-safe model promotion (3f3a3d0) The predecessor laptop evidence showed a real race: bootstrap model promotion changed the routed model and restarted the Pixel gateway while a Pixel turn was still active. Reporting the resulting SSE error as interrupted fixed false success, but did not prevent the interruption. Commit 3f3a3d0347fab9b9aaa86b9f719cf3dfb8474283 adds a root-custodied model-promotion transaction. Before the first route mutation it records durable intent, acquires and drains both native and externally reachable Pixel admission gates, and holds them across route mutation, gateway restart, ingress checks, and the final authenticated runtime/configuration proof. Rollback restores the exact begin-time route while the gates remain held. Release is native first and edge last, so any partial-release failure keeps external admission closed. A bounded root-only completion receipt makes an exact finish replayable after a lost Unix-socket response without exposing the admission token or accepting a different outcome. Exact-head source verification at 3f3a3d0347fab9b9aaa86b9f719cf3dfb8474283: - clean detached Tower2 checkout: 57/57 host qualification/integration checks and 284/284 Pixel host-install contracts - 13/13 model-transition tests and 17/17 access-bridge tests - complete Pixel agent Node suite: 1,044 passed, 1 skipped, 0 failed - git diff --check, installer bash -n, native-search, compose-wiring, timeout, and artifact-promoter suites all passed - independent Tower2 GLM-5.3-Flash and Tower3 Qwen3.6 no-think reviews found no remaining actionable defect after two earlier review findings were repaired All current-head GitHub checks are green, including integration-smoke, API/frontends, Pixel inference, PowerShell lint, environment-schema tiers, security review, and the ten-distro smoke matrix. This remains source-level evidence, not installed fleet acceptance. A fresh exact 3f3a3d03 / harness fb1d8584 run must prove installation, Pixel chat retention/cancellation, repeated model switching, broad consumers, restoration, cleanup, and restart-separated stability on all five named physical hosts. ### Paired clean-checkout harness unblock (e923a6f) The first exact-pair laptop attempt at ODS 3f3a3d03 and harness 939ead84 stopped in two seconds before installation because the clean harness checkout required interactive sudo for Playwright OS dependencies on Tower2. Harness PR #39 commit e923a6f2e305869785820ef272232d9cdd685d69 adds the already-established browser-only fallback and a behavioral regression test. The complete Tower2 harness suite passes. This was a harness preparation defect, not an ODS result; exact installed qualification is restarting against the updated pair. Harness follow-up fb1d85846f17e83db56f8724a3479770d74eecfa moves the Playwright coordination lock outside the Git checkout so subsequent install runs remain clean-attested. The dirty preflight was stopped and makes no product claim. ### Isolated model-promotion client repair (5e0d740) The first clean bundle-backed laptop install at product 3f3a3d03, harness fb1d8584, and Pixel bbd1d2d6 completed installation and post-install verification, but the downloaded and SHA-verified Qwen 3.5 9B promotion rolled back to the 2B bootstrap model. The root-owned promotion log isolated the failure: python3 -I /usr/local/libexec/ods-pixel-access/pixel_model_transition.py could not import its sibling pixel_access_client. Python isolated mode excludes the script directory from sys.path; the transition helper had not implemented the root-custody-checked runtime import already used by the reconciliation helper. Commit 5e0d7404933d7df7962ccbdf7728ff0039cd83d5 validates the installed helper directory, all parents, and the sibling client as root-owned, nonsymlinked, and not group/other writable before narrowly adding that protected directory to the isolated interpreter path. Dependency injection remains available for unit tests; the privileged client is loaded only at runtime. Exact clean Tower2 source evidence at this head: - 14/14 model-transition tests - 284/284 Pixel host-install contract checks - full harness suite green at paired harness 02bdcdd6531b879849afa519f0955d8fce4d01bc This directly repairs the observed public-beta promotion rollback, but it is not yet installed acceptance. The next required result is a fresh exact-pair laptop install proving bootstrap status complete, active/model-router/Pixel runtime identity on Qwen 3.5 9B, then Pixel chat retention, cancellation, model switching, broad capabilities, cleanup, and lifecycle. The other four physical hosts remain required for fleet green. ### Owner-ready access socket activation (c41c6ed) The exact 5e0d7404 / 02bdcdd bundle-backed laptop reinstall made the isolated transition import succeed, then exposed the next activation race. Both the 9B promotion and verified rollback invoked the owner client immediately after systemctl restart ods-pixel-access.service; the simple service process had spawned, but its Unix socket was still in the root-only bind window, so both attempts failed with PermissionError: [Errno 13]. The installer itself returned green while bootstrap-status.json correctly recorded failed promotion, proving again that install exit alone is not beta acceptance. Commit c41c6ede6d4f29033dba66a54558ec0574157779 makes access-service installation wait at most 30 seconds for the exact root directory mode, root-owned socket, admitted owner group, mode 0660, and a live read-only protocol response. The response check prevents a correctly owned but stale socket inode from satisfying readiness. It does not retry model-begin, whose lost-response semantics must remain fail closed. Clean Tower2 evidence is 14/14 model-transition tests, bash -n, and 284/284 Pixel host-install contract checks. A fresh exact install remains required to prove full-model promotion and Pixel runtime behavior. ### Preserve the installed Pixel gateway port during model activation (82ff7f5) The exact c41c6ede / 02bdcdd laptop reinstall is terminal and diagnostic. It installed successfully, completed bootstrap promotion to the SHA-verified Qwen 3.5 9B model, passed installed verification, cloud contracts, Hermes, Playwright UI, all broad capability probes, and all 22 replayed regressions. A standalone Pixel phase before model switching proved authenticated runtime identity, verified sandboxed access, nonce-bound non-retained chat, retained replay/hash equality, and active cancellation. Dashboard model activation then returned HTTP 502, and the later Pixel phase failed only the protected access-mode status. The installed runtime remained safely on Qwen 3.5 9B. Host-agent and system journals isolated the failure: the installation selected PIXEL_GATEWAY_PORT=18790, but _reconcile_ods_managed_pixel_model constructed a minimal child environment without that value. The sourced shell reconciliation therefore fell back to 18789 and rewrote /etc/ods/pixel-access.json to the wrong native gateway port; both activation and rollback correctly failed closed as runtime-unavailable-or-busy. Commit 82ff7f5dd702aa76db790d99f49254d206818ff6 loads the root-installed .env value, validates it as a canonical TCP port before any privileged subprocess runs, and passes it through the existing minimal environment. Legacy installs with no explicit port retain the 18789 default. Tests cover the live custom-port case, the default, and malformed/zero/out-of-range inputs. Clean detached Tower2 evidence at this head: - dashboard model-activation module: 304 passed, 1 skipped - Pixel host-install contracts: 284/284 passed - Pixel model-transition tests: 14/14 passed - Python compile and git diff --check: passed Current-head GitHub checks are running. This is still not installed acceptance: a fresh exact-head laptop reinstall must prove the same dashboard switch plus a second Pixel access/chat/cancel pass, the six-cycle browser model matrix, lifecycle, and cleanup. Strixy and the three tower installs remain required for fleet green. ### Fail closed on an explicitly empty Pixel gateway port (2161359) The exact predecessor 82ff7f5d / 02bdcdd laptop run supplied the missing installed proof for the custom-port fix. Dashboard model activation returned HTTP 200 activated with Pixel reconciled, and the subsequent Pixel phase passed authenticated runtime identity, available=true, runtime_verified=true, matching sandboxed configured/effective modes, nonce-bound non-retained chat, retained replay/hash equality, and active cancellation. The installed Qwen 3.5 9B runtime stayed healthy. An independent Tower2 GLM review found no critical or exploitable defect, but identified one fail-closed edge: an explicitly present but empty PIXEL_GATEWAY_PORT= would have been treated like an absent key and defaulted to 18789. Commit 851563f6066f05c85eb83ff72e3d29789db29701 defaults only when the key is absent and rejects explicit empty/whitespace values before subprocess execution. Coverage also pins quoted valid input, port 65535, overlong numeric input, zero, leading-zero, negative, and nonnumeric cases. Clean detached Tower2 evidence at this head: - dashboard model-activation module: 309 passed, 1 skipped - focused managed-Pixel reconciliation cases: 14/14 passed - Pixel host-install contracts: 284/284 passed - Pixel model-transition tests: 14/14 passed - Python compile and git diff --check: passed The predecessor run later recorded a Windows-to-WSL transport failure while invoking the unrelated Aider extension request, so it is diagnostic rather than Product Green. A fresh exact 21613597 install remains required for release acceptance.

Exact predecessor live result and paired harness hardening

The terminal exact predecessor laptop run at product 82ff7f5dd702aa76db790d99f49254d206818ff6 and harness 02bdcdd6531b879849afa519f0955d8fce4d01bc proves the product-side custom gateway-port repair end to end. The isolated install and full Qwen 3.5 9B promotion passed. Dashboard activation returned HTTP 200, all configured consumers reported successful reconciliation, and the response explicitly recorded pixel: reconciled. The immediately following Pixel phase passed authentication enforcement, exact Qwen runtime identity, verified sandboxed access mode on the installed custom port, nonce-bound non-retained SSE, retained replay with hash equality, and active cancellation.

That run is still diagnostic red. After the product proof, native Windows-to-WSL execution degraded to Wsl/Service/0x8007274c; Aider/API UI, broad capabilities, and seven regression fixtures then failed through the same transport boundary. Those are not evidence that the seven fixed product bugs returned. Paired harness PR #39 now advances to d268d3ab2f1a14bac693691528075c45548ea5e7, which stops later mutations after transport loss, never retries mutating WSL requests, collapses the cascade into one environment signature, preserves regression evidence through triage, and triages before reporting.

Current ODS head 851563f6066f05c85eb83ff72e3d29789db29701 is fully GitHub-CI green. Its explicit-empty/quoted/boundary port hardening has passed the 309-test dashboard module, 284 Pixel host-install contracts, and 14 model-transition tests on Tower2. A fresh exact 21613597 / d7b7a01 install remains required before acceptance; neither predecessor result nor source CI is a fleet-green claim.

Remote installer custody and host-agent classification (851563f / d268d3a)

The first exact 21613597 / d7b7a01 laptop attempt was not a valid product result. The staged WSL installer kept running after stop cleanup refused its identity: the old cleanup required the run token in ps command=, while large WSL payloads execute as a staged bash <path> whose argv does not contain that token. A second run then began uninstall at 23:08:38Z and killed/removed the first run's healthy ods-host-agent.service at 23:08:41Z while the first install was still finishing Pixel access reproof.

ODS commit 851563f6066f05c85eb83ff72e3d29789db29701 keeps that overlap attribution separate from product behavior and hardens the observed edge. SystemdAccessBridge.discover() now checks the host-agent unit's load state, active root process, positive PID, and process-read race before touching /proc. Missing, stopped, malformed, empty-cmdline, or raced root agents return the stable fail-closed host-agent-unavailable reason; root execution still must prove isolated -I execution and root code custody.

Paired harness commit d268d3ab2f1a14bac693691528075c45548ea5e7 adds target-side install exclusion and exact staged-process custody. It holds a target flock for the destructive run, validates PID start time, PGID, and command hash rather than searching argv for the token, refuses a second install before native cleanup/product mutation, retains markers when termination is unproven, and refuses to release the orchestrator lock after incomplete remote cleanup.

Clean Tower2 exact-pair evidence:

  • ODS access bridge: 23/23
  • ODS model transition: 14/14
  • ODS Pixel host-install contracts: 284/284
  • dashboard model tests: 105/105 in the maintained test environment
  • complete DS-Fleet-Test portable suite: passed
  • both checkouts clean at the exact commits above

GitHub CI is running. No installed host is counted green at this new pair; the next valid result must be a non-overlapping fresh laptop reinstall with both Pixel license variables explicitly set to true.

Candidate-forward reinstall and dashboard draft preservation (9e98a52)

The current exact ODS candidate is 9e98a52a0898319a66656fbde50d017c3f3b4f66, paired with fleet harness 579b124d94b3435f25848c9a494fc3c391540f97.

Commit 126a461ccfd8603330dc0d1c787e4cd0170868b4 fixes the Ubuntu-reproduced Remote Provider form race: a delayed status response can no longer overwrite an in-progress user draft. The previously failing focused test passed 100/100 repetitions on Tower2, and a clean exact-base checkout with the patch passed all 898 dashboard tests.

Commit 9e98a52a0898319a66656fbde50d017c3f3b4f66 adds candidate-forward forced reinstall. The bootstrap clones and verifies the requested source before it invokes that candidate's uninstaller against a fingerprinted existing install. It refuses unsafe, symlinked, root-level, and bootstrap-root targets; a failed candidate uninstall prevents source overlay. This specifically targets the current laptop state where the installed predecessor uninstaller cannot recover a marker-bound interrupted Pixel release, while the newer candidate can. The clean Tower2 installer-hardening suite passes end to end, including exact-SHA, candidate-uninstaller, failure, symlink, and root-target contracts.

GitHub CI on this exact head is running. This is source/candidate evidence only, not installed acceptance. The next decisive proof is candidate-forward recovery and a fresh exact-pair release run on the authorized disposable laptop, followed by the other four named hosts.

Real-uninstaller forward-recovery fidelity (5da6a15)

The first live candidate-forward laptop attempt at predecessor 9e98a52a failed safely before replacement. The bootstrap cloned and selected the requested candidate, but the real ods-uninstall.sh auto-detected its temporary candidate directory and overrode the target environment accepted by the earlier contract double. Pixel then correctly refused because the old install marker did not bind the temporary checkout. The existing install was not overlaid.

Commit 5da6a15664a7163c3d1c11d03e3a0ab8f4ae9045 replaces that implicit interface with explicit --install-dir. The candidate uninstaller independently canonicalizes and fingerprints the requested ODS tree, rejecting relative paths, symlinks, filesystem root, the user home, its own source directory, and missing or symlinked ODS/compose fingerprints before mutation. The bootstrap and behavioral double now use the same argument interface.

Clean Tower2 verification passes: installer-hardening contracts, Pixel uninstall 68/68, uninstall/compose safety, bash -n, and git diff --check. GitHub CI is running. A fresh exact 5da6a156 live recovery is required before the laptop can start full release qualification; the predecessor failure is diagnostic evidence only.

Non-interactive candidate-uninstall hang follow-up (34a7939)

The first live candidate-forward recovery at product 5da6a156, harness 579b124d, and Pixel bbd1d2d6 reached the requested candidate uninstaller and removed the interrupted Pixel release, then hung indefinitely at an interactive sudo -v while the bootstrap itself was running with --non-interactive --force. The captured process tree was the candidate ods-uninstall.sh --install-dir /home/odsbeta/ods --force waiting on its sudo -v child. The run was stopped, the exact leftover processes were retired, and the target install lock is free. No candidate overlay occurred; the prior ODS tree remains, while the interrupted Pixel state is absent.

Commit 34a7939f3263d4b7414818fe611fff30d5c825a0 makes the candidate bootstrap propagate --non-interactive to the uninstaller. The uninstaller now validates privilege with sudo -n -v before Pixel, container, or install-tree mutation, fails promptly with a recovery instruction when no cached/passwordless credential exists, and returns nonzero if complete install-directory cleanup was not achieved. Interactive uninstall keeps its terminal-attached sudo -v behavior.

Clean exact-head Tower2 evidence at 34a7939f:

  • installer-hardening contract passed
  • uninstall compose/safety regression passed, including a five-second no-hang and pre-mutation assertion
  • Pixel uninstall suite: 68/68 passed
  • Bash syntax and clean-checkout checks passed

A live exact-head rerun on the laptop test distro exited through the intended failure path in 4 seconds (not the 120-second timeout), emitted the cached/passwordless-sudo recovery instruction, and preserved the SHA-256 fingerprints of .env, ods-cli, ods-uninstall.sh, and docker-compose.base.yml; Pixel marker/current remained absent before and after. All current-head GitHub checks are terminal green: 33 succeeded, 3 intentionally skipped, 0 pending, and 0 failed. This is a live-derived fail-fast repair, not installed acceptance. A fresh exact-pair run remains required after the laptop test target has a valid noninteractive privilege path.

Installed laptop Pixel and model-promotion follow-up (481cacf)

A fresh disposable laptop installation completed at ODS 34a7939, harness 579b124d94b3435f25848c9a494fc3c391540f97, and Pixel bbd1d2d62c7260f822ba1e727728a0a02f78895f with both Pixel license acceptance variables set true. Install, core verification, dashboard, cloud mode, full-model capabilities, lifecycle, and all 22 installed regression fixtures passed. Six zero-prerequisite lanes, the 10-distribution Docker matrix, and the five-distribution Incus matrix also passed.

The first full-model promotion reached a healthy gateway but one strict Pixel verification reported that pixel-operations-broker was not loaded and correctly rolled back. A later supported lifecycle retry succeeded. On the stabilized Qwen 3.5 9B runtime, the exact Pixel harness passed authentication, schema rejection, runtime identity, verified sandbox access mode, nonce-bound non-retained and retained SSE, exact replay hash equality, and active cancellation. Six immediate restart/health/plugin-inspect cycles then found the expected broker version and active release root every time.

The independent model-management run reproduced the product impact through the real UI: Granite 4.0 H-Micro downloaded, the Run action issued POST /api/models/granite4.0-h-micro-q4/load, and the product returned HTTP 502 with ODS-managed Pixel model reconciliation failed; rollback could not be proved. Harness recovery restored the Qwen 9B baseline and deleted the harness-owned test model.

Commit 481cacf retries only the same complete strict Pixel verification, at most three times with two-second settling intervals. Persistent failure still aborts and invokes the existing rollback path. Clean Tower2 validation: 286 Pixel installer tests passed, Bash syntax and diff checks passed. An independent Tower1 read-only review found no actionable correctness, security, timeout, side-effect, or fail-closed issue.

This is a live-derived repair, not a five-host-green claim. A fresh exact-head installation and model-management rerun at 481cacf remains required.

Hermes activation health-window follow-up (02a2e9b)

The exact predecessor laptop run at ODS 481cacf56f06ffcd60852261bcb92d1b31a2a6d1 and harness d47a1a3bb08820c38640d43f69c61707b61ee696 passed destructive reinstall, installed verification, cloud-mode contracts, dashboard, Hermes, UI, the complete Pixel phase, broad chat/search/files/code/skills/Talk/context/grounding capabilities, lifecycle, and all 22 regression fixtures before the model loop.

Model cycle 1 activated Granite H-Micro and passed Talk, every required switched-model consumer, and both retained and non-retained Pixel SSE. Restoring Qwen then returned an observed HTTP 502 because ods-hermes was still in Docker's starting state when the generic 120-second health budget expired. The product journal showed the gateway became ready shortly afterward, and cycle 3 completed the same switch/restore/cleanup path successfully. This is timing-sensitive installed evidence, not a consistently broken route.

Commit 02a2e9bb47d1a558de796cbe8a29a50d9acfec69 gives only Hermes a 90-attempt/180-second bounded health window during model activation. Other containers retain the 60-attempt default, explicit overrides still win, explicit unhealthy/invalid/exited states still fail immediately, and persistent starting still fails closed at the new bound.

Exact clean Tower2 verification: 313 passed, 1 skipped for test_model_activate.py. Independent Tower1 Qwen3.6 no-think review and Codex diff review found no actionable defect. The paired harness head is afb1cb00b02943e0b7dd19076355db61fc45eeab, which separately makes the observed 502 fail immediately with preserved response evidence. Fresh exact-pair installed qualification is required; neither PR claims fleet green.

Fresh exact-pair patched laptop rerun

A fresh destructive laptop qualification started at 2026-09-13T04:40:51Z against exact clean ODS 02a2e9bb47d1a558de796cbe8a29a50d9acfec69, exact clean harness abebafe2af292f074bcb7164fbc39346f3043a85, and immutable Pixel bbd1d2d62c7260f822ba1e727728a0a02f78895f. Both PIXEL_LICENSE_ACCEPTED=true and DREAM_FLEET_PIXEL_LICENSE_ACCEPTED=true are explicit. The predecessor red run released the heavy lock; only the disposable ODS-Public-Beta-Test WSL distro was restarted after its service entered Wsl/Service/0x8007274c.

Run directory: /home/michael/.local/state/dream-fleet/tasks/ods-public-beta-fleet-green-20260911/exact-02a2e9bb-abebafe2/release/20260913T044051Z-windows-laptop. Zero-prerequisite distro lanes are progressing green. Installed Pixel/model-switch acceptance remains pending.

Exact installed full-model evidence and paired lifecycle fix

The exact laptop run at this ODS head and harness predecessor abebafe2af292f074bcb7164fbc39346f3043a85 completed a clean install, installed Pixel reproving, core verification, and cloud-mode contracts. Its dashboard phase then failed only after a fixed 360-by-five-second observation budget expired while the authenticated lifecycle response still reported a healthy active Qwen 3.5 9B download. The full 5,680,522,464-byte model subsequently completed and promoted successfully.

On the installed full model, ODS 02a2e9bb47d1a558de796cbe8a29a50d9acfec69 passed Hermes and the complete Pixel phase: required authentication, exact Qwen 3.5 9B runtime identity, verified sandboxed access with no busy/pending state, nonce-bound non-retained SSE, retained completion and byte-identical replay hash, and real active cancellation. Broad chat, search, files, code, skills, Dream Talk status/pool/SSE/context/grounding, model identity, and strict 90-second runtime probes passed, as did all 22 regression fixtures.

This is strong installed product evidence but not acceptance: the target-blind harness timeout made dashboard and lifecycle red, UI Aider transition also remained red, and the six-cycle model matrix was correctly blocked. Paired harness PR #39 commit 827159afaa46d6abfd021ff0df309fe67fb9cd50 now gives only the two disposable Windows WSL lanes their declared four-hour lifecycle observation budget while preserving every fail-closed terminal, transport, exhaustion, and no-follow-on-mutation rule. Its focused behavior, complete Tower2 suite, Windows-native contract, and all four GitHub jobs are green.

A fresh exact 02a2e9bb / 827159af laptop release run is active with both Pixel license acknowledgements set to true; the matching Strixy run is queued behind it. No ODS code changed for this harness defect, and no five-host green claim is made.

Talk disconnect abort propagation (fb69ff0)

The exact 02a2e9bb / 49065336 Strixy release run passed zero-prerequisite bootstrap, install, verify, cloud-mode contracts, dashboard/model management, Hermes agentic, Playwright UI, the complete Pixel phase, and lifecycle. Pixel proved authenticated full-model runtime identity, verified sandboxed access, nonce-bound non-retained SSE, retained byte-identical replay, and observed-active cancellation.

Broad capabilities proved runtime performance, chat, search, files, code, skills, exact model identity, Talk status frames, pooled-session reuse, and persona/context. Talk SSE and grounding then failed because the first Talk client disconnected while Hermes continued the detached weather-agent turn for more than twelve minutes. The installed Hermes log recorded detached_sessions=1; the abandoned turn made three local model calls, ran two web searches, and occupied the single CPU llama slot long enough for two 900-second idle gates to expire.

Commit fb69ff0b sends Hermes's documented, session-scoped session.interrupt abort before closing and evicting the pooled Talk connection. It preserves the existing approval denial ordering, bounds all cleanup writes/closes, never opens a new connection or interrupts an idle session, and prevents late frames from an abandoned turn contaminating a later request.

Verification:

  • full dashboard Talk suite: 46 passed
  • Python compile and git diff --check
  • independent Tower3 Qwen3.6-27B Q4 no-think review found no blocker after checking lock ordering, session scope, pool eviction, bounded cleanup, and regression coverage
  • live source inspection of the pinned Hermes build confirmed session.interrupt is the gateway's abort equivalent and clears only the named session

The predecessor Strixy run is diagnostic red and does not count as acceptance. A fresh exact product/harness pair must prove a deliberately aborted Talk stream releases the llama slot promptly, then complete Pixel, broad capabilities, lifecycle, and all six model cycles.

Terminal c9 WSL diagnostic and live model-reload follow-up

The terminal predecessor run at ODS c9d0b48b4287a8d62f41bfbfdecd88005e9e24ea, harness d809ebf16d2c6bf6fdbb67ff5291abbbdbc8b742, and Pixel bbd1d2d62c7260f822ba1e727728a0a02f78895f is diagnostic only.

  • Strixy passed clean installation, verification, cloud-mode contracts, dashboard/model management, Hermes, Playwright UI, authenticated Pixel status, verified sandboxed access, nonce-bound retained and non-retained SSE, byte-identical replay, active cancellation, the broad chat/search/files/code/skills/model-identity/Talk/SSE/pool/context/grounding matrix, and lifecycle.
  • The laptop passed installation, verification, cloud mode, and UI, and completed the 5,680,522,464-byte Qwen 3.5 9B promotion. Its old dashboard probe reused the deleted Qwen 2B bootstrap id after promotion; harness docs: improve international accessibility of README #39 65a0516 now refreshes inventory after the lifecycle-idle gate and selects the current loadable model. The complete clean Tower2 suite and all four GitHub jobs are green at that harness head.
  • The laptop Hermes and three early Pixel HTTP 000 failures coincided with native Wsl/Service/0x8007274c; direct same-distro commands reproduced that transport failure while later semantic Pixel checks passed. These results are environment evidence, not a Pixel product regression.

After a targeted WSL test-distro restart restored command execution, a live harness-65a0516 replay correctly selected qwen3.5-9b-q4, proving the stale-bootstrap fix. The same-model /load then returned HTTP 502 because the root Pixel transition controller reported runtime-proof-required after the WSL/gateway restart, and rollback could not be proved. Host-agent journal evidence shows the target model itself became ready before Pixel reconciliation failed. This was a current-candidate product defect at 96b5ca04; the follow-up fix and validation are recorded below. No predecessor run is promoted to acceptance.

Stale Pixel runtime reproof under model hold (0642ad0)

A targeted live replay after laptop WSL recovery proved a model lifecycle deadlock at predecessor 96b5ca04: Qwen 3.5 9B became ready, but both forward Pixel reconciliation and rollback were rejected because the gateway restart had invalidated verified.json and the root controller reported the exact fail-closed state available=true, configured_mode=sandboxed, effective_mode=unknown, runtime_verified=false, busy=false, pending=false, reason=runtime-proof-required.

Commit 0642ad018f5c91c861a1666011045f705ab1400e lets a model transaction admit only that exact owner-host stale-proof projection. The root controller journals first, closes and drains the external and native admission gates, then re-proves the configured access mode while both gates remain held. Ambiguous, unavailable, busy, pending, invalid-revision, or mismatched states still fail closed. A failed proof retains the error journal and both holds; a successful transition still re-proves the post-change route before releasing native admission and then edge admission.

Verification before push:

  • local: 18/18 model-transition tests, 5/5 access-reconcile tests, and 286/286 Pixel host-install contracts
  • clean detached Tower2 checkout: the same focused suites pass; complete Pixel source suite passes, including 59/59 Pixel integration checks, 286/286 host-install contracts, and the Node agent suite with 1,067 passed, 1 intentionally skipped, 0 failed
  • disposable installed laptop: the exact stale-proof projection was reproduced; a real begin/finish recovery returned runtime_verified=true with no pending transition; the full authenticated dashboard replay then selected Qwen 9B and changed the former HTTP 502 into HTTP 200 with pixel=reconciled, while Aider remained installed

The installed replay used a narrowly patched predecessor installation and is diagnostic proof of the fix, not exact-head fleet acceptance. A fresh exact 0642ad01 / harness 65a0516 / Pixel bbd1d2d6 release run is required.

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/phases/03-features.sh

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/phases/03-features.sh

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/phases/03-features.sh

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/phases/03-features.sh
ods/installers/windows/install-windows.ps1

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/phases/03-features.sh
ods/installers/windows/install-windows.ps1
ods/installers/windows/phases/06-directories.ps1

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/phases/03-features.sh
ods/installers/windows/install-windows.ps1
ods/installers/windows/phases/06-directories.ps1

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/phases/03-features.sh
ods/installers/windows/install-windows.ps1
ods/installers/windows/phases/06-directories.ps1

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/phases/03-features.sh
ods/installers/windows/install-windows.ps1
ods/installers/windows/lib/ui.ps1
ods/installers/windows/phases/06-directories.ps1

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/phases/03-features.sh
ods/installers/windows/install-windows.ps1
ods/installers/windows/lib/ui.ps1
ods/installers/windows/phases/06-directories.ps1

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/phases/03-features.sh
ods/installers/windows/install-windows.ps1
ods/installers/windows/lib/ui.ps1
ods/installers/windows/phases/06-directories.ps1

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/phases/03-features.sh
ods/installers/windows/install-windows.ps1
ods/installers/windows/lib/ui.ps1
ods/installers/windows/phases/06-directories.ps1

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/phases/03-features.sh
ods/installers/windows/install-windows.ps1
ods/installers/windows/lib/ui.ps1
ods/installers/windows/phases/06-directories.ps1

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/phases/03-features.sh
ods/installers/phases/05-docker.sh
ods/installers/windows/install-windows.ps1
ods/installers/windows/lib/ui.ps1

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/phases/03-features.sh
ods/installers/phases/05-docker.sh
ods/installers/windows/install-windows.ps1
ods/installers/windows/lib/ui.ps1

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/lib/ui.sh
ods/installers/lib/wsl-memory.sh
ods/installers/macos/install-macos.sh
ods/installers/macos/lib/env-generator.sh

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

…ic-beta-local-reinstall

# Conflicts:
#	ods/tests/pixel_inference/test_advice_setup_process.py
@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/lib/ui.sh
ods/installers/lib/wsl-memory.sh
ods/installers/macos/install-macos.sh
ods/installers/macos/lib/env-generator.sh

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@Lightheartdevs

Lightheartdevs commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator Author

Fleet checkpoint after the terminal paired-WSL run and a current-public-beta forward merge.

Terminal frozen run: ODS 8cd5a887 / harness 6d96d3e1

  • zero-prerequisite bootstrap passed 6/6 lanes
  • destructive install and lifecycle passed on both laptop and Strixy
  • Strixy passed verify, cloud contracts, dashboard/model management, Hermes, Playwright UI, the complete Pixel phase, and every broad capability except talk_abort
  • Strixy independently reproved chat, search, files, code, 64 skills, exact Qwen3.5-9B identity, Talk status/SSE/session reuse/context/grounding, Pixel authentication, retained byte-identical replay, and active Pixel cancellation
  • the Talk-abort probe twice returned an immediate empty HTTP 200 without observing busy; a manual long-running SSE replay did enter busy and client disconnect reached Hermes
  • live Hermes logged Interrupted during API call plus message.complete status=interrupted at 20:34:05Z. Timestamp correlation showed the earlier diagnostic had overlapping requests, so its delay could not safely identify a failing component.
  • a follow-up started from three sustained idle samples and correlated exactly one Talk session (7e3d3fa1) to one llama task (8683). Killing the target-local client produced Hermes interruption plus explicit llama cancel/release, and the slot returned idle after 1,746 ms. The installed ODS cancellation path is therefore working; the remaining automated talk_abort red is a harness attribution race, now patched in Lightheartdevs/DS-Fleet-Test#39 at e73f1c55.
  • the laptop passed install, core verify, cloud contracts, and lifecycle, then its Windows-to-WSL transport failed with Wsl/Service/0x8007274c; later dashboard/Hermes/UI/Pixel/capability symptoms share that transport loss and are not being treated as independent product bugs
  • the six-cycle model matrix correctly remained blocked. This run is diagnostic red and is not a fleet-green claim.

New WSL model-memory safety work

9feb78eb budgets model selection against WSL VM memory rather than Windows host RAM, prevents hardware-profile models rejected by RAM gates from falling through to unprofiled activation/recommendation, and emits a fail-closed Pixel model-readiness signal. a56f57df pins Windows RAM in a dashboard inventory test so inventory remains visible while unsafe recommendation remains excluded.

Exact a56f57df CI completed with 33 successful checks, zero failures, and zero pending checks. Focused local verification: 314 passed/1 skipped model-activation tests, 54 performance-oracle tests, all 64 selector combinations plus 15 grouped assertions, 14 safe-env tests, Python compile, and git diff --check.

Public beta then advanced 225 commits beyond the prior integrated tip. Commit f50ac591 merges current public-beta d49ad4bb; the only conflict was a duplicated explanatory comment in test_advice_setup_process.py, resolved in favor of the newer community wording with behavior unchanged. The focused suites above remain green after the merge. Exact f50ac591 GitHub CI is now the next gate.

Remaining sequence: complete exact-head CI for harness e73f1c55, freeze one new exact product/harness pair, rerun both WSL hosts through Pixel, broad capabilities, lifecycle, and six model cycles, then execute the three native tower lanes. No merge is requested or authorized by this qualification loop.

@Lightheartdevs

Copy link
Copy Markdown
Collaborator Author

Follow-up: exact merged head f50ac591 has now completed all 33 GitHub checks successfully, with zero failures or pending jobs. The disposable laptop WSL lane was also recovered with a targeted wsl --terminate ODS-Public-Beta-Test restart; a fresh direct command now succeeds and the installed dashboard route responds again (authenticated route correctly returned HTTP 401 without a key). This is environment recovery, not installed acceptance; the next release run must reinstall and qualify the new exact head.

@Lightheartdevs

Copy link
Copy Markdown
Collaborator Author

Fresh exact-pair qualification is live: product f50ac591fe31af1c15677a1648acd59389463697, harness e73f1c55fe16e5bc9f420089e16ca38d74c21b36, Pixel bbd1d2d62c7260f822ba1e727728a0a02f78895f. Both PIXEL_LICENSE_ACCEPTED=true and DREAM_FLEET_PIXEL_LICENSE_ACCEPTED=true are explicit. All six zero-prerequisite distro lanes passed, both physical WSL preflights passed (laptop 15.3 GB VM RAM; Strixy 46.9 GB), and destructive installs are now running in parallel under the unified host lock. Run evidence: /home/michael/.local/state/dream-fleet/tasks/ods-public-beta-fleet-green-20260911/exact-f50ac591-e73f1c55/release/20260913T2126Z-two-wsl. No installed acceptance is claimed until the run is terminal.

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/lib/ui.sh
ods/installers/lib/wsl-memory.sh
ods/installers/macos/install-macos.sh
ods/installers/macos/lib/env-generator.sh

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/lib/pixel-integration.sh
ods/installers/lib/ui.sh
ods/installers/lib/wsl-memory.sh
ods/installers/macos/install-macos.sh

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/lib/pixel-integration.sh
ods/installers/lib/ui.sh
ods/installers/lib/wsl-memory.sh
ods/installers/macos/install-macos.sh

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/lib/pixel-integration.sh
ods/installers/lib/ui.sh
ods/installers/lib/wsl-memory.sh
ods/installers/macos/install-macos.sh

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/lib/pixel-integration.sh
ods/installers/lib/ui.sh
ods/installers/lib/wsl-memory.sh
ods/installers/macos/install-macos.sh

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@Lightheartdevs

Copy link
Copy Markdown
Collaborator Author

Tower3 exact-head canary found and this branch now fixes a real Ubuntu 26.04 Pixel bootstrap failure.

  • Exact failing ODS head: b35e41f0ee2b0216cb0fbc78c06f761f5ce73219
  • Host: Tower3, Ubuntu 26.04
  • Failure: unsafe Pixel Operations executable: /usr/bin/uname
  • Root cause: Ubuntu 26.04 supplies /usr/bin/uname as a root-owned symlink to /usr/lib/cargo/bin/coreutils/uname; the policy generator hard-coded and then rejected the symlink instead of validating the resolved executable.
  • Fix: 7e80b2cf648a3cab71d3ffeaae97dd964d4355dd resolves named commands before applying the existing regular-file, root-owner, non-writable, executable checks.
  • Focused regression: ods/tests/test-pixel-host-install.sh — 286 passed, 0 failed locally and again from the clean exact Tower2 checkout.
  • GitHub's Ubuntu 26.04 distribution smoke is green on the new head; the full five-host fleet is not yet green.

Tower3 was restored to its exact Qwen workload after the failed disposable install, and a new exact-head canary is running with this commit.

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/lib/pixel-integration.sh
ods/installers/lib/ui.sh
ods/installers/lib/wsl-memory.sh
ods/installers/macos/install-macos.sh

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@Lightheartdevs

Copy link
Copy Markdown
Collaborator Author

Exact fleet candidate advanced to b4b359dda1091de1ebce83a4b148e661c3e8de45.

Tower3's real Ubuntu 26.04 install reached Pixel Operations Broker activation and exposed a Pixel access-probe defect: Rust coreutils test -r ignored a valid effective POSIX ACL even though the target owner could read the inventory. Pixel PR #240 fixes all production broker access probes at exact head 5983c27edb1c41d6e944abd13b6e6f780dd6cb4c; this ODS commit pins that repaired source in the client, default install phase, example, and contract test.

Evidence: the isolated Tower3 bridge reproduction showed external test false, actual read true, Bash read true, and Bash write denied; a clean Tower2 exact checkout passed tests/test-pixel-host-install.sh (286/286). Tower3 release canary revision 4 is now running with ODS b4b359dd, Pixel 5983c27, and harness eabb064. This is not yet a fleet-green claim.

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/lib/pixel-integration.sh
ods/installers/lib/ui.sh
ods/installers/lib/wsl-memory.sh
ods/installers/macos/install-macos.sh

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@Lightheartdevs

Copy link
Copy Markdown
Collaborator Author

Fleet update — exact candidate 942607145f99c61d003a93ae0f9dad9080a9109d

  • Re-pins Pixel to evidence-bound PR head 1cb3a8a7cb58fd467fe7d0ce37057208724d6ddb; exact clean Tower2 ods/tests/test-pixel-host-install.sh passes 286/286.
  • Tower3 functional canary at the preceding heads completed the full ODS image build and reached Pixel apply/verify. It then exposed a qualification-environment collision: the preserved trusted fleet Pixel gateway already owned loopback 18789, while the test target had not supplied ODS's supported PIXEL_GATEWAY_PORT override.
  • The run failed closed before product launch; the original Qwen3.6 workload, exact image and Docker networks were restored healthy, and the temporary Docker qualification bridge was absent afterward.
  • DS-Fleet-Test PR docs: improve international accessibility of README #39 now assigns physical tower qualification Pixels to 18790. A new exact Tower3 canary is running with this ODS head, harness 38d6769c6ebd431c1352627d6dc21b094fdae1cd, and Pixel 1cb3a8a7.

No fleet-green claim yet. The strict five-machine chain remains gated on Tower3 release confidence, installed Pixel journeys, model switching, distro lanes, and restored pre-state.

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/lib/pixel-integration.sh
ods/installers/lib/ui.sh
ods/installers/lib/wsl-memory.sh
ods/installers/macos/install-macos.sh

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@Lightheartdevs

Copy link
Copy Markdown
Collaborator Author

Fleet update — exact candidate fe4e53b

Tower3 retry run r6 reproduced an interrupted ODS-managed Pixel installation: the exact release, active link, candidate sandbox image, OpenClaw configuration, and gateway unit existed, but runtime attestation had not yet been written. ODS correctly detected the state, then failed closed because uninstall required fields that are only persisted after successful verification.

This commit adds a narrowly bounded recovery path:

  • Only the exact original six-field installing marker with one unattested active or staged release is eligible; partially enriched and mixed markers fail before mutation.
  • Release version, source identity, source commit and tree, full install manifest, deterministic candidate image identity, ownership, and reserved containers are verified under the deployment lock.
  • Unbound shared image tags are preserved. Unbound OpenClaw configuration is archived privately and checksum-verified rather than trusted as deletion authority.
  • The deactivating marker records the derived bindings atomically and supports bounded crash resumption.

Evidence on the clean exact Tower2 checkout matching the pushed diff:

  • ods/tests/test-pixel-uninstall.sh: 71 passed, 0 failed
  • ods/tests/test-pixel-host-install.sh: 286 passed, 0 failed
  • Bash syntax and git diff --check passed
  • ShellCheck found no production-script warning; test-suite SC2015 notices are the existing assertion idiom

The next acceptance step is an exact-head Tower3 retry against the untouched interrupted state, followed by Pixel capability and model-switch qualification. No fleet-green claim yet.

@github-actions

Copy link
Copy Markdown

Sensitive Files Detected

Trigger: Security-sensitive files detected: ods/installers/

Files flagged:

ods/installers/lib/pixel-host-install.sh
ods/installers/lib/pixel-integration.sh
ods/installers/lib/ui.sh
ods/installers/lib/wsl-memory.sh
ods/installers/macos/install-macos.sh

Extra human review is recommended for this PR.


Claude Code Review | Sensitive File Detection | ~$1.50

@Lightheartdevs

Lightheartdevs commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator Author

Tower3 retry update — exact candidate abfe807

Canary r7 safely reached the preserved interrupted Pixel state and exposed one narrower lifecycle boundary: the freshly cloned ODS tree already ships extension_manager.py and workspace_preview.py, while their generated owner-unit contracts are not created until a later apply step. The retry validator treated that source-only installing state as an incomplete deployed service.

This commit permits source-only lifecycle files only for the already bounded inactive-installing or exact minimal unattested-install recovery states. A generated unit without its source still fails closed, and ready or partially bound markers remain rejected.

Regression evidence from the clean Tower2 checkout:

  • ods/tests/test-pixel-uninstall.sh: 71 passed, 0 failed, including the active unattested source-only fixture
  • ods/tests/test-pixel-host-install.sh: 286 passed, 0 failed
  • Bash syntax and git diff --check passed
  • Tower3 protected Qwen workload and networks restored successfully after r7; the strict chain stopped before other hosts

An exact-head Tower3 r8 retry is next. No fleet-green claim yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant