[Ubuntu 24.04][Sandbox] dcode rebuild --yes cannot recover a sandbox stuck in Error with a live but unresponsive container

[hulynn]

Description

nemoclaw {sandbox} rebuild --yes cannot recover a Deep Agents Code (dcode) sandbox that is stuck in Phase: Error with a live-but-unresponsive container. Rebuild's inference-route revalidation step always probes inference by executing a command inside the target sandbox; when the sandbox's shell is itself broken, that probe can never succeed, so rebuild permanently refuses to run and the sandbox stays broken.

Platform scope: Reproduced on Ubuntu 24.04 only; other platforms not tested.
Regression: Unknown — earlier versions not tested.
OpenShell issue: No

Environment

Device:        Ubuntu 24.04 x86_64 workstation with NVIDIA GeForce RTX 5090
OS:            Ubuntu 24.04.4 LTS
Architecture:  x86_64
Node.js:       Not captured
npm:           Not captured
Docker:        server 29.5.2
OpenShell CLI: 0.0.106
NemoClaw:      v0.0.120
OpenClaw:      N/A (dcode/Deep Agents Code agent; harness reports LangChain Deep Agents Code v0.1.55, not OpenClaw)

Steps to Reproduce

1. Onboard a Deep Agents Code sandbox and confirm it reaches Ready:

   NEMOCLAW_PROVIDER=build NEMOCLAW_MODEL=nvidia/nemotron-3-super-120b-a12b nemoclaw onboard --name {sandbox} --agent dcode --non-interactive --yes --yes-i-accept-third-party-software

2. Capture the running container id:

   docker ps -q --filter 'label=openshell.ai/sandbox-name={sandbox}'

3. As root inside the container, corrupt the managed shell profile:

   docker exec --user 0 {container id} sh -c 'rm -f /sandbox/.bash_profile && ln -s /sandbox/hostile-env.sh /sandbox/.bash_profile'

4. nemoclaw {sandbox} stop (exits 0)
5. nemoclaw {sandbox} start (exits 1, correctly fails closed — sandbox enters Phase: Error, never reaches Ready)
6. nemoclaw {sandbox} rebuild --yes
7. Repeat step 6 once more, unmodified, to rule out a transient failure.

Expected Result

Step 6 exits 0, recreates the sandbox from a clean image, discards the corrupted profile, and restores a trusted-healthy Ready sandbox.

Actual Result

Step 6 exits 1 on both attempts, with identical output. The sandbox remains permanently stuck in Phase: Error. A follow-up nemoclaw {sandbox} connect --probe-only also fails: the container starts, cycles Status: Error for ~21s, briefly shows Provisioning, then returns to Phase: Error.

No documented bypass exists: rebuild --help's --force flag only skips the confirmation prompt, not this preflight check. The only working recovery is destroy + re-onboard, which discards the sandbox's existing state rather than recovering it in place.

Logs

nemoclaw {sandbox} start (step 5):
  Waiting for sandbox '{sandbox}' to be ready...
  Status: Error (repeated) then Status: Provisioning then ...
  Sandbox '{sandbox}' entered 'Error' state.
  Post-start status: Inference: not probed ("Could not probe https://inference.local/v1/models from inside the sandbox."); Agent version not verified.
  (exit code 1)

nemoclaw {sandbox} rebuild --yes (step 6, both attempts identical):
  Rebuild sandbox '{sandbox}'
    Current:  LangChain Deep Agents Code v0.1.55
    Target:   LangChain Deep Agents Code v0.1.55
  [preflight green: Docker daemon, CDI GPU, bridge containers, host DNS, container DNS, container runtime docker, openshell CLI 0.0.106]
  Rebuild preflight failed: recorded inference credentials or route were rejected: sandbox inference invocation probe was unavailable
  Sandbox is untouched — no data was lost.
  Recorded inference route smoke check failed
  (exit code 1)

nemoclaw {sandbox} connect --probe-only (follow-up):
  Sandbox '{sandbox}' container is stopped — starting it...
  Waiting for sandbox '{sandbox}' to be ready...
    Status: Error (repeated ~21s) then Status: Provisioning (24s) then
  Sandbox '{sandbox}' entered 'Error' state.
  Probe timing: result=failed failedStage=gateway readiness.firstFailedObservation=sandbox-identity
  (exit code 1)