Advice for how to debug startup issues

[humphd]

I'm trying to debug a new startup error we're seeing in a self-hosted GitHub Actions CI runner (Linux X64 ubuntu-22.04). This workflow has been running great for weeks (we've been loving the devcontainers/ci action!), but all of a sudden it refuses to work. The error is really opaque (to me):

🏃 start container
  About to run devcontainer up --workspace-folder /home/github/actions-runner/_work/DeepStructure/DeepStructure --remote-env GITHUB_OUTPUT=/mnt/github/output --remote-env GITHUB_ENV=/mnt/github/env --remote-env GITHUB_PATH=/mnt/github/path --remote-env GITHUB_STEP_SUMMARY=/mnt/github/step-summary --mount type=bind,source=/home/github/actions-runner/_work/_temp/_runner_file_commands/set_output_39dffcd4-c68e-4716-9ada-e377f11f5af8,target=/mnt/github/output --mount type=bind,source=/home/github/actions-runner/_work/_temp/_runner_file_commands/set_env_39dffcd4-c68e-4716-9ada-e377f11f5af8,target=/mnt/github/env --mount type=bind,source=/home/github/actions-runner/_work/_temp/_runner_file_commands/add_path_39dffcd4-c68e-4716-9ada-e377f11f5af8,target=/mnt/github/path --mount type=bind,source=/home/github/actions-runner/_work/_temp/_runner_file_commands/step_summary_39dffcd4-c68e-4716-9ada-e377f11f5af8,target=/mnt/github/step-summary
  [2023-11-02T00:48:54.924Z] @devcontainers/cli 0.52.1. Node.js v20.8.1. linux 5.15.0-1045-kvm x64.
  
  Error: Command failed: docker start 2252cc6b00073132e1ddd4ebfa2931bdc8f9e759d63446c07a4e1e7537ea6dfd
      at EAA (/home/github/actions-runner/_work/_tool/node/20.8.1/x64/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:462:1260)
      at kK (/home/github/actions-runner/_work/_tool/node/20.8.1/x64/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:462:1002)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async NAA (/home/github/actions-runner/_work/_tool/node/20.8.1/x64/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:479:3660)
      at async vC (/home/github/actions-runner/_work/_tool/node/20.8.1/x64/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:479:4775)
      at async etA (/home/github/actions-runner/_work/_tool/node/20.8.1/x64/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:613:10941)
      at async AtA (/home/github/actions-runner/_work/_tool/node/20.8.1/x64/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:613:10682)
  
  Error: Dev container up failed: Command failed: docker start 2252cc6b00073132e1ddd4ebfa2931bdc8f9e759d63446c07a4e1e7537ea6dfd (exit code: undefined)
  An error occurred setting up the container.
  Error: Command failed: docker start 2252cc6b00073132e1ddd4ebfa2931bdc8f9e759d63446c07a4e1e7537ea6dfd

The build stage that precedes this completes without issue, but for whatever reason it dies trying to start a container. I also know it isn't a disk space issue.

We can't reproduce locally, so it's something about this CI environment I guess. Any advice on how to dig into this deeper?

[humphd]

Looks like we die in https://github.com/devcontainers/ci/blob/main/github-action/src/main.ts#L151

[humphd]

This turned out to be a stuck devcontainer in CI that couldn't be restarted on any subsequent job. I removed it with docker in a shell script, and was able to get CI jobs passing again.

It would be great if there was a better way to make sure old containers were properly cleaned up in CI.

[derekdowling]

@humphd I'm hitting this issue as well. Any idea how to fix this?

[humphd]

We ended-up running a python script at the start of our CI job that does a bunch of docker cleanup (e.g., kill old containers, make sure there's enough disk space) so things work properly once the dev container needs to run.