[Bug]: Ryuk "Could not start container"

[donbeave]

Module

Core

Testcontainers version

1.17.6

Using the latest Testcontainers version?

Yes

Host OS

macOS Ventura 13.0.1 (22A400)

Host Arch

Apple M1 Max (ARM)

Docker version

Client:
 Cloud integration: v1.0.29
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:01:18 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.14.1 (91661)
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 17:59:41 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.9
  GitCommit:        1c90a442489720eec95342e1789ee8a5e1b9536f
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

Getting error when starting Ruyk container: No such container.

But without ruyk everything works, if I define TESTCONTAINERS_RYUK_DISABLED=true all containers (of course except ryuk) will be started successfully.

I can not say when this problem appeared the first time, as I didn't use testcontainers locally for a while, and after that I have installed from scratch the new macOS Ventura with a new Docker Desktop, so problem can be related to these upgrades. If you can help me know how to better check what's the root cause of this issue, I will provide more details.

Relevant log output

13:03:28.457 [testcontainers-lifecycle-0] INFO  o.t.d.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///Users/donbeave/.docker/run/docker.sock
13:03:28.459 [testcontainers-lifecycle-0] INFO  o.testcontainers.DockerClientFactory - Docker host IP address is localhost
13:03:28.486 [testcontainers-lifecycle-0] INFO  o.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 20.10.21
  API Version: 1.41
  Operating System: Docker Desktop
  Total Memory: 19987 MB
13:03:28.530 [testcontainers-lifecycle-0] INFO  🐳 [testcontainers/ryuk:0.3.4] - Pulling docker image: testcontainers/ryuk:0.3.4. Please be patient; this may take some time but only needs to be done once.
13:03:28.919 [testcontainers-lifecycle-0] INFO  o.t.utility.RegistryAuthLocator - Credential helper/store (docker-credential-desktop) does not have credentials for https://index.docker.io/v1/
13:03:32.174 [docker-java-stream-257369379] INFO  🐳 [testcontainers/ryuk:0.3.4] - Starting to pull image
13:03:32.208 [docker-java-stream-257369379] INFO  🐳 [testcontainers/ryuk:0.3.4] - Pulling image layers:  0 pending,  0 downloaded,  0 extracted, (0 bytes/0 bytes)
13:03:33.918 [docker-java-stream-257369379] INFO  🐳 [testcontainers/ryuk:0.3.4] - Pulling image layers:  2 pending,  1 downloaded,  0 extracted, (52 KB/? MB)
13:03:34.044 [docker-java-stream-257369379] INFO  🐳 [testcontainers/ryuk:0.3.4] - Pulling image layers:  1 pending,  2 downloaded,  0 extracted, (2 MB/? MB)
13:03:34.064 [docker-java-stream-257369379] INFO  🐳 [testcontainers/ryuk:0.3.4] - Pulling image layers:  0 pending,  3 downloaded,  0 extracted, (2 MB/4 MB)
13:03:34.132 [docker-java-stream-257369379] INFO  🐳 [testcontainers/ryuk:0.3.4] - Pulling image layers:  0 pending,  3 downloaded,  1 extracted, (3 MB/4 MB)
13:03:34.181 [docker-java-stream-257369379] INFO  🐳 [testcontainers/ryuk:0.3.4] - Pulling image layers:  0 pending,  3 downloaded,  2 extracted, (3 MB/4 MB)
13:03:34.239 [docker-java-stream-257369379] INFO  🐳 [testcontainers/ryuk:0.3.4] - Pulling image layers:  0 pending,  3 downloaded,  3 extracted, (4 MB/4 MB)
13:03:34.244 [docker-java-stream-257369379] INFO  🐳 [testcontainers/ryuk:0.3.4] - Pull complete. 3 layers, pulled in 2s (downloaded 4 MB at 2 MB/s)
13:03:34.284 [testcontainers-lifecycle-0] INFO  🐳 [testcontainers/ryuk:0.3.4] - Creating container for image: testcontainers/ryuk:0.3.4
13:03:34.386 [testcontainers-lifecycle-0] INFO  🐳 [testcontainers/ryuk:0.3.4] - Container testcontainers/ryuk:0.3.4 is starting: 10365f73545106c1fb464cb32841d20f3e41a5f6ab2a4faa5f54a89d0c40c12d
13:03:34.715 [testcontainers-lifecycle-0] ERROR 🐳 [testcontainers/ryuk:0.3.4] - Could not start container
com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such container: 10365f73545106c1fb464cb32841d20f3e41a5f6ab2a4faa5f54a89d0c40c12d"}

Additional Information

No response

[ubaid4j]

Hi donbeave,
I got the same error while having the following stack:

OS: Ubuntu 22.10 5.19.0-23-generic

Docker (Linux Native):
Client: Docker Engine - Community
 Cloud integration: v1.0.29
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:47:58 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:45:50 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker Desktop 4.14.1 (91661)
Engine: v20.10.21

ubaid:~$ docker context ls;
NAME                TYPE                DESCRIPTION                               DOCKER ENDPOINT                                  KUBERNETES ENDPOINT   ORCHESTRATOR
default *           moby                Current DOCKER_HOST based configuration   unix:///home/ubaid/.docker/desktop/docker.sock                         swarm
desktop-linux       moby                                                          unix:///home/ubaid/.docker/desktop/docker.sock                         

My understanding is that:
dockerd was listening to the old docker-end point. We need to add a new endpoint for dockerd to listen. As docker engine was already installed for Linux and then I installed Docker Desktop. It created a new context and a new endpoint so we make sure that our daemon (dockerd) is communicating with right docker context.

If you execute the following command:

sudo dockerd -H unix:///home/$USER/.docker/desktop/docker.sock

dockerd will start to listen to new endpoint unix:///home/$USER/.docker/desktop/docker.sock and that exception related to container not found will resolve as it will get the container from right docker context.

Summary

Just execute

sudo dockerd -H unix:///home/$USER/.docker/desktop/docker.sock

let me know if you need help.
Thanks

[marcellorinaldo]

I had the same issue but with the following environment (docker server on Jenkins):

DockerClientFactory - Connected to docker: 
  Server Version: 1.13.1
  API Version: 1.26
  Operating System: JBoss
  Total Memory: 7609 MB

Could not solve the issue with the above-proposed solution so I disabled Ryuk. Ryuk is working locally (macOS Intel) but not on my Jenkins CI. It might be because Ryuk requires superuser privileges.

FYI Ryuk can be disabled by setting the environment variable TESTCONTAINERS_RYUK_DISABLED to true. For the readers that, like me, have the issue on the CI, in a Jenkinsfile this can be accomplished with:

environment {
    TESTCONTAINERS_RYUK_DISABLED = 'true'
}

[donbeave]

Try to unset DOCKER_HOST environment if you set it.

In my case, it was because I set DOCKER_HOST environment in my .zshrc file when Docker released Docker Desktop 4.13.0. The env originally was required for me to fix the Docker detection in the Gradle docker plugin: bmuschko/gradle-docker-plugin#1109

I forget after that they release a hotfix where rollback changes to Docker Desktop.

With the current Docker Desktop 4.16.2 (95914) for macOS ARM I don't have any problem with this issue if I remove the explicit setting of DOCKER_HOST environment.

Hope this information can be useful and can fix your guys' issues as well.

[kiview]

Thanks for updating us on this @donbeave. I will close this issue, since especially the comments mention very different failure scenarios.

[ZachChuba]

Im running docker as a sidecar via gitlab services so unsetting the DOCKER_HOST variable is not going to work. Nor will dockerd be recognized. Does this really mean there is no solution to use ryuk?