[Bug]: Can not connect to Ryuk at 172.17.0.1:61798: Connection refused

[adamjshook]

Module

Core

Testcontainers version

1.17.3

Using the latest Testcontainers version?

Yes

Host OS

Mac OS X

Host Arch

x86_64

Docker version

Client:
 Cloud integration: 1.0.17
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:55:20 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:10 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

Testcontainers is unable to connect to Ryuk after it starts. I would expect it to be able to connect and launch the additional containers.

Note I am using running tests using Testcontainers inside a Docker container on Mac OS. The Docker container runs a Maven build to run the tests. This same image to run the tests is used on the CI server which uses Linux (don't know which flavor) and the build succeeds.

The command is essentially:

docker run -it --rm -u 0:0 -v /var/run/docker.sock:/var/run/docker.sock:ro -P -v ~/.m2/repository:/root/.m2/repository -v /Users/adamjshook/project:/Users/adamjshook/project -w /Users/adamjshook/project internal/ubuntu18-openjdk11-maven /bin/bash -c "mvn install"

Relevant log output

[main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
[main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
[main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 20.10.8
  API Version: 1.41
  Operating System: Docker Desktop
  Total Memory: 16011 MB
[main] WARN org.testcontainers.utility.ConfigurationFileImageNameSubstitutor - Image name testcontainers/ryuk:0.3.3 was substituted by configuration to internal/ryuk:latest. This approach is deprecated and will be removed in the future
[main] INFO org.testcontainers.utility.ImageNameSubstitutor - Using internal/ryuk:latest as a substitute image for testcontainers/ryuk:0.3.3 (using image substitutor: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor'))
[main] INFO docker[internal/ryuk:latest] - Creating container for image: internal/ryuk:latest
[main] INFO org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: internal/ryuk:latest, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
[main] INFO docker[internal/ryuk:latest] - Container internal/ryuk:latest is starting: 086f10a9ab6496f1cf4cab982890a69fbc09f86548a2e8bdd468ddbdb8416d5f
[main] INFO docker[internal/ryuk:latest] - Container internal/ryuk:latest started in PT1.337369S
[testcontainers-ryuk] WARN org.testcontainers.utility.RyukResourceReaper - Can not connect to Ryuk at 172.17.0.1:61798
java.net.ConnectException: Connection refused (Connection refused)
        at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
        at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
        at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.base/java.net.Socket.connect(Socket.java:609)
        at org.testcontainers.utility.RyukResourceReaper.lambda$null$0(RyukResourceReaper.java:92)
        at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
        at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1(RyukResourceReaper.java:88)
        at java.base/java.lang.Thread.run(Thread.java:829)

Additional Information

Setting TESTCONTAINERS_RYUK_DISABLED=true allows my test cases to pass. I do not experience this on previous version 1.15.3; the tests run as expected. I hit this issue starting in 1.16.0 and is still present in 1.16.3.

[hariohmprasath]

Hi @adamjshook,
Just curious, seems like you are using image name substitution feature to use internal/ryuk:latest image instead of testcontainers/ryuk:0.3.3, can you try directly using testcontainers/ryuk:0.3.3 and see whether you are seeing this error?

[adamjshook]

@hariohmprasath

Yes, we are using that feature. I changed it to use testcontainers/ryuk:0.3.3 and I still see the error on 1.17.3 and 1.16.0. Works on 1.15.3.

[hariohmprasath]

Hi @adamjshook,
Thanks for sharing the details, looks like ryuk container didn't start properly which would have resulted in a socket connection failure:

Here a sample code that would create the ryuk container and connect to it using Socket object, you can run this code in a test case and it would be easy to debug the failure from here:

Note: Make sure to set this environment variable (TESTCONTAINERS_RYUK_DISABLED=true) before running the unit test orelse we would run in container conflict exception.

@Test
    public void testGetValue() {

        final GenericContainer<?> ryukContainer = new GenericContainer<>("testcontainers/ryuk:0.3.4")
            .withExposedPorts(8080)
            .withCreateContainerCmdModifier(cmd -> {
                cmd.withName("testcontainers-ryuk-" + DockerClientFactory.SESSION_ID);
                cmd.withHostConfig(
                    cmd
                        .getHostConfig()
                        .withAutoRemove(true)
                        .withPrivileged(TestcontainersConfiguration.getInstance().isRyukPrivileged())
                        .withBinds(
                            new Bind(
                                DockerClientFactory.instance().getRemoteDockerUnixSocketPath(),
                                new Volume("/var/run/docker.sock")
                            )
                        )
                );
            })
            .waitingFor(Wait.forLogMessage(".*Started.*", 1));

        ryukContainer.start();

        String host = ryukContainer.getHost();
        Integer ryukPort = ryukContainer.getFirstMappedPort();
        try (Socket clientSocket = new Socket()) {
            clientSocket.connect(new InetSocketAddress(host, ryukPort), 5 * 1000);
        } catch (Exception e) {
            throw new IllegalStateException("Could not connect to Ryuk at " + host + ":" + ryukPort, e);
        }
}

[adamjshook]

I ran the provided code (with some additional logging) and in both cases the Ryuk container starts and the client attempts to connect at 172.17.0.1 but it fails on version 1.17.3.

Logs from the test case:

1.17.3

[main] WARN org.testcontainers.utility.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/root/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /root/.testcontainers.properties (No such file or directory)
[main] INFO org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
[main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
[main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
[main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 20.10.8
  API Version: 1.41
  Operating System: Docker Desktop
  Total Memory: 16011 MB
[main] INFO org.testcontainers.DockerClientFactory - Checking the system...
[main] INFO org.testcontainers.DockerClientFactory - ?? Docker server version should be at least 1.6.0
[main] WARN org.testcontainers.utility.ConfigurationFileImageNameSubstitutor - Image name testcontainers/ryuk:0.3.4 was substituted by configuration to testcontainers/ryuk:0.3.3. This approach is deprecated and will be removed in the future
[main] INFO org.testcontainers.utility.ImageNameSubstitutor - Using testcontainers/ryuk:0.3.3 as a substitute image for testcontainers/ryuk:0.3.4 (using image substitutor: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor'))
[main] INFO docker[testcontainers/ryuk:0.3.3] - Creating container for image: testcontainers/ryuk:0.3.3
[main] INFO org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: testcontainers/ryuk:0.3.3, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
[main] INFO docker[testcontainers/ryuk:0.3.3] - Container testcontainers/ryuk:0.3.3 is starting: 7ca3969a762cde2c61d0c16a9f9488ba1eae88fdc849ae1ed0bed8d7e0f42441
[docker-java-stream--391206907] INFO ryuk - STDERR: 2022/09/26 14:01:44 Pinging Docker...
[docker-java-stream--391206907] INFO ryuk - STDERR: 2022/09/26 14:01:44 Docker daemon is available!
[docker-java-stream--391206907] INFO ryuk - STDERR: 2022/09/26 14:01:44 Starting on port 8080...
[docker-java-stream--391206907] INFO ryuk - STDERR: 2022/09/26 14:01:44 Started!
[main] INFO docker[testcontainers/ryuk:0.3.3] - Container testcontainers/ryuk:0.3.3 started in PT1.458051S
[main] INFO host - Connecting to Ryuk at 172.17.0.1:51042
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.107 s <<< FAILURE! - in com.foo.TestRyuk
[ERROR] testGetValue  Time elapsed: 6.061 s  <<< ERROR!
java.lang.IllegalStateException: Could not connect to Ryuk at 172.17.0.1:51042
        at com.foo.TestRyuk.testGetValue(TestRyuk.java:48)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at com.foo.TestRyuk.testGetValue(TestRyuk.java:46)

1.15.3

[main] WARN org.testcontainers.utility.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/root/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /root/.testcontainers.properties (No such file or directory)
[main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/apache-maven/bin, /java/bin, /usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin])
[main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
[main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
[main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 20.10.8
  API Version: 1.41
  Operating System: Docker Desktop
  Total Memory: 16011 MB
[main] INFO org.testcontainers.DockerClientFactory - Checking the system...
[main] INFO org.testcontainers.DockerClientFactory - ?? Docker server version should be at least 1.6.0
[main] INFO org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
[main] INFO org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: alpine:3.5, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
[main] INFO org.testcontainers.DockerClientFactory - ?? Docker environment should have more than 2GB free disk space
[main] WARN org.testcontainers.utility.ConfigurationFileImageNameSubstitutor - Image name testcontainers/ryuk:0.3.4 was substituted by configuration to testcontainers/ryuk:0.3.3. This approach is deprecated and will be removed in the future
[main] INFO org.testcontainers.utility.ImageNameSubstitutor - Using testcontainers/ryuk:0.3.3 as a substitute image for testcontainers/ryuk:0.3.4 (using image substitutor: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor'))
[main] INFO docker[testcontainers/ryuk:0.3.3] - Creating container for image: testcontainers/ryuk:0.3.3
[main] INFO docker[testcontainers/ryuk:0.3.3] - Starting container with ID: 52d67782ec5e99098b1411cf6a8e22e597ed51a95605fafa9e59a019832a4e0c
[main] INFO docker[testcontainers/ryuk:0.3.3] - Container testcontainers/ryuk:0.3.3 is starting: 52d67782ec5e99098b1411cf6a8e22e597ed51a95605fafa9e59a019832a4e0c
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 Pinging Docker...
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 Docker daemon is available!
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 Starting on port 8080...
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 Started!
[main] INFO docker[testcontainers/ryuk:0.3.3] - Container testcontainers/ryuk:0.3.3 started in PT0.947427S
[main] INFO host - Connecting to Ryuk at 172.17.0.1:55003
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 New client connected: 172.17.0.1:55040
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 EOF
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 Client disconnected: 172.17.0.1:55040
[docker-java-stream--576576817] INFO ryuk - STDERR: 2022/09/26 14:07:11 Received the first connection
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.614 s - in com.foo.TestRyuk

[kiview]

Testcontainers inside a Docker container on Mac OS

I think you are running into this upstream issue: #4395
Please try out the workaround specified here and let us know if it helps.

@hariohmprasath Thanks a lot for your supportt in triaging this issue 🙇

[adamjshook]

let us know if it helps.

I tried the workaround of mounting /var/run/docker.sock.raw:/var/run/docker.sock instead of /var/run/docker.sock:/var/run/docker.sock and I got the same Connection refused result.

[kiview]

@adamjshook Unfortunately I don't know what else we can do here, it probably requires more detailed debugging of the Docker environment. Did you verify that /var/run/docker.sock.raw exists and that you can interact with it?

Can you please share your finding, that the workaround does not work for you in #4395? I will close this issue as a duplicate.

[bigunyak]

I have the same issue with fresh colima installation and testcontainers-1.17.5.
I'm not sure how to try the workaround you suggested @kiview with a different socket mounting?

[eddumelendez]

@bigunyak take a look at our documentation. Also, Testcontainers Desktop with embedded runtime works OOTB with Testcontainers

[bigunyak]

@eddumelendez , starting colima with --network-address and setting TESTCONTAINERS_HOST_OVERRIDE like it now mentioned in the docs, removed all warnings about Ryuk that I had. Thank you very much! ❤️

[rdehuyss]

Mentioning issue #7036 here as it solved it for me. The magic trick was to have this environment variable: TESTCONTAINERS_HOST_OVERRIDE=host.docker.internal

This is on Mac M4 Pro.