Network cleanup not always working

[vmassol]

Module

Core

Testcontainers version

1.17.3

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host Arch

x86

Docker version

$ sudo docker version
Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:37 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:46 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

We're getting regularly the error Caused by: com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network"} in our TC-based tests.

While researching this, I've found that several networks created by TC remain and are not deleted after the tests. For example on our CI agent:

$ sudo docker network ls
NETWORK ID     NAME                                   DRIVER    SCOPE
fc00e553e083   40fde7d3-324c-42cc-aeac-bfa9cd68cd0d   bridge    local
0a2f5e0ebc2f   47c53062-6c06-415d-a961-b45cf3f686af   bridge    local
2b7f4773ec7b   81ea6de2-0237-4eb7-b5c3-ed545f34d33c   bridge    local
1a56194c0416   38864bbb-de57-4cb3-9fd2-759056daa61c   bridge    local
d5d4b01c9f0a   41787e72-4ba5-43c2-a83f-6bf6949ca4c1   bridge    local
62ae2d1e22f2   52294b16-867d-4a66-930a-3e2432c898f1   bridge    local
79bd7f2e9a9a   2628661f-de18-45a8-80bd-f13c7a8f5aa1   bridge    local
e2072e729583   a7b7b131-15f4-4dd1-aa7e-35f548f4b7fd   bridge    local
13ecf70ca868   b29b8b18-9c8a-48cb-b628-763f936a12bc   bridge    local
88216c595688   bridge                                 bridge    local
623732a6b674   c864c52c-f630-42e6-a123-d0ad590063a5   bridge    local
08640b19df0f   ebaca507-973e-4ba5-be7f-d01987344b3f   bridge    local
4bcdbd714e78   f9111034-9c6f-4e67-9ce7-89f2719ee689   bridge    local
3f318d295a89   host                                   host      local
c3aad684575a   none                                   null      local

If I inspect the networks with the numbers in their name, I can see they're labelled as TC ones:

$ sudo docker  network inspect e2072e729583
[
    {
        "Name": "a7b7b131-15f4-4dd1-aa7e-35f548f4b7fd",
        "Id": "e2072e729583dd48d347dc333dc7a0f05f5627d93f3a4514cf25a2c76cfced00",
        "Created": "2022-07-20T04:22:05.82718144Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.21.0.0/16",
                    "Gateway": "172.21.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {
            "org.testcontainers": "true",
            "org.testcontainers.sessionId": "91cbf7fa-3a02-4279-94dd-c59397225ce1"
        }
    }
]

Is this a known bug/limitation? Is there something I can do about it?

Thanks!

Relevant log output

No response

Additional Information

No response

[bsideup]

@vmassol do you happen to have Ryuk disabled?

[vmassol]

@bsideup nope, we love Ryuk and we never disable it ;)

[vmassol]

Also, most of the time the networks are cleaned fine (judging by the amount of tests we do every day and the number of stray docker networks remaining).

[marinapmb]

We faced the same problem

[eddumelendez]

Hi @vmassol, are you calling close() from networks in every test class? can you share how are you managing networks?

[vmassol]

Hi @vmassol, are you calling close() from networks in every test class? can you share how are you managing networks?

I don't think we do. I was expecting ryuk to clean up stuff.

Here's what we do re network when starting the servlet container for example:

        List<String> networkAliases = new ArrayList<>();
        networkAliases.add(this.testConfiguration.getServletEngine().getInternalIP());
        networkAliases.addAll(this.testConfiguration.getServletEngineNetworkAliases());

        // Note: TestContainers will wait for up to 60 seconds for the container's first mapped network port to
        // start listening.
        this.servletContainer
            .withNetwork(Network.SHARED)
            .withNetworkAliases(networkAliases.toArray(new String[networkAliases.size()]))
            .waitingFor(
                Wait.forHttp("/xwiki/rest")
                    .forStatusCode(200).withStartupTimeout(Duration.of(480, SECONDS)));
...

What we do in the test teardown is to call stop() on the GenericContainer objects (this.servletContainer in this example).

Isn't that enough to clean the network too?

Thanks for your reply @eddumelendez , appreciated!

[eddumelendez]

Thanks for the quick reply, @vmassol!
Yes, Ryuk does clean up remaining networks, but only at the end of the Java process.
When using the GenericContainer class, you should explicitly call stop() or use the @Container annotation to have Testcontainers manage the container lifecycle automatically.
The same applies to networks—ensure they are properly cleaned up by either calling stop() manually or using lifecycle management mechanisms like JUnit 5’s @AutoCloseable.

[vmassol]

@eddumelendez thx for your response.

Yes, Ryuk does clean up remaining networks, but only at the end of the Java process.

This is the issue I was reporting above: sometimes it doesn't (in rare cases).

When using the GenericContainer class, you should explicitly call stop() or use the @container annotation to have Testcontainers manage the container lifecycle automatically.

Yep we do this, see above.

The same applies to networks—ensure they are properly cleaned up by either calling stop() manually or using lifecycle management mechanisms like JUnit 5’s @AutoCloseable.

But we don't create networkds. I guess they are created automatically by TC when we use the withNetworkconstruct above, right? Thud, I'm expecting that calling stop() should be enough for GenericContainer to close the network it has created.

Saod differently, I don't have any network object on which I can call close() since we don't create networks.

Thx!