"Listening ports" wait strategy does not work well on Podman, for images with multiple default exposed ports

[richvdh]

Given an image which contains multiple EXPOSEd ports, such as this test image.

Attempt to start it, but only expose one port:

const container = await new GenericContainer("richvdh/testcontainer:1.0.2")
    .withExposedPorts(8080)
    .start();

Expected Behaviour

GenericContainer.start should complete and return.

Actual Behaviour

At least under Podman, GenericContainer.start waits for the timeout and then throws an error.

Testcontainer Logs

  testcontainers [DEBUG] Checking container runtime strategy "TestcontainersHostStrategy"... +0ms
  testcontainers [DEBUG] Loading ".testcontainers.properties" file... +0ms
  testcontainers [DEBUG] Loaded ".testcontainers.properties" file +2ms
  testcontainers [DEBUG] Found custom configuration: dockerHost: "unix:///run/user/1000/podman/podman.sock" +0ms
  testcontainers [DEBUG] Container runtime strategy "TestcontainersHostStrategy" is not applicable +0ms
  testcontainers [DEBUG] Checking container runtime strategy "ConfigurationStrategy"... +0ms
  testcontainers [TRACE] Fetching Docker info... +0ms
  testcontainers [TRACE] Fetching remote container runtime socket path... +371ms
  testcontainers [TRACE] Resolving host... +0ms
  testcontainers [TRACE] Fetching Compose info... +1ms
  testcontainers [TRACE] Looking up host IPs... +0ms
  testcontainers [TRACE] Initialising clients... +2ms
  testcontainers [TRACE] Container runtime info:
  testcontainers {
  testcontainers   "node": {
  testcontainers     "version": "v22.17.0",
  testcontainers     "architecture": "x64",
  testcontainers     "platform": "linux"
  testcontainers   },
  testcontainers   "containerRuntime": {
  testcontainers     "host": "localhost",
  testcontainers     "hostIps": [
  testcontainers       {
  testcontainers         "address": "::1",
  testcontainers         "family": 6
  testcontainers       },
  testcontainers       {
  testcontainers         "address": "127.0.0.1",
  testcontainers         "family": 4
  testcontainers       }
  testcontainers     ],
  testcontainers     "remoteSocketPath": "/run/user/1000/podman/podman.sock",
  testcontainers     "indexServerAddress": "https://index.docker.io/v1/",
  testcontainers     "serverVersion": "4.3.1",
  testcontainers     "operatingSystem": "debian",
  testcontainers     "operatingSystemType": "linux",
  testcontainers     "architecture": "amd64",
  testcontainers     "cpus": 16,
  testcontainers     "memory": 33247903744,
  testcontainers     "runtimes": [
  testcontainers       "crun",
  testcontainers       "kata",
  testcontainers       "krun",
  testcontainers       "runc",
  testcontainers       "runj",
  testcontainers       "runsc"
  testcontainers     ],
  testcontainers     "labels": []
  testcontainers   }
  testcontainers } +0ms
  testcontainers [DEBUG] Container runtime strategy "ConfigurationStrategy" works +0ms
  testcontainers [DEBUG] Checking if image exists "richvdh/testcontainer:1.0.2"... +0ms
  testcontainers [DEBUG] Checked if image exists "richvdh/testcontainer:1.0.2" +10ms
  testcontainers [DEBUG] Image "richvdh/testcontainer:1.0.2" already exists +0ms
  testcontainers [DEBUG] Acquiring lock file "/tmp/testcontainers-node.lock"... +2ms
  testcontainers [DEBUG] Acquired lock file "/tmp/testcontainers-node.lock" +2ms
  testcontainers [DEBUG] Listing containers... +0ms
  testcontainers [DEBUG] Listed containers +8ms
  testcontainers [DEBUG] Reusing existing Reaper for session "fc4d4da4e2e0"... +0ms
  testcontainers [DEBUG] [e7d6f54cf93e] Connecting to Reaper (attempt 1) on "localhost:45189"... +0ms
  testcontainers [DEBUG] [e7d6f54cf93e] Connected to Reaper +1ms
  testcontainers [DEBUG] Releasing lock file "/tmp/testcontainers-node.lock"... +0ms
  testcontainers [DEBUG] Released lock file "/tmp/testcontainers-node.lock" +1ms
  testcontainers [DEBUG] Creating container for image "richvdh/testcontainer:1.0.2"... +0ms
  testcontainers [DEBUG] [70103837631a] Created container for image "richvdh/testcontainer:1.0.2" +192ms
  testcontainers [INFO] [70103837631a] Starting container for image "richvdh/testcontainer:1.0.2"... +1ms
  testcontainers [DEBUG] [70103837631a] Starting container... +0ms
  testcontainers [DEBUG] [70103837631a] Started container +183ms
  testcontainers [INFO] [70103837631a] Started container for image "richvdh/testcontainer:1.0.2" +0ms
  testcontainers [ERROR] [70103837631a] Timed out after 10000ms while waiting for container ports to be bound to the host +10s
/home/rav/work/testcontainer/node_modules/testcontainers/build/generic-container/inspect-container-util-ports-exposed.js:15
        return new Error(message);
               ^

Error: Timed out after 10000ms while waiting for container ports to be bound to the host
    at /home/rav/work/testcontainer/node_modules/testcontainers/build/generic-container/inspect-container-util-ports-exposed.js:15:16
    at IntervalRetry.retryUntil (/home/rav/work/testcontainer/node_modules/testcontainers/build/common/retry.js:29:24)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async inspectContainerUntilPortsExposed (/home/rav/work/testcontainer/node_modules/testcontainers/build/generic-container/inspect-container-util-ports-exposed.js:6:20)
    at async GenericContainer.startContainer (/home/rav/work/testcontainer/node_modules/testcontainers/build/generic-container/generic-container.js:139:31)
    at async file:///home/rav/work/testcontainer/main.mjs:9:19

Steps to Reproduce

https://github.com/richvdh/testcontainer contains a demonstration.

1. Clone the repo
2. npm run start
3. Observe that the command hangs, and then times out after 10 seconds.

Environment Information

• Operating System: Debian bookworm
• Docker Version: podman version 4.3.1
• Node version: v22.17.0
• Testcontainers version: 11.5.1

[cristianrgreco]

@richvdh I've run the repro locally with podman and it works for me on an M1 Mac (following these setup instructions). I notice you're using podman 4.x whereas I'm using podman 5.1.2, could you perhaps try updating? Either that or there's some issue running on Debian. Bear in mind we run a comprehensive suite of podman tests on Ubuntu, many of which start containers with multiple ports, as part of this project's build.

If the issue persists afer upgrading, I'd need to you to debug a bit more. E.g. look at the container's logs/inspect result and see why the container ports are not being bound to the host.

[richvdh]

Thanks for the quick feedback @cristianrgreco. You may well be right that it's something that changed between Podman 4 and 5; unfortunately there doesn't seem to be any sensible way to run Podman 5 on Debian 12, and updating my whole system to Debian 13 (or, for that matter, fiddling with VMs) isn't something I really have bandwidth for today. I'll try and find a workaround in my application.

E.g. look at the container's logs/inspect result and see why the container ports are not being bound to the host.

I think this is expected: it's because I am only running .withExposedPorts with one of the default ports from the image. I think the problem is the way podman reports it:

$ podman container inspect trusting_engelbart | jq '.[0].NetworkSettings.Ports' gives:

{
  "8080/tcp": [
    {
      "HostIp": "",
      "HostPort": "38041"
    }
  ],
  "8081/tcp": null
}

[cristianrgreco]

Please reopen this issue if it is reproducible in podman 5.x, but official support for podman 4.3 ended almost 3 years ago, 10 Nov 2022 -- it doesn't make sense for testcontainers to try to support it.

[richvdh]

For my own reference: this was introduced in #1087, so can be worked around by rolling back to 11.3.2.

[richvdh]

(can confirm this seems fine in podman 5.4.)

[joebowbeer]

I'm seeing the 10000ms timeout error message in Rancher Desktop w/Moby on Apple Silicon when I run the following

    await using _ = await new GenericContainer("confluentinc/cp-zookeeper:5.5.4")
      .withNetwork(network)
      .withNetworkAliases(zooKeeperHost)
      .withEnvironment({ ZOOKEEPER_CLIENT_PORT: zooKeeperPort.toString() })
      .withExposedPorts(zooKeeperPort)
      .start();

I found it confusing that the message mentions a 10000ms timeout but I can't see a way to override it.

Changing the WaitStrategy didn't help either.

The underlying problem in my case was that there was no arm64 image for this tag.

It works if I specify a multiarch version that provides arm64:

    await using _ = await new GenericContainer("confluentinc/cp-zookeeper:7.5.0")
      .withNetwork(network)
      .withNetworkAliases(zooKeeperHost)
      .withEnvironment({ ZOOKEEPER_CLIENT_PORT: zooKeeperPort.toString() })
      .withExposedPorts(zooKeeperPort)
      .start();

@cristianrgreco Is there a way to improve the error message?