[Bug]: Broken pipe when creating container with podman

[gaeljw]

Module

MongoDB

Testcontainers version

1.17.6

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host Arch

x86

Docker version

podman version  
Client:       Podman Engine
Version:      4.3.1
API Version:  4.3.1
Go Version:   go1.19.2
Built:        Fri Nov 11 16:01:27 2022
OS/Arch:      linux/amd64

What happened?

I'm using a standard MongoDBContainer in 3 test classes. Running them with docker works perfectly fine. Running them with podman raises a weird issue "Broken pipe". This always reproduces, always same test failing.

If I run only the failing test, the test works fine.

If I run the 3 tests and add an ugly "retry" around container start method, the previously failing test works on 2nd retry.

It look's to me like some time-sentitive thing or a cleanup thing which does not occur correctly maybe?

Here's the code of each test (the 3 classes are identical excepted with the actual tests content):

class MyTest {

  MongoDBContainer fakeMongo = new MongoDBContainer(...)
    //.withStartupAttempts(2)

  public void beforeAll() {
    fakeMongo.start()
  }

  public void afterAll() {
   fakeMongo.stop()
  }

 // Some tests

}

Relevant log output

Here are the logs with DEBUG level and the full stacktrace:

[debug] o.t.u.PrefixingImageNameSubstitutor - Applying changes to image name mongo:4.2.5: applying prefix 'registry.mycompany.net/'
[info] o.t.u.ImageNameSubstitutor - Using registry.mycompany.net/mongo:4.2.5 as a substitute image for mongo:4.2.5 (using image substitutor: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor'))
[debug] o.t.i.AbstractImagePullPolicy - Using locally available and not pulling image: registry.mycompany.net/mongo:4.2.5
[debug] o.t.u.RegistryAuthLocator - Looking up auth config for image: registry.mycompany.net/mongo:4.2.5 at registry: registry.mycompany.net
[debug] o.t.u.RegistryAuthLocator - Cached auth found: [AuthConfig{username=xxxx, password=hidden non-blank value, auth=hidden non-blank value, email=null, registryAddress=registry.mycompany.net, registryToken=blank}]
[debug] o.t.d.AuthDelegatingDockerClientConfig - Effective auth config [AuthConfig{username=xxxx, password=hidden non-blank value, auth=hidden non-blank value, email=null, registryAddress=registry.mycompany.net, registryToken=blank}]
[debug] o.t.s.c.g.d.c.c.AbstrDockerCmd - Cmd: org.testcontainers.shaded.com.github.dockerjava.core.command.CreateContainerCmdImpl@49ebec71[aliases=<null>,argsEscaped=<null>,attachStderr=<null>,attachStdin=<null>,attachStdout=<null>,authConfig=AuthConfig(username=xxxx, email=null, registryAddress=registry.mycompany.net, stackOrchestrator=null),cmd={--replSet,docker-rs},domainName=<null>,entrypoint=<null>,env={},exposedPorts=ExposedPorts(exposedPorts=[27017/tcp]),healthcheck=<null>,hostConfig=HostConfig(binds=[], blkioWeight=null, blkioWeightDevice=null, blkioDeviceReadBps=null, blkioDeviceWriteBps=null, blkioDeviceReadIOps=null, blkioDeviceWriteIOps=null, memorySwappiness=null, nanoCPUs=null, capAdd=null, capDrop=null, containerIDFile=null, cpuPeriod=null, cpuRealtimePeriod=null, cpuRealtimeRuntime=null, cpuShares=null, cpuQuota=null, cpusetCpus=null, cpusetMems=null, devices=null, deviceCgroupRules=null, deviceRequests=null, diskQuota=null, dns=null, dnsOptions=null, dnsSearch=null, extraHosts=[], groupAdd=null, ipcMode=null, cgroup=null, links=[], logConfig=LogConfig(type=null, config=null), lxcConf=null, memory=null, memorySwap=null, memoryReservation=null, kernelMemory=null, networkMode=null, oomKillDisable=null, init=null, autoRemove=null, oomScoreAdj=null, portBindings={27017/tcp=[Lcom.github.dockerjava.api.model.Ports$Binding;@7e3aa866}, privileged=null, publishAllPorts=null, readonlyRootfs=null, restartPolicy=null, ulimits=null, cpuCount=null, cpuPercent=null, ioMaximumIOps=null, ioMaximumBandwidth=null, volumesFrom=[], mounts=null, pidMode=null, isolation=null, securityOpts=null, storageOpt=null, cgroupParent=null, volumeDriver=null, shmSize=null, pidsLimit=null, runtime=null, tmpFs=null, utSMode=null, usernsMode=null, sysctls=null, consoleSize=null, cgroupnsMode=null),hostName=<null>,image=registry.mycompany.net/mongo:4.2.5,ipv4Address=<null>,ipv6Address=<null>,labels={org.testcontainers=true, org.testcontainers.lang=java, org.testcontainers.version=1.17.6, org.testcontainers.sessionId=16b4f81c-0035-4172-a4a4-db8e3299db9c},macAddress=<null>,name=<null>,networkDisabled=<null>,networkingConfig=<null>,onBuild=<null>,platform=<null>,portSpecs=<null>,shell=<null>,stdInOnce=<null>,stdinOpen=<null>,stopSignal=<null>,stopTimeout=<null>,tty=<null>,user=<null>,volumes=Volumes(volumes=[]),workingDir=<null>]
[trace] o.t.s.c.g.d.c.e.CreateContainerCmdExec - POST: DefaultWebTarget{path=[/containers/create], queryParams={}} 
[error] ?.a.c.k.n.2.5] - Could not start container
java.lang.RuntimeException: java.io.IOException: com.sun.jna.LastErrorException: [32] Relais brisé (pipe)
	at com.github.dockerjava.zerodep.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:195)
	at com.github.dockerjava.zerodep.ZerodepDockerHttpClient.execute(ZerodepDockerHttpClient.java:8)
	at org.testcontainers.dockerclient.HeadersAddingDockerHttpClient.execute(HeadersAddingDockerHttpClient.java:23)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:124)
	at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:37)
	at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13)
	at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
	at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
	at org.testcontainers.shaded.com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:608)
Caused by: java.io.IOException: com.sun.jna.LastErrorException: [32] Relais brisé (pipe)
	at com.github.dockerjava.transport.DomainSocket.write(DomainSocket.java:92)
	at com.github.dockerjava.transport.DomainSocket$DomainSocketOutputStream.write(DomainSocket.java:185)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.io.LoggingOutputStream.write(LoggingOutputStream.java:72)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:117)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flush(SessionOutputBufferImpl.java:126)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.ContentLengthOutputStream.flush(ContentLengthOutputStream.java:99)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.io.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:131)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:153)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:141)
	at com.github.dockerjava.zerodep.HijackingHttpRequestExecutor.execute(HijackingHttpRequestExecutor.java:50)
	Suppressed: java.io.IOException: com.sun.jna.LastErrorException: [32] Relais brisé (pipe)
		at com.github.dockerjava.transport.DomainSocket.write(DomainSocket.java:92)
		at com.github.dockerjava.transport.DomainSocket$DomainSocketOutputStream.write(DomainSocket.java:185)
		at com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.io.LoggingOutputStream.write(LoggingOutputStream.java:72)
		at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:117)
		at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flush(SessionOutputBufferImpl.java:126)
		at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.ContentLengthOutputStream.close(ContentLengthOutputStream.java:93)
		at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:154)
		at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:141)
		at com.github.dockerjava.zerodep.HijackingHttpRequestExecutor.execute(HijackingHttpRequestExecutor.java:50)
		at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:218)
	Caused by: com.sun.jna.LastErrorException: [32] Relais brisé (pipe)
		at com.github.dockerjava.transport.DomainSocket.write(Native Method)
		at com.github.dockerjava.transport.DomainSocket.write(DomainSocket.java:90)
		at com.github.dockerjava.transport.DomainSocket$DomainSocketOutputStream.write(DomainSocket.java:185)
		at com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.io.LoggingOutputStream.write(LoggingOutputStream.java:72)
		at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:117)
		at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flush(SessionOutputBufferImpl.java:126)
		at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.ContentLengthOutputStream.close(ContentLengthOutputStream.java:93)
		at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:154)
		at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:141)
		at com.github.dockerjava.zerodep.HijackingHttpRequestExecutor.execute(HijackingHttpRequestExecutor.java:50)
Caused by: com.sun.jna.LastErrorException: [32] Relais brisé (pipe)
	at com.github.dockerjava.transport.DomainSocket.write(Native Method)
	at com.github.dockerjava.transport.DomainSocket.write(DomainSocket.java:90)
	at com.github.dockerjava.transport.DomainSocket$DomainSocketOutputStream.write(DomainSocket.java:185)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.client5.http.impl.io.LoggingOutputStream.write(LoggingOutputStream.java:72)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:117)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flush(SessionOutputBufferImpl.java:126)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.ContentLengthOutputStream.flush(ContentLengthOutputStream.java:99)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.io.entity.ByteArrayEntity.writeTo(ByteArrayEntity.java:131)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:153)
	at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:141)

Additional Information

No response

[gaeljw]

Additional information: this also happens with a Redis container, that's not related to Mongo container.

The more tests classes I have using a container with testcontainers, the more likely the issue is to appear.

For now, the workaround of retries (thanks to withStartupAttempts(3)) do work.

[eddumelendez]

Hi, can you test with latest podman version? I know there are improvements to be Testcontainers friendly.

[eddumelendez]

Closing due to lack of activity. @gaeljw feel free to reopen. I'd suggest to use Podman version > 4.5

[xstefank]

I am running into this error occasionally with Podman 4.5.0.

[nickbreen]

Experiencing this with:

TestContainers java 1.20.0
Podman 4.9.4-rhel
RockyLinux 8.8

[igieon]

Hi all the bug is not problem of testcontainers but your configuration of podman is because testcontaeiner cache http conection to API server if you start via podman.socket via systemd you need change:
/etc/containers/containers.conf:

[engine]         
service_timeout=0

Default timeout is 5s which you can hit