Unable to use dev service containers with Podman on Linux Fedora #44802

MikaelAnderssonWigander · 2024-11-28T11:45:19Z

MikaelAnderssonWigander
Nov 28, 2024

My current setup is on a Linux Fedora 41 with Podman 5.3.1 and Quarkus 3.17.0.

When starting Dev Services the image for Artemis (in my case) can't be pulled. It can be pulled using Podman commands but not when running quarkus dev.

Eventually after timeout I get this exception:

Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=quay.io/artemiscloud/activemq-artemis-broker:artemis.2.37.0, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@ee0130d)

My environment has the following:

TESTCONTAINERS_RYUK_DISABLED=true
DOCKER_HOST=unix:///run/user/1000/podman/podman.sock

Podman was pre-installed but I have also downloaded Podman-desktop and Podman-remote
Also issued the following according to docs:

systemctl --user enable podman.socket --now

I'm behind a corporate firewall and have set HTTP_PROXY and HTTPS_PROXY.

@geoand · 2024-11-28T11:45:22Z

quarkus-bot[bot]
bot Nov 28, 2024

/cc @geoand (devservices), @maxandersen (podman), @stuartwdouglas (devservices)

3 replies

geoand Nov 28, 2024
Collaborator

@zakkak and @gastaldi might have some insights on this

gastaldi Nov 28, 2024
Collaborator

It sounds like the corporate firewall is blocking the image download.

Are you running quarkus dev after podman pull quay.io/artemiscloud/activemq-artemis-broker:artemis.2.37.0?

MikaelAnderssonWigander Nov 28, 2024
Author

Tried both, same issue

Have a Mac with Podman with the same proxy settings that is working

maxandersen · 2024-11-28T13:32:18Z

maxandersen
Nov 28, 2024
Maintainer

is there a more full stacktrace that might reveal underlying issue? (I agree it most likely is ability to not connect, but could also be https certs being munged with which would require different fix)

0 replies

MikaelAnderssonWigander · 2024-11-28T13:36:17Z

MikaelAnderssonWigander
Nov 28, 2024
Author

2024-11-28 14:33:21,053 ERROR traceId=, parentId=, spanId=, sampled= [com.git.doc.api.asy.ResultCallbackTemplate] (docker-java-stream-2060035367) Error during callback: com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"i/o timeout"}tails: (dockerImageName: quay.io/artemiscloud/activemq-artemis-broker:latest, configFile: /home/maw/.docker/config.json, configEnv: DOCKER_AUTH_
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
	at java.base/java.lang.Thread.run(Thread.java:1583)

2024-11-28 14:35:24,263 ERROR traceId=, parentId=, spanId=, sampled= [com.git.doc.api.asy.ResultCallbackTemplate] (docker-java-stream-2060035367) Error during callback: com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"i/o timeout"}

	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
	at java.base/java.lang.Thread.run(Thread.java:1583)

2024-11-28 14:31:16,208 INFO  traceId=, parentId=, spanId=, sampled= [org.tes.DockerClientFactory] (build-46) Testcontainers version: 1.20.2
2024-11-28 14:31:17,257 INFO  traceId=, parentId=, spanId=, sampled= [org.tes.doc.DockerClientProviderStrategy] (build-46) Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///run/user/1000/podman/podman.sock
2024-11-28 14:31:17,337 INFO  traceId=, parentId=, spanId=, sampled= [org.tes.DockerClientFactory] (build-46) Docker host IP address is localhost
2024-11-28 14:31:17,619 INFO  traceId=, parentId=, spanId=, sampled= [org.tes.DockerClientFactory] (build-46) Connected to docker: 
  Server Version: 5.3.1
  API Version: 1.41
  Operating System: fedora
  Total Memory: 7912 MB

2024-11-28 14:31:17,633 WARN  traceId=, parentId=, spanId=, sampled= [org.tes.uti.ResourceReaper] (build-46) 
********************************************************************************
Ryuk has been disabled. This can cause unexpected behavior in your environment.
********************************************************************************

2024-11-28 14:31:17,640 INFO  traceId=, parentId=, spanId=, sampled= [org.tes.DockerClientFactory] (build-46) Checking the system...
2024-11-28 14:31:17,658 INFO  traceId=, parentId=, spanId=, sampled= [org.tes.DockerClientFactory] (build-46) ✔︎ Docker server version should be at least 1.6.0
2024-11-28 14:31:17,853 INFO  traceId=, parentId=, spanId=, sampled= [org.tes.ima.PullPolicy] (build-46) Image pull policy will be performed by: DefaultPullPolicy()
2024-11-28 14:31:17,858 INFO  traceId=, parentId=, spanId=, sampled= [org.tes.uti.ImageNameSubstitutor] (build-46) Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
2024-11-28 14:31:17,997 INFO  traceId=, parentId=, spanId=, sampled= [tc.qua.io/.2.37.0] (build-46) Pulling docker image: quay.io/artemiscloud/activemq-artemis-broker:artemis.2.37.0. Please be patient; this may take some time but only needs to be done once.
2024-11-28 14:31:18,007 INFO  traceId=, parentId=, spanId=, sampled= [org.tes.uti.RegistryAuthLocator] (build-46) Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: quay.io/artemiscloud/activemq-artemis-broker:latest, configFile: /home/maw/.docker/config.json, configEnv: DOCKER_AUTH_CONFIG). Falling back to docker-java default behaviour. Exception message: Status 404: No config supplied. Checked in order: /home/maw/.docker/config.json (file not found), DOCKER_AUTH_CONFIG (not set)
2024-11-28 14:33:21,056 WARN  traceId=, parentId=, spanId=, sampled= [tc.qua.io/.2.37.0] (build-46) Retrying pull for image: quay.io/artemiscloud/activemq-artemis-broker:artemis.2.37.0 (-4s remaining)
2024-11-28 14:33:21,197 INFO  traceId=, parentId=, spanId=, sampled= [tc.qua.io/.2.37.0] (build-46) Pulling docker image: quay.io/artemiscloud/activemq-artemis-broker:artemis.2.37.0. Please be patient; this may take some time but only needs to be done once.
2024-11-28 14:35:24,264 WARN  traceId=, parentId=, spanId=, sampled= [tc.qua.io/.2.37.0] (build-46) Retrying pull for image: quay.io/artemiscloud/activemq-artemis-broker:artemis.2.37.0 (-4s remaining)
2024-11-28 14:35:24,424 INFO  [io.qua.dep.dev.IsolatedDevModeMain] (main) Attempting to start live reload endpoint to recover from previous Quarkus startup failure
2024-11-28 14:35:25,605 ERROR [io.qua.dep.dev.IsolatedDevModeMain] (main) Failed to start quarkus: java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
	[error]: Build step io.quarkus.artemis.core.deployment.DevServicesArtemisProcessor#startArtemisDevService threw an exception: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=quay.io/artemiscloud/activemq-artemis-broker:artemis.2.37.0, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@57e71fe8)
	at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1364)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:359)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:330)
	at io.quarkus.artemis.core.deployment.DevServicesArtemisProcessor.lambda$startArtemis$1(DevServicesArtemisProcessor.java:250)
	at java.base/java.util.Optional.orElseGet(Optional.java:364)
	at io.quarkus.artemis.core.deployment.DevServicesArtemisProcessor.startArtemis(DevServicesArtemisProcessor.java:266)
	at io.quarkus.artemis.core.deployment.DevServicesArtemisProcessor.start(DevServicesArtemisProcessor.java:128)
	at io.quarkus.artemis.core.deployment.DevServicesArtemisProcessor.startArtemisDevService(DevServicesArtemisProcessor.java:87)
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
	at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:856)
	at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
	at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
	at java.base/java.lang.Thread.run(Thread.java:1583)
	at org.jboss.threads.JBossThread.run(JBossThread.java:499)
Caused by: org.testcontainers.shaded.org.awaitility.core.ConditionTimeoutException: Condition org.testcontainers.images.RemoteDockerImage$$Lambda/0x00007f037485a510 was not fulfilled within 2 minutes.
	at org.testcontainers.shaded.org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.testcontainers.shaded.org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
	at org.testcontainers.shaded.org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
	at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
	at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:954)
	at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:106)
	at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:34)
	at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
	at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:41)
	at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1362)
	... 17 more

	at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:354)
	at io.quarkus.runner.bootstrap.AugmentActionImpl.createInitialRuntimeApplication(AugmentActionImpl.java:272)
	at io.quarkus.runner.bootstrap.AugmentActionImpl.createInitialRuntimeApplication(AugmentActionImpl.java:62)
	at io.quarkus.deployment.dev.IsolatedDevModeMain.firstStart(IsolatedDevModeMain.java:89)
	at io.quarkus.deployment.dev.IsolatedDevModeMain.accept(IsolatedDevModeMain.java:428)
	at io.quarkus.deployment.dev.IsolatedDevModeMain.accept(IsolatedDevModeMain.java:55)
	at io.quarkus.bootstrap.app.CuratedApplication.runInCl(CuratedApplication.java:138)
	at io.quarkus.bootstrap.app.CuratedApplication.runInAugmentClassLoader(CuratedApplication.java:93)
	at io.quarkus.deployment.dev.DevModeMain.start(DevModeMain.java:131)
	at io.quarkus.deployment.dev.DevModeMain.main(DevModeMain.java:62)
Caused by: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
	[error]: Build step io.quarkus.artemis.core.deployment.DevServicesArtemisProcessor#startArtemisDevService threw an exception: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=quay.io/artemiscloud/activemq-artemis-broker:artemis.2.37.0, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@57e71fe8)
	at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1364)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:359)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:330)
	at io.quarkus.artemis.core.deployment.DevServicesArtemisProcessor.lambda$startArtemis$1(DevServicesArtemisProcessor.java:250)
	at java.base/java.util.Optional.orElseGet(Optional.java:364)
	at io.quarkus.artemis.core.deployment.DevServicesArtemisProcessor.startArtemis(DevServicesArtemisProcessor.java:266)
	at io.quarkus.artemis.core.deployment.DevServicesArtemisProcessor.start(DevServicesArtemisProcessor.java:128)
	at io.quarkus.artemis.core.deployment.DevServicesArtemisProcessor.startArtemisDevService(DevServicesArtemisProcessor.java:87)
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
	at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:856)
	at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
	at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
	at java.base/java.lang.Thread.run(Thread.java:1583)
	at org.jboss.threads.JBossThread.run(JBossThread.java:499)
Caused by: org.testcontainers.shaded.org.awaitility.core.ConditionTimeoutException: Condition org.testcontainers.images.RemoteDockerImage$$Lambda/0x00007f037485a510 was not fulfilled within 2 minutes.
	at org.testcontainers.shaded.org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.testcontainers.shaded.org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
	at org.testcontainers.shaded.org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
	at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
	at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:954)
	at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:106)
	at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:34)
	at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
	at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:41)
	at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1362)
	... 17 more

	at io.quarkus.builder.Execution.run(Execution.java:124)
	at io.quarkus.builder.BuildExecutionBuilder.execute(BuildExecutionBuilder.java:79)
	at io.quarkus.deployment.QuarkusAugmentor.run(QuarkusAugmentor.java:161)
	at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:350)
	... 9 more
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=quay.io/artemiscloud/activemq-artemis-broker:artemis.2.37.0, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@57e71fe8)
	at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1364)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:359)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:330)
	at io.quarkus.artemis.core.deployment.DevServicesArtemisProcessor.lambda$startArtemis$1(DevServicesArtemisProcessor.java:250)
	at java.base/java.util.Optional.orElseGet(Optional.java:364)
	at io.quarkus.artemis.core.deployment.DevServicesArtemisProcessor.startArtemis(DevServicesArtemisProcessor.java:266)
	at io.quarkus.artemis.core.deployment.DevServicesArtemisProcessor.start(DevServicesArtemisProcessor.java:128)
	at io.quarkus.artemis.core.deployment.DevServicesArtemisProcessor.startArtemisDevService(DevServicesArtemisProcessor.java:87)
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
	at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:856)
	at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2675)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2654)
	at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1627)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1594)
	at java.base/java.lang.Thread.run(Thread.java:1583)
	at org.jboss.threads.JBossThread.run(JBossThread.java:499)
Caused by: org.testcontainers.shaded.org.awaitility.core.ConditionTimeoutException: Condition org.testcontainers.images.RemoteDockerImage$$Lambda/0x00007f037485a510 was not fulfilled within 2 minutes.
	at org.testcontainers.shaded.org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.testcontainers.shaded.org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
	at org.testcontainers.shaded.org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
	at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
	at org.testcontainers.shaded.org.awaitility.core.ConditionFactory.until(ConditionFactory.java:954)
	at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:106)
	at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:34)
	at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
	at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:41)
	at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1362)
	... 17 more

11 replies

gastaldi Nov 28, 2024
Collaborator

Good to know, but I wonder if the fix in the PR solves your problem when running inside the corporate firewall

turing85 Nov 28, 2024

Hi!
So... as far as I see - whether it is podman or docker used - testcontainers communicates with the socket. Thus, the actual operations should be executed by the podman-/docker-process, which should already respect HTTP_PROXY and HTTPS_PROXY (see e.g. here (stackoverflow.com) and here (testcontainers.com)).

@MikaelAnderssonWigander If I understand you correctly, podman pull ... does not work either, right? If so, then this signals to me that the problem lies elsewhere and will not be fixed by adding the environment variables in the devservice @gastaldi.

MikaelAnderssonWigander Nov 29, 2024
Author

My Podman pull works perfectly fine in all cases, it's when running quarkus devit fails if behind firewall

turing85 Dec 6, 2024

@MikaelAnderssonWigander does the fix from @gastaldi resolve your issue?

MikaelAnderssonWigander Dec 9, 2024
Author

Can't tell really, it seems to be working now for some reason and I'm using the most recent production version. Also I have tested using other containers such as Kafka and it's working fine now…
I think we can close this

Unable to use dev service containers with Podman on Linux Fedora #44802

Uh oh!

Uh oh!

MikaelAnderssonWigander Nov 28, 2024

Replies: 3 comments · 14 replies

Uh oh!

quarkus-bot[bot] bot Nov 28, 2024

Uh oh!

geoand Nov 28, 2024 Collaborator

Uh oh!

gastaldi Nov 28, 2024 Collaborator

Uh oh!

MikaelAnderssonWigander Nov 28, 2024 Author

Uh oh!

maxandersen Nov 28, 2024 Maintainer

Uh oh!

Uh oh!

MikaelAnderssonWigander Nov 28, 2024 Author

Uh oh!

gastaldi Nov 28, 2024 Collaborator

Uh oh!

Uh oh!

turing85 Nov 28, 2024

Uh oh!

Uh oh!

MikaelAnderssonWigander Nov 29, 2024 Author

Uh oh!

turing85 Dec 6, 2024

Uh oh!

MikaelAnderssonWigander Dec 9, 2024 Author

MikaelAnderssonWigander
Nov 28, 2024

Replies: 3 comments 14 replies

quarkus-bot[bot]
bot Nov 28, 2024

geoand Nov 28, 2024
Collaborator

gastaldi Nov 28, 2024
Collaborator

MikaelAnderssonWigander Nov 28, 2024
Author

maxandersen
Nov 28, 2024
Maintainer

MikaelAnderssonWigander
Nov 28, 2024
Author

gastaldi Nov 28, 2024
Collaborator

MikaelAnderssonWigander Nov 29, 2024
Author

MikaelAnderssonWigander Dec 9, 2024
Author