Podman Rootless Container-to-Host Communication Not Working Despite Service Listening on Host

[Hamzablm]

I'm trying to set up a rootless Podman environment with containers in the same pod that can communicate with each other, access a non-containerized Java application on the host, and allow the host to communicate with the containers. Here’s the setup and all the steps I’ve tried.

Environment:

• Host OS: Ubuntu 22.04.5 LTS

• Podman Version: 3.4.4

  OS/Arch: linux/amd64

• Setup: Rootless Podman, single pod with multiple containers

Goal: I want:

1. Container-to-Container Communication on specific ports inside the pod.
2. Host-to-Container Communication via specific exposed ports.
3. Container-to-Host Communication to access a non-containerized Java application running on the host.

Network Configuration:

• Pod Ports: 0.0.0.0:10443->1443/tcp, 0.0.0.0:13000->3000/tcp, 0.0.0.0:13306->3306/tcp, 0.0.0.0:14000->4000/tcp, 0.0.0.0:18080->8080/tcp, 0.0.0.0:18888->8888/tcp, 0.0.0.0:19201->9201/tcp, 0.0.0.0:11234->12345/tcp, 0.0.0.0:13270->32700/tcp

Host Service:

• A Java application on the host, listening on 0.0.0.0:8080, confirmed to be running with ss -tuln | grep 8080.

What I Tried:

1. Pod Creation with Exposed Ports:

   • Created the pod with all required ports exposed at the pod level:

     podman pod create --name mypod -p 10443:1443 -p 13000:3000 -p 13306:3306 -p 14000:4000 -p 18080:8080 -p 18888:8888 -p 19201:9201 -p 11234:12345 -p 13270:32700

   • Added containers to the pod without using -p or --publish flags, since all network configurations are handled at the pod level.

2. Host-to-Container and Container-to-Container Communication:

   • Host-to-container works fine via localhost:<host_port>.
   • Container-to-container communication works as expected over localhost:<port>.

3. Container-to-Host Communication Attempts:

   • Tried using curl http://host.containers.internal:8080 and curl http://10.88.0.1:8080 (after confirming 10.88.0.1 as the gateway IP for Podman’s default network).
   • Tried different IPs like 10.0.2.2 and 10.0.2.100.
   • No connection to the host service on 8080 from within the containers, despite the service running on 0.0.0.0:8080 on the host.

4. Firewall and SELinux Checks:

   • Temporarily disabled the firewall:

     sudo systemctl stop firewalld

   • Tried setting SELinux to permissive mode:

     sudo setenforce 0

   • None of these changes resolved the issue.

5. Using --network slirp4netns:allow_host_loopback=true:

   • Recreated the pod with --network slirp4netns:allow_host_loopback=true to allow loopback access:

     podman pod create --name mypod --network slirp4netns:allow_host_loopback=true -p 10443:1443 -p 13000:3000 -p 13306:3306 -p 14000:4000 -p 18080:8080 -p 18888:8888 -p 19201:9201 -p 11234:12345 -p 13270:32700

   • Still unable to access http://host.containers.internal:8080 or any other expected IP.

6. Host DNS Resolution:

   • Tried resolving host.containers.internal inside the container using:

     getent hosts host.containers.internal

   • Confirmed it resolves to 10.0.2.100 but still unable to reach the host service.

7. Attempting to Use --network host as a Workaround:

   • Attempted --network host (not officially supported in rootless Podman):

     podman pod create --name mypod --network host

   • Containers could now access the host, but this setup exposed all network interfaces and isn’t ideal.

Summary of Problem:
Container-to-host communication does not work in rootless Podman, despite following various troubleshooting steps. I have confirmed that the service is accessible on 0.0.0.0:8080 on the host, but containers cannot connect to it using host.containers.internal, the gateway IP, or other Podman-recommended methods.

Question:
How can I enable container-to-host communication in rootless Podman? Is there a reliable way to access a host service from containers in a rootless Podman pod, given that the service is listening on all interfaces (0.0.0.0) on the host?

Let me know if there's any other information I should add.

[Luap99]

podman 3.4.4 in ancient and not something we support. We only support the latest upstream version.

In particular when you use podman 5.3 with a recent pasta then host.containers.internal should always work: https://blog.podman.io/2024/10/podman-5-3-changes-for-improved-networking-experience-with-pasta/