Can't connect to host from container when running `podman compose` with rootless Podman 5 #24133

piotr-dobrogost · 2024-10-02T09:50:23Z

piotr-dobrogost
Oct 2, 2024

Having the following docker-compose.yaml file:

version: "3.9"

networks:
    test-net:
        driver: bridge
services:
    fed:
        image: registry.fedoraproject.org/fedora:latest
        networks:
            - test-net
        platform: linux/amd64
        command: bash
        tty: true

I run

podman compose up

then

python3 -m http.server                       
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...

and then

podman exec -it podman_fed_1 bash

on the host.

Now I can't connect to http server running on the host from the container

[root@f41aa36665bd /]# curl 10.89.1.1:8000
curl: (7) Failed to connect to 10.89.1.1 port 8000 after 0 ms: Couldn't connect to server
[root@f41aa36665bd /]# curl 10.92.2.90:8000
curl: (7) Failed to connect to 10.92.2.90 port 8000 after 0 ms: Couldn't connect to server

although I can connect to it from the host itself:

[piotr@fedora]~/projects/podman% curl localhost:8000           
<!DOCTYPE HTML>
…

Container network:

[root@f41aa36665bd /]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo 
       valid_lft forever preferred_lft forever
2: eth0@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:3e:9e:b0:3b:ab brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.89.1.4/24 brd 10.89.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::503e:9eff:feb0:3bab/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

[root@f41aa36665bd /]# ip route
default via 10.89.1.1 dev eth0 proto static metric 100 
10.89.1.0/24 dev eth0 proto kernel scope link src 10.89.1.4

Host network:

[piotr@fedora]~/projects/podman% ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether cc:28:aa:a4:b4:c0 brd ff:ff:ff:ff:ff:ff
    altname enp6s0
    inet 10.92.2.90/24 metric 1024 brd 10.92.2.255 scope global dynamic eno1
       valid_lft 511866sec preferred_lft 511866sec
    inet6 fe80::ce28:aaff:fea4:b4c0/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

[piotr@fedora]~/projects/podman% ip route
default via 10.92.2.1 dev eno1 proto dhcp src 10.92.2.90 metric 1024 
10.92.2.0/24 dev eno1 proto kernel scope link src 10.92.2.90 metric 1024 
10.92.2.1 dev eno1 proto dhcp scope link src 10.92.2.90 metric 1024

SELinux is in permissive mode:

[piotr@fedora]~/projects/podman% sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

and firewall is disabled on host:

[piotr@fedora]~/projects/podman% systemctl status firewalld
○ firewalld.service - firewalld - dynamic firewall daemon
     Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead)
       Docs: man:firewalld(1)

Any help much appreciated as I couldn't find solution after spending a few hours reading through all I could find on the subject.

afbjorklund · 2024-10-02T11:07:32Z

afbjorklund
Oct 2, 2024

You should be able to reach host.containers.internal:8000 (formerly known as host.docker.internal)

4 replies

piotr-dobrogost Oct 2, 2024
Author

[root@f41aa36665bd /]# curl host.containers.internal:8000
curl: (6) Could not resolve host: host.containers.internal

That's because there's no host.containers.internal in /etc/hosts/:

[root@f41aa36665bd /]# cat /etc/hosts
127.0.0.1       localhost
::1     localhost
10.89.1.4       f41aa36665bd podman_fed_1

Trying to force it by adding

        extra_hosts:
            - thehost:host-gateway

to service definition in docker-compose.yaml results in the following error:

[piotr@fedora]~/projects/podman% podman compose up   
>>>> Executing external compose provider "/usr/bin/docker-compose". Please see podman-compose(1) for how to disable this message. <<<<

Removing podman_fed_1
Recreating f41aa36665bd_podman_fed_1 ... error

ERROR: for f41aa36665bd_podman_fed_1  preparing container 7670f3e035f7529efebfe09d80a6763f926549d727d4fa36d09db719543ca780 for attach: failed to create new hosts file: unable to replace "host-gateway" of host entry "thehost:host-gateway": host containers internal IP address is empty

ERROR: for fed  preparing container 7670f3e035f7529efebfe09d80a6763f926549d727d4fa36d09db719543ca780 for attach: failed to create new hosts file: unable to replace "host-gateway" of host entry "thehost:host-gateway": host containers internal IP address is empty
ERROR: Encountered errors while bringing up the project.
Error: executing /usr/bin/docker-compose up: exit status 1

I have no idea why Podman can't establish this IP address (maybe being rootless is the reason?).

afbjorklund Oct 2, 2024

Possibly a bug, or missing documentation about pasta (or whatever networking that docker-compose is using)

Host unreachable from container with bridge network on Podman v5 #22653

There are some hints about it in the blog post: (how to migrate from slirp4netns to pasta, and how to config...)

https://blog.podman.io/2024/03/podman-5-0-breaking-changes-in-detail/ # Pasta default for rootless networking

piotr-dobrogost Oct 2, 2024
Author

I've seen #22653 but did not read the whole thing as it's long and I wasn't sure it's something that would help in my case at the time. Now, returning to it and adding default_rootless_network_cmd = "slirp4netns" to ~/.config/containers/containers.conf fixes the problem. Btw, setting default_rootless_network_cmd = "pasta" does not solve the problem probably due to problems with pasta being discussed in that issue.
Thanks for help!

afbjorklund Oct 2, 2024

I was using podman4, which used slirp4netns by default (so didn't see the problem of podman5)

eriksjolund · 2024-10-02T13:11:08Z

eriksjolund
Oct 2, 2024

This PR

libpod: make use of new pasta option from c/common #23791

adds support for

host.containers.internal
host.docker.internal
--add-host=example.com:host-gateway

when using pasta.

As I understand it, this feature will hopefully be released in Podman 5.3.0

Currently you need to build the Podman git main branch to use the feature.
If you are running an older Podman version, you could also try using the pasta option --map-guest-addr
I verified that --map-guest-addr works for Podman 5.2.1. I would guess it works for even older Podman versions.
Note, when using --map-guest-addr , you need to add the chosen IP address into /etc/hosts yourself.

I wrote a minimal example of how the Podman (built from the git main branch) behaves:
https://github.com/eriksjolund/podman-networking-docs#example-connect-to-hosts-main-network-interface-using-pasta

3 replies

Luap99 Oct 2, 2024
Maintainer

Yes this will be in 5.3 in a few weeks, I have a WIP blog post about it that I plan to publish around the 5.3 release but I can post the text here already if you like to give me some early feedback:

As of Podman 5.0 we default to “pasta” as rootless networking application to provide network connectivity for the rootless containers. Compared to our previous default slirp4netns pasta works a bit differently, pasta by default does no NAT. This means it will copy the host address into the container as well which means both the container and host namespace use the same ip address and this in turn means if you try to connect to the host ip from the container it will refer to itself not the host. This caused quite some confusion for many users.

Also for quite some time podman provides the host.containers.internal host entry in containers to allow containers to use that in order to reach the host. Podman tried to be smart enough and not add the same host ip that is already used in the container namespace to avoid the issue. However in many cases the host only has one ip address (excluding localhost) so in this case Podman was unable to provide a suitable ip address for host.containers.internal thus the entry was not added.

As this was a common use case the pasta maintainers added a new pasta option called --map-guest-addr <ip> that allows the user choose any ip address inside the namespace to refer to the actual host ip on the host namespace. Starting with Podman 5.3 we will now make use of this by default as we use 169.254.1.2 as address which should be mapped to the host.
I also made sure that the host.containers.internal entry will be set to this ip which will make the hostname work again even if there is no secondary ip to use.

devurandom Nov 28, 2024

For reference: The final blog post was https://blog.podman.io/2024/10/podman-5-3-changes-for-improved-networking-experience-with-pasta/.

fsdrw08 Jan 29, 2025

Yes this will be in 5.3 in a few weeks, I have a WIP blog post about it that I plan to publish around the 5.3 release but I can post the text here already if you like to give me some early feedback:

As of Podman 5.0 we default to “pasta” as rootless networking application to provide network connectivity for the rootless containers. Compared to our previous default slirp4netns pasta works a bit differently, pasta by default does no NAT. This means it will copy the host address into the container as well which means both the container and host namespace use the same ip address and this in turn means if you try to connect to the host ip from the container it will refer to itself not the host. This caused quite some confusion for many users.

Also for quite some time podman provides the host.containers.internal host entry in containers to allow containers to use that in order to reach the host. Podman tried to be smart enough and not add the same host ip that is already used in the container namespace to avoid the issue. However in many cases the host only has one ip address (excluding localhost) so in this case Podman was unable to provide a suitable ip address for host.containers.internal thus the entry was not added.

As this was a common use case the pasta maintainers added a new pasta option called --map-guest-addr <ip> that allows the user choose any ip address inside the namespace to refer to the actual host ip on the host namespace. Starting with Podman 5.3 we will now make use of this by default as we use 169.254.1.2 as address which should be mapped to the host. I also made sure that the host.containers.internal entry will be set to this ip which will make the hostname work again even if there is no secondary ip to use.

How can I get the real host IP from the pod after podman 5.0?

piotr-dobrogost · 2024-10-02T19:04:45Z

piotr-dobrogost
Oct 2, 2024
Author

This means it will copy the host address into the container as well which means both the container and host namespace use the same ip address and this in turn means if you try to connect to the host ip from the container it will refer to itself not the host.

Looking from the noob perspective (I know nothing about pasta) what's the purpose of this ip sharing if it doesn't allow to connect from container to host?

I also made sure that the host.containers.internal entry will be set to this ip which will make the hostname work again even if there is no secondary ip to use.

What about host.docker.internal?

1 reply

Luap99 Oct 7, 2024
Maintainer

Looking from the noob perspective (I know nothing about pasta) what's the purpose of this ip sharing if it doesn't allow to connect from container to host?

Any application that refuses or cannot work across subnets will generally not work with our NAT'ed setups by default. Sure there are not many applications that this applies to but it is a nice thing for people that otherwise need more complicated SNAT rules and stuff like that.
If we do not do that it has to pick another random ip (e.g. slirp4netns using 10.0.2.100) so then you are also unable to connect to that host, be it a remote system or your actual host. In fact I am sure most user do not recognize that conflict as all as they are not using that ip but if you do use that ip you have the exact same problem as with pasta. In a sense pasta makes predictable as the communication to the host will never work and not only in some network environments (e.g. after you move from testing to prod) so that has also it benefits.

What about host.docker.internal?

The same thing it works exactly host.containers.internal

Can't connect to host from container when running podman compose with rootless Podman 5 #24133

Uh oh!

Uh oh!

piotr-dobrogost Oct 2, 2024

Replies: 3 comments · 8 replies

Uh oh!

Uh oh!

afbjorklund Oct 2, 2024

Uh oh!

Uh oh!

piotr-dobrogost Oct 2, 2024 Author

Uh oh!

Uh oh!

afbjorklund Oct 2, 2024

Uh oh!

piotr-dobrogost Oct 2, 2024 Author

Uh oh!

afbjorklund Oct 2, 2024

Uh oh!

Uh oh!

eriksjolund Oct 2, 2024

Uh oh!

Luap99 Oct 2, 2024 Maintainer

Uh oh!

devurandom Nov 28, 2024

Uh oh!

fsdrw08 Jan 29, 2025

Uh oh!

Uh oh!

piotr-dobrogost Oct 2, 2024 Author

Uh oh!

Luap99 Oct 7, 2024 Maintainer

Can't connect to host from container when running `podman compose` with rootless Podman 5 #24133

piotr-dobrogost
Oct 2, 2024

Replies: 3 comments 8 replies

afbjorklund
Oct 2, 2024

piotr-dobrogost Oct 2, 2024
Author

piotr-dobrogost Oct 2, 2024
Author

eriksjolund
Oct 2, 2024

Luap99 Oct 2, 2024
Maintainer

piotr-dobrogost
Oct 2, 2024
Author

Luap99 Oct 7, 2024
Maintainer