Issues with pasta network stack, special setting and join an existing podman network

[dozenposture]

I'm trying to have this container to work with a VPN connection (WireGuard).
Container has been working fine with podman <5, so with slirp4netns. Now that I migrated to podman version 5.1.2, I'm having some issues with this container and the new pasta network stack.

This is what I see in the container logs when it starts without successfully connecting to the VPN (VPN endpoint and port have been masked with 123.123.123.123 and 1234:

[INF] [2024-08-13 10:09:37] [VPN] WireGuard is started.
[INF] [2024-08-13 10:09:37] [VPN] No modifications will be done to resolv.conf.
[INF] [2024-08-13 10:09:37] [VPN] resolv.conf is:
# Generated by resolvconf
nameserver 10.128.0.1
[INF] [2024-08-13 10:09:37] [VPN] WireGuard [wg0] endpoint [123.123.123.123][1234].
[INF] [2024-08-13 10:09:37] [VPN] WireGuard [wg0] allowed ips [0.0.0.0/0].
[INF] [2024-08-13 10:09:37] [VPN] Network [default][enp0s1][100][192.168.1.0/24].
[INF] [2024-08-13 10:09:37] [VPN] Added [192.168.1.0/24][LAN] as route via interface [enp0s1].
[INF] [2024-08-13 10:09:37] [VPN] Ports opened on [enp0s1] are [8080/tcp,8080/udp].
[INF] [2024-08-13 10:09:37] [VPN] Ports closed on [wg0] are [8080/tcp,8080/udp].
[INF] [2024-08-13 10:09:37] [VPN] Ports redirected on [wg0] are [].
[INF] [2024-08-13 10:09:37] [VPN] Routes overview:
default via 192.168.1.1 dev enp0s1 proto dhcp metric 100
192.168.1.0/24 via 192.168.1.1 dev enp0s1
192.168.1.0/24 dev enp0s1 proto kernel scope link metric 100
[INF] [2024-08-13 10:09:37] [VPN] Added firewall rules:
table inet hotio {
	chain forward {
		type filter hook forward priority filter; policy drop;
	}

	chain input {
		type filter hook input priority filter; policy drop;
		iifname "enp0s1" tcp dport 8080 ip daddr 0.0.0.100 counter accept
		iifname "wg0" tcp dport 8080 counter drop
		iifname "enp0s1" udp dport 8080 ip daddr 0.0.0.100 counter accept
		iifname "wg0" udp dport 8080 counter drop
		iifname "enp0s1" ip saddr 192.168.1.0/24 ip daddr 0.0.0.100 counter accept
		iifname "wg0" counter accept
		iifname "lo" counter accept
		icmp type echo-reply counter accept
		icmpv6 type echo-reply counter accept
		iifname "enp0s1" ip daddr 0.0.0.100 ip saddr 123.123.123.123 udp sport 1234 counter accept
	}

	chain output {
		type filter hook output priority filter; policy drop;
		oifname "enp0s1" tcp sport 8080 ip saddr 0.0.0.100 counter accept
		oifname "wg0" tcp sport 8080 counter drop
		oifname "enp0s1" udp sport 8080 ip saddr 0.0.0.100 counter accept
		oifname "wg0" udp sport 8080 counter drop
		oifname "enp0s1" ip saddr 0.0.0.100 ip daddr 192.168.1.0/24 counter accept
		oifname "wg0" counter accept
		oifname "lo" counter accept
		icmp type echo-request counter accept
		icmpv6 type echo-request counter accept
		oifname "enp0s1" ip saddr 0.0.0.100 ip daddr 123.123.123.123 udp dport 1234 counter accept
	}

	chain prerouting {
		type nat hook prerouting priority dstnat; policy accept;
	}
}
[INF] [2024-08-13 10:09:38] [VPN] Performing internet connectivity test...
[INF] [2024-08-13 10:09:38] [VPN] [IPV4] [PING: ]
[INF] [2024-08-13 10:10:57] [VPN] [IPV6] [PING: ]

As you can see both IPV4 and IPV6 ping are failing. Container starts, it's marked as healthy but I can't even connect to its web UI.

Reading the documentation, I passed the following settings into my quadlet file:
Network=pasta:-I,tap0,--ipv4-only,-a,10.0.2.0,-n,24,-g,10.0.2.2,--dns-forward,10.0.2.3,--no-ndp,--no-dhcpv6,--no-dhcp

With these settings, I can have the container to work, it connects to the VPN successfully (and the connection is way faster than with slirp4netns btw...).
These are the logs:

[INF] [2024-08-13 10:17:33] [VPN] No modifications will be done to resolv.conf.
[INF] [2024-08-13 10:17:33] [VPN] resolv.conf is:
# Generated by resolvconf
nameserver 10.128.0.1
[INF] [2024-08-13 10:17:34] [VPN] WireGuard [wg0] endpoint [123.123.123.123][1234].
[INF] [2024-08-13 10:17:34] [VPN] WireGuard [wg0] allowed ips [0.0.0.0/0].
[INF] [2024-08-13 10:17:34] [VPN] Network [default][tap0][10.0.2.0][10.0.2.0/24].
[INF] [2024-08-13 10:17:34] [VPN] Added [192.168.1.0/24][LAN] as route via interface [tap0].
[INF] [2024-08-13 10:17:34] [VPN] Ports opened on [tap0] are [8080/tcp,8080/udp].
[INF] [2024-08-13 10:17:34] [VPN] Ports closed on [wg0] are [8080/tcp,8080/udp].
[INF] [2024-08-13 10:17:34] [VPN] Ports redirected on [wg0] are [].
[INF] [2024-08-13 10:17:34] [VPN] Routes overview:
default via 10.0.2.2 dev tap0
10.0.2.0/24 dev tap0 proto kernel scope link src 10.0.2.0
192.168.1.0/24 via 10.0.2.2 dev tap0
[INF] [2024-08-13 10:17:34] [VPN] Added firewall rules:
table inet hotio {
	chain forward {
		type filter hook forward priority filter; policy drop;
	}

	chain input {
		type filter hook input priority filter; policy drop;
		iifname "tap0" tcp dport 8080 ip daddr 10.0.2.0 counter accept
		iifname "wg0" tcp dport 8080 counter drop
		iifname "tap0" udp dport 8080 ip daddr 10.0.2.0 counter accept
		iifname "wg0" udp dport 8080 counter drop
		iifname "tap0" ip saddr 10.0.2.0/24 ip daddr 10.0.2.0 counter accept
		iifname "wg0" counter accept
		iifname "lo" counter accept
		icmp type echo-reply counter accept
		icmpv6 type echo-reply counter accept
		iifname "tap0" ip daddr 10.0.2.0 ip saddr 123.123.123.123 udp sport 1234 counter accept
	}

	chain output {
		type filter hook output priority filter; policy drop;
		oifname "tap0" tcp sport 8080 ip saddr 10.0.2.0 counter accept
		oifname "wg0" tcp sport 8080 counter drop
		oifname "tap0" udp sport 8080 ip saddr 10.0.2.0 counter accept
		oifname "wg0" udp sport 8080 counter drop
		oifname "tap0" ip saddr 10.0.2.0 ip daddr 10.0.2.0/24 counter accept
		oifname "wg0" counter accept
		oifname "lo" counter accept
		icmp type echo-request counter accept
		icmpv6 type echo-request counter accept
		oifname "tap0" ip saddr 10.0.2.0 ip daddr 123.123.123.123 udp dport 1234 counter accept
	}

	chain prerouting {
		type nat hook prerouting priority dstnat; policy accept;
	}
}
[INF] [2024-08-13 10:17:34] [VPN] Performing internet connectivity test...
[INF] [2024-08-13 10:17:34] [VPN] [IPV4] [PING: 8.653/12.761/17.046 ms] [A_CITY, REGION, COUNTRY] [Global Layer network] [123.123.123.123]
[INF] [2024-08-13 10:17:38] [VPN] [IPV6] [PING: ]

With the settings above I can connect to the container WEB UI and everything works as expcted.

Now the issue is, I have to connect this container to an existing network and it seems I can't. I tried with --netns but no luck.
Maybe I passed the wrong option for --netns, I'm not sure.

How can I have that container to work with the VPN, pasta network stack and connect it to an existing network called my_network?

[Luap99]

There are network modes (e.g. pasta,slirp4netns,bridge...) and "custom/user-defined" networks (podman network ls). The bridge mode means it uses custom networks which default to the default network "podman".

In order to use network connect/disconnect you must a container with custom networks (aka bridge mode). Given you are using pasta (rootless default since v5) network connect/disconnect will not work. It is a either or situation.

If you want to use custom network as rootless it used a setup as shown there: #22943 (comment)
So this still uses a one pasta process, if you like to pass custom options to that process you must set these options in containers.conf via pasta_options = [] under the [Network] section. When you do that your containers on custom network can then reach your host ip.

[sbrivio-rh]

Cross-posted with https://www.reddit.com/r/podman/comments/1epjdlu/how_can_i_specify_a_network_name_to_join_together/ (no links added on either side...).

[dozenposture]

Thanks for the reply.
For me it's rather hard to understand what's happening and why would I need to pass extra options in the quadlet file to have that container to work, to begin with.

I think what I would like to achieve is rather simple.
1 - different containers in the same network. This is already done, I just can't add this last one (since extra pasta options are needed...)
2 - getting that container to work with the VPN. I'm really not sure why I need to specify the interface name (maybe because is a wireguard connection and tap0 is mandatory?) and most importantly I don't get why I would need to specify a subnet and a gateway.

Thanks for the diagram @Luap99 but I think it's really beyond my competences at the moment. I simply can't fully digest it.

What are the steps I have to take to achieve what I'm trying to achieve?

[dozenposture]

Ok, this is what I did:
I created a file called 010_network.conf in /home/podman/.config/containers/containers.conf.d and this is its content:

[Network]
pasta_options = ["-I","tap0","--ipv4-only","-a","10.0.2.0","-n","24","-g","10.0.2.2","--dns-forward","10.0.2.3","--no-ndp","--no-dhcpv6","--no-dhcp"]

This are the config that I was passing in the quadlet file.

I created a network with podman network create my_network.
In the quadlet definition I changed the line Network=pasta:-I,tap0,--ipv4-only,-a,10.0.2.0,-n,24... to Network=my_network and I can see the container connected to the my_network network, I can reach it from another container in the same network using its container name and the VPN works! 🎉

Now, as I'm going down to the rabbit hole, if I do podman inspect qbittorrent I see this:

"Networks": {
                    "my_network": {
                         "EndpointID": "",
                         "Gateway": "10.89.0.1",
                         "IPAddress": "10.89.0.7",
                         "IPPrefixLen": 24,

Shouldn't the gateway be 10.0.2.2 as specified in the ~.config/containers/containers.conf.d/010_network.conf file?
Also, the IP address seems using a 10.89... instead of 10.0....

Also, if I inspect the container:

bc01a036da63:/# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo
       valid_lft forever preferred_lft forever
2: eth0@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether aa:5e:32:4a:56:4d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.89.0.7/24 brd 10.89.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::a85e:32ff:fe4a:564d/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
3: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1320 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none
    inet 10.184.199.252/32 scope global wg0
       valid_lft forever preferred_lft forever

I don't see any tap0 interface as specified in the mentioned config file.

Why is that? Why the container IP doesn't match the pasta option given range and why the interface has a different name?

Can someone please ELI5 to me, I'm really trying hard to learn these things! :)

Is what I did good/acceptable or am I missing something else?

[Luap99]

Yes that is what I meant by adding the options to containers.conf

Again see the graph at #22943 (comment), look at where the tap0 interface is (podman unshare --rootless-netns can be used to enter the rootless netns). The actual containers all have their own netns and are then connected via a bridge and veth pair as defined by your network (podman network inspect my_network shows you the ip information for the containers)

So all the traffic is routed from the container to the rootless-netns and from there via the pasta tap interface to the host namespace and then from there like any other "normal" program on the host.

[dozenposture]

Thank you very much both @Luap99 and @sbrivio-rh for your help and extra documentation/links. Learned a lot while troubleshooting this!
This is what happened last evening during some testing:
I simply created a podman network (no options, simply a name) and attached the container subject of this thread. With the container attached to the network, no extra pasta options were needed (like in a file in containers.conf.d)!!! The container started immediately and it could reach the internet and connect to the VPN.
Why is that? This shows that the only extra pasta relevant options were the ip-range and the gateway. But as said, with the container attached to a podman network, it simply works.
This was not the case with podman <5 (slirp4netns). The same container not attached to any network was simply working out of the box.

I'm rather confused at the moment...