systemd NetworkNamespacePath for rootful podman run container

[chetan-reddy]

On Debian 12 for the past two years I've used NetworkNamespacePath=/run/netns/myvpn to launch containers inside my vpn(wireguard) network namespace. When I upgraded to Debian 13, I started seeing this message whenever i ran ip addr in the myvpn namespace

Error: Peer netns reference is invalid.

On occasion the container would fail to exit cleanly and an orphaned interface would remain in the myvpn namespace. A little googling led me to a systemd change between Debian 12 and 13 which now necessitates an additional PrivateMounts=no for podman run to work well inside the network namespace. I have two questions about this:

1 - Is it safe/supported for podman run to be invoked inside a network namespace that is not the root namespace? PrivateMounts=no causes the sysfs to have incorrect network device info - does that negatively affect podman?

2 - Since only netavark needs to run inside the vpn network namespace and not the rest of podman, is it possible to run only netavark inside the namespace and not all of podman? Do quadlet's .network files allow for specifying a NetworkNamespacePath for the network?

I was using podman version 5.5.2 (from guix) on both Debian 12 and Debian 13.

[chetan-reddy]

Here's a simple way to reproduce the issue.

sudo ip netns add testns
sudo systemd-run --wait -t -p NetworkNamespacePath=/run/netns/testns podman run -it --rm docker.io/debian:bookworm-slim sleep infinity

In a different terminal

$ sudo ip -n testns addr
1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: podman0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 0e:07:76:64:cf:07 brd ff:ff:ff:ff:ff:ff
inet 10.88.0.1/16 brd 10.88.255.255 scope global podman0
valid_lft forever preferred_lft forever
inet6 fe80::c07:76ff:fe64:cf07/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
3: veth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master podman0 state UP group default qlen 1000
Error: Peer netns reference is invalid.
link/ether 72:07:ac:97:4a:84 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::7007:acff:fe97:4a84/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever
$ sudo podman rm -f -l
WARN[0010] StopSignal SIGTERM failed to stop container eager_chandrasekhar in 10 seconds, resorting to SIGKILL
Error: cleaning up container 0070a4794079ee5c464528960071dc0708359bf335ea720d3506a84b75784e40: removing container 0070a4794079ee5c464528960071dc0708359bf335ea720d3506a84b75784e40 network: netavark: setns: IO error: Invalid argument (os error 22)
$ sudo ip -n testns addr
1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: podman0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 0e:07:76:64:cf:07 brd ff:ff:ff:ff:ff:ff
inet 10.88.0.1/16 brd 10.88.255.255 scope global podman0
valid_lft forever preferred_lft forever
inet6 fe80::c07:76ff:fe64:cf07/64 scope link proto kernel_ll
valid_lft forever preferred_lft forever

The podman rm command fails with the same netavark error message even if i prefix it with nsenter --net=/run/netns/testns

[Luap99]

PrivateMounts=no is 100% a requirement, your reproducer doesn't seem to use it.
When you run podman in different mount namespaces you basically break all sort of mount tracking.

Different network namespaces can work but it means you must be very very careful from where you run podman commands. You cannot run podman stop/rm from a different netns and then expect the cleanup to work. Podman has no idea what the "host" namespace originally was when it created the container so it must be the same when stopping so it finds the right interfaces.

In general I would strongly recommend against doing such things. If you want the container to join the vpn namespace you can use something like --network ns:/run/netns/testns instead (which isn't the same since the container will simply join the namespace instead of doing the bridge network setup via netavark)

sysfs should not matter as we don't use it netvark, we do use /proc/sys/net tough to configure network sysctl's but AFAIK that does not require a mount namespace, it switches properly with the netns.

Maybe a better option is to use vrf interfaces to route the container bridge traffic via your vpn. podman network create supports setting --opt vrf=name which causes netavark to attach the bridge to a vrf.

2 - Since only netavark needs to run inside the vpn network namespace and not the rest of podman, is it possible to run only netavark inside the namespace and not all of podman? Do quadlet's .network files allow for specifying a NetworkNamespacePath for the network?

No that is not how things work at all, the network create command is simply creating a json file it doesn't do anything in terms of network setup so setting namespaces there is pointless.

[chetan-reddy]

Thanks @Luap99 . I will look into the vrf option further to see if it can work for me. Directly joining with --network ns:/run/netns/vpnns is not an option for me because the runtime i use (gvisor's runsc) expects to take over the interfaces in the network namespace specified in the bundle's config.json. podman's bridge networking has been working well for me even in the vpn namespace. It continues to work well with the added PrivateMounts=no on Debian 13. Now that you've clarified that the incorrect network device info in sysfs doesn't really matter to podman, I will keep doing what I've been doing.
I do wonder though, is it possible to launch netavark independently of podman inside the vpnns to have it create the bridge, nftables rules, container network namespace etc ; and then run podman in the root namespace with --network ns:/run/netns/created_by_netavark_with_a_bridge_in_vpnns.

[Luap99]

I do wonder though, is it possible to launch netavark independently of podman inside the vpnns to have it create the bridge, nftables rules

Sure it is just a command that podman executes so you can do it yourself, but it isn't really supported or recommend to do so.
If you run podman with --log-level trace you should see the command with json payload that we execute, though the json format isn't really nice to use. And the podman side manages the ip tracking currently so you would need to assign the static ips yourself.

[chetan-reddy]

Thanks again @Luap99 . I created a ticket asking if podman could support specifying the network namespace the bridge is created in. I expect it'll be closed as a wontfix since I already have a solution that works, but just putting the idea out there. Thank you for thoroughly answering my questions.