weird podman API client behavior: unit test unwantingly getting switched into user ns created by catatonit

[thediveo]

While working on the Podman-awareness of my Linux namespaces+container discovery tool I've noticed one podman-related unit test to behave weird: when the tests run they always run as uid 0, even if the unit test has been started was under my own user uid 1000.

• Linux Ubuntu 22.04.1LTS

• $ podman version
  
  Client:       Podman Engine
  Version:      4.2.0
  API Version:  4.2.0
  Go Version:   go1.19
  Built:        Thu Jan  1 01:00:00 1970
  OS/Arch:      linux/amd64
  

• podman 4.2.0 was installed from the Debian experimental podman package.

Please note that my code base imports podman v3, not v4!

The user namespaces in my system are as in this screenshot (please excuse me for touting my own lxkns horn):

The user namespaces relevant to us here are:

• user:[...1837] the initial user namespace
• user:[...2717] the user namespace for my user uid 1000, created by the "catatonit" process.

Only lately I've noticed a strange behavior of my podman-related test https://github.com/thediveo/lxkns/blob/master/decorator/podman/decorator_test.go: when run as effective uid != 0 then this test should be skipped. However, running this test go test -v -tags podman ./decorator/podman doesn't skip the test but instead makes it fail when trying to connect to the system podman service at unix:///run/podman/podman.sock. After some head scratching I've put the following code into my test:

func TestPodmanDecorator(t *testing.T) {
	b, _ := exec.Command("ls", "-l", "/proc/self/ns").Output()
	panic(string(b))
}

Now, running the test again as an ordinary user:

$ go test -v -tags podman ./decorator/podman/
=== RUN   TestPodmanDecorator
--- FAIL: TestPodmanDecorator (0.00s)
panic: total 0
lrwxrwxrwx 1 root root 0 Sep 20 21:33 cgroup -> cgroup:[4026531835]
lrwxrwxrwx 1 root root 0 Sep 20 21:33 ipc -> ipc:[4026531839]
lrwxrwxrwx 1 root root 0 Sep 20 21:33 mnt -> mnt:[4026532718]
lrwxrwxrwx 1 root root 0 Sep 20 21:33 net -> net:[4026531840]
lrwxrwxrwx 1 root root 0 Sep 20 21:33 pid -> pid:[4026531836]
lrwxrwxrwx 1 root root 0 Sep 20 21:33 pid_for_children -> pid:[4026531836]
lrwxrwxrwx 1 root root 0 Sep 20 21:33 time -> time:[4026531834]
lrwxrwxrwx 1 root root 0 Sep 20 21:33 time_for_children -> time:[4026531834]
lrwxrwxrwx 1 root root 0 Sep 20 21:33 user -> user:[4026532717]
lrwxrwxrwx 1 root root 0 Sep 20 21:33 uts -> uts:[4026531838]

The unit test is run inside user:[...2717] and that's the "catatonit" user namespace 🤯

How comes? Can someone please explain to me why importing podman v3 code and before even calling some binding functions sends my applications into a different user namespace? I've noticed that at the time my initializer runs, podman code has already switched into the different user namespace.

I tried to reproduce this behavior with just an empty main() and dash-importing podman v3 packages, but without any success.

However, the same behavior can also be reproduced with the smaller sealwatcher module. Here, go test -v ./podman as a non-user triggers the same problem.

[mheon]

This looks like the rootless user namespace, held open by a catatonit pause process. However, I'm not sure why the bindings require it.

@giuseppe Does this look like a bug to you? I don't see a reason why an API client needs to join the rootless user namespace.

[thediveo]

go mod why github.com/containers/storage/pkg/reexec

# github.com/containers/storage/pkg/reexec
github.com/thediveo/sealwatcher/podman
github.com/containers/podman/v3/pkg/bindings/containers
github.com/containers/storage/pkg/archive
github.com/containers/storage/pkg/unshare
github.com/containers/storage/pkg/reexec

[mheon]

I doubt it's reexec (though that also should not be dragged in either), but rather SetupRootless() which joins the rootless user namespace using pkg/rootless

[thediveo]

go mod why github.com/containers/podman/v3/pkg/rootless

# github.com/containers/podman/v3/pkg/rootless
github.com/thediveo/sealwatcher/test
github.com/containers/podman/v3/pkg/specgen
github.com/containers/podman/v3/pkg/rootless

[thediveo]

is there some way to detect that my unit test has ended up "rootless"? So that I can at least skip the test for the moment?

[Luap99]

Yes pkg/rootless causes a setns if the pause process is already running, I think that is why we need the remote build tag.

[Luap99]

I think we have to rewrite pkg/rootless. It should never be used in the binding or remote client.

It should not matter if the client runs rootless or as root. Only he server is the source of truth.