Security implications of --cap-add=CAP_SYS_ADMIN for rootless containers?

[smcv]

For some context, I'm trying to use Podman as an alternative to lxc/lxd/Incus in Debian's automated testing framework, to test proposed OS-level packages (including daemons, etc.), without the performance overhead and portability issues of qemu/kvm or the "containers don't contain" issues of a rootful container. To be able to run automated tests against system services, we want the container to be running a full Debian system from init upwards, so that if the package under test has a missing/broken systemd unit, we can reproduce and report that bug.

By default, rootless podman runs the container payload (systemd and our packages under test) without CAP_SYS_ADMIN, which is ideal for security hardening of the boundary between the user and their container, but prevents some systemd unit options (hardening the boundary between systemd and individual system services) from working as intended. Depending on the option, either systemd ignores the option with a warning (which is very convenient, but makes our test container a less realistic example of a "normal" Debian system), or it refuses to start the service (systemd/systemd#29860, it's not currently clear to me whether this is considered to be a bug or whether it's working-as-intended).

As documented, we can get access to CAP_SYS_ADMIN in the container by adding --cap-add=CAP_SYS_ADMIN to the podman run command-line - but it isn't obvious to me what are the security implications of doing that. rootless.md says "the security implications should be carefully evaluated", which doesn't really help me either :-)

I am aware that Podman has several overlapping mechanisms intended to prevent the code inside a container from compromising confidentiality/integrity/availability on the host system, including dropping capabilities, entering a user namespace, entering a pid namespace, applying seccomp filters, and so on. --privileged turns many of these off, which I believe is assumed to allow root in the container to compromise confidentiality/integrity outside the container (but that isn't considered to be a security vulnerability, because that's the documented purpose of --privileged).

My question is: if we run a rootless podman container with --cap-add=CAP_SYS_ADMIN, but without other security-weakening options:

• is that assumed to make uid 0 in the container able to compromise the confidentiality/integrity of the user running podman outside the container, similar to --privileged, so that it's only valid to do this for a completely trusted container payload?
• or are Podman's other security mechanisms (seccomp, userns, etc.) designed to prevent uid 0 from compromising confidentiality/integrity outside the container even if they have CAP_SYS_ADMIN inside the container, and denying CAP_SYS_ADMIN is merely a hardening measure intended to mitigate possible vulnerabilities in e.g. Podman or the kernel?
• or something else?

[rhatdan]

Tough to say. CAP_SYS_ADMIN is the most powerful Linux Capability, although far less capable in a Rootless Container. You do get the power over the usernamespace and most powerfully over the mount namespace.

We mount lots of file systems to block access to the host, which can be unmounted with CAP_SYS_ADMIN. So there is a chance of exposure for kernel file system attacks. You might be able to break out and do not have SELinux controls, on Debian. Not sure if AppArmor can be modified with CAP_SYS_ADMIN, but probably.

Bottom line it would still be difficult for malicious code to attack the users processes, but less so with CAP_SYS_ADMIN.

If you trust the code you are testing, I think this is a reasonable tradeoff.

[smcv]

We mount lots of file systems to block access to the host, which can be unmounted with CAP_SYS_ADMIN

Ah, that's a good point that I hadn't thought of! So anything that podman wants to block that way is defeated.

[smcv]

Not sure if AppArmor can be modified with CAP_SYS_ADMIN, but probably

My understanding is that ability to alter the AppArmor profiles is gated by CAP_MAC_ADMIN instead, which should be the same as SELinux.

[smcv]

If you trust the code you are testing, I think this is a reasonable tradeoff

This is sounding like something that we should perhaps not enable by default, but only if the caller specifies --trust-root-in-testbed or something. A lot of the time we will trust the code under test (in the configuration that I expect to be most common, only packages from official Debian repositories will be accepted), but it seems like it would be better if that was a choice that the infrastructure operator had to acknowledge.

[rhatdan]

Well whoever is doing podman run --cap-add SYS_ADMIN is making the choice. This should never the the default.

[smcv]

This should never [be] the default

Right, I'd got that impression. The problem is that what our user will actually be invoking is a wrapper around podman which implements a particular interface required by the autopkgtest framework, and that has its own command-line options, plus a ---based mechanism to pass through arbitrary options to podman run - so in fact the command-line ends up being more like:

autopkgtest-virt-podman --init debian:sid -- --cap-add=CAP_SYS_ADMIN

which is pretty awkward and hard to remember/understand. Here, --init is the option meaning we run a full init system in the container (sharing only the kernel with the host system), similar to how lxc/lxd/Incus are normally used (as opposed to "it's a chroot but better", like the way Docker is normally used, which is what we do without --init).

So we might want some syntactic sugar like

autopkgtest-virt-podman --init --trust-root-in-testbed debian:sid

that ends up doing the same thing.

Part of what I was trying to establish by asking this question is whether the name of that option would need to be something as scary as --trust-root-in-testbed (I think your answer is: yes), and whether it would be reasonable to make --init automatically imply --cap-add=CAP_SYS_ADMIN (I think your answer is: no, because that makes the trust decision non-obvious).

[sbrivio-rh]

For some context, I'm trying to use Podman as an alternative to lxc/lxd/Incus in Debian's automated testing framework, to test proposed OS-level packages (including daemons, etc.), without the performance overhead and portability issues of qemu/kvm or the "containers don't contain" issues of a rootful container.

By the way, I'm looking at possible options to add autopkgtests for passt, which would surely need to include tests with Podman... which might run on top of Podman itself if this is implemented, so it's quite interesting. :)

• is that assumed to make uid 0 in the container able to compromise the confidentiality/integrity of the user running podman outside the container, similar to --privileged, so that it's only valid to do this for a completely trusted container payload?

One consideration I would like to add is that stuff like CVE-2022-0492 is definitely more likely to happen. I wonder how much it matters, though. You have Debian Developers and Debian Maintainers developing and triggering tests. If they had malicious intents, they could readily find ways to at least compromise the availability of the test runners, I suppose.

If there's no malicious intent, though, it's difficult to have container escapes happening by accident. All the related weaknesses and vulnerabilities I've seen in the past years are generally based on doing something rather particular with CAP_SYS_ADMIN. If you can trust the user to a given extent, I think that you're unlikely to compromise stability.

[smcv]

By the way, I'm looking at possible options to add autopkgtests for passt, which would surely need to include tests with Podman... which might run on top of Podman itself if this is implemented

Honestly, I think you'd probably be better off flagging those tests with isolation-machine so they only run in qemu or similar, instead of trying to nest containers - that's what I do for most of autopkgtest's own test coverage, such as the test that asserts that we can run tests under podman. There was a regression in the qemu backend caused by a kernel bug in 9pfs, but now that that's been worked around, it should be possible for the ci.debian.net maintainers to mark selected packages to have their tests run under qemu again (on architectures that support it, at least).

The use of lxc (or maybe podman) on ci.debian.net is mainly about getting faster testing with less resource use on the 99% of packages whose test suites work fine in a container, because the test suite doesn't actually need to do anything container-related itself.

[smcv]

If there's no malicious intent, though, it's difficult to have container escapes happening by accident

I think that could be a good answer for ci.debian.net, where the same people who can trigger tests and select packages to be tested are also already being trusted not to upload anything malicious to Debian itself. But, in the testing framework implementation, it would probably be less surprising for the default to be "code inside the testbed is kept at arm's length", and specific users of autopkgtest (like ci.debian.net) can relax that restriction if they want to.