Run as non-root user with CAP_CHOWN on bind mount directories #16905

wilee8 · 2022-12-20T21:55:43Z

wilee8
Dec 20, 2022

We have a web interface that allows users to schedule batch jobs on a server, and it will chown files created so they are owned by the user that submitted the batch job. Right now it runs as a systemd service, and we have it set to run as a non-root user with CAP_CHOWN added to AmbientCapabilities so the service user can chown the files.

We're trying to containerize all the services in the interface so it stops breaking when the admin updates the system, and I have the chown working fine in a container created as root and ran in Podman by root. However, I'm trying to set it to run as a non-root user (usually with the -u flag in podman run, although I've tried other things as well) with CAP_CHOWN capabilities to modify the owner of files in a bind mounted directory, but it always fails at the chown command with an "Operation not permitted" error. Tried running setcap commands when building the container in Buildah, tried --cap-add and --privileged flags when running in Podman, can't get it to work any time I try to make the user someone other than root.

Is it possible to run the container as a non-root and give it CAP_CHOWN on bind-mounted directories? How would I need to configure this, both when building the container and when running it?

Answered by rhatdan

Dec 21, 2022

Let's simplify, this might be something that was fixed in newer versions of podman.

$ podman -v
podman version 4.3.1
$ podman run --rm --user bin --cap-add cap_chown fedora capsh --print
Current: cap_chown=eip
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap
Ambient set =cap_chown
Current IAB: ^cap_chown,!cap_dac_read_search,!cap_linux_immutable,!cap_net_broadcast,!cap_net_admin,!cap_net_raw,!cap_ipc_lock,!cap_ipc_owner,!cap_sys_module,!cap_sys_rawio,!cap_sys_ptrace,!cap_sys_pacct,!cap_sys_admin,!cap_sys_boot,!cap_sys_nice,!cap_sys_resource,!cap_sys_time,!cap_sys_tty_config,!cap_mknod,…

View full answer

rhatdan · 2022-12-20T22:22:28Z

rhatdan
Dec 20, 2022
Maintainer

Can you give us a simple repeater to play with?

2 replies

rhatdan Dec 20, 2022
Maintainer

Also can you give us the exact podman command you are running? Are you running the podman command as root with a --user XYZ option? Or attempting to run a rootless container?

Finally give output of podman info

wilee8 Dec 20, 2022
Author

Uh, I don't know what you mean by a simple repeater, sorry.
Current command, but I've changed it around many times without anything working:

podman run -d -v /opt/web-server:/ws_mount/:Z \
  --security-opt label=disable -v /etc/passwd:/etc/passwd:ro \
  -v /etc/group:/etc/group:ro --name django --pod ws --cap-add=cap_chown \
  --user=apache:apache localhost/django 8001

I'm running the podman command as root. I started trying with the --user flag, but I've also tried running it without the --user flag and changing the user in the entrypoint - that doesn't work either.
Podman info:

# podman info
host:
  BuildahVersion: 1.11.7
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.15-1.el7_8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.15, commit: 372b4a12f1c2df4f70c280d41173b60acd3f1260'
  Distribution:
    distribution: '"rhel"'
    version: "7.9"
  MemFree: 965128192
  MemTotal: 16655740928
  OCIRuntime:
    name: runc
    package: runc-1.0.0-69.rc10.el7_9.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 7423127552
  SwapTotal: 8455712768
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: dev02.myhost.com
  kernel: 3.10.0-1160.49.1.el7.x86_64
  os: linux
  rootless: false
  uptime: 7848h 15m 39.67s (Approximately 327.00 days)
registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 8
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 7
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

rhatdan · 2022-12-21T10:52:09Z

rhatdan
Dec 21, 2022
Maintainer

You are using an ancient version of Podman on RHEL7 is there anyway to move this to RHEL8 (Or even better RHEL9) so you could use a more recent version of Podman?

To diagnose the issue, first I would grep Cap /proc/self/status or easier run capsh --print within the container to make sure Podman is handling the capabiltiies correctly.

2 replies

wilee8 Dec 21, 2022
Author

Unfortunately, I'm stuck at RHEL7 on the production machines, that's out of my control.

Here's the output from capsh --print on a container using the podman run command above:

# podman exec -it django capsh --print
Current: =
Bounding set =
Ambient set =
Current IAB: !cap_chown,!cap_dac_override,!cap_dac_read_search,!cap_fowner,!cap_fsetid,!cap_kill,!cap_setgid,!cap_setuid,!cap_setpcap,!cap_linux_immutable,!cap_net_bind_service,!cap_net_broadcast,!cap_net_admin,!cap_net_raw,!cap_ipc_lock,!cap_ipc_owner,!cap_sys_module,!cap_sys_rawio,!cap_sys_chroot,!cap_sys_ptrace,!cap_sys_pacct,!cap_sys_admin,!cap_sys_boot,!cap_sys_nice,!cap_sys_resource,!cap_sys_time,!cap_sys_tty_config,!cap_mknod,!cap_lease,!cap_audit_write,!cap_audit_control,!cap_setfcap,!cap_mac_override,!cap_mac_admin,!cap_syslog,!cap_wake_alarm,!cap_block_suspend
Securebits: 00/0x0/1'b0 (no-new-privs=0)
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=48(apache) euid=48(apache)
gid=48(apache)
groups=
Guessed mode: UNCERTAIN (0)

Can't help but notice !cap_chown even though I included the --cap-add=cap_chown when running the container.

wilee8 Dec 21, 2022
Author

FWIW here's the same command after re-running the container in podman with the same run command above except the --user=apache:apache flag removed:

# podman exec -it django capsh --print
Current: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=eip
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Ambient set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Current IAB: ^cap_chown,^cap_dac_override,!cap_dac_read_search,^cap_fowner,^cap_fsetid,^cap_kill,^cap_setgid,^cap_setuid,^cap_setpcap,!cap_linux_immutable,^cap_net_bind_service,!cap_net_broadcast,!cap_net_admin,^cap_net_raw,!cap_ipc_lock,!cap_ipc_owner,!cap_sys_module,!cap_sys_rawio,^cap_sys_chroot,!cap_sys_ptrace,!cap_sys_pacct,!cap_sys_admin,!cap_sys_boot,!cap_sys_nice,!cap_sys_resource,!cap_sys_time,!cap_sys_tty_config,^cap_mknod,!cap_lease,^cap_audit_write,!cap_audit_control,^cap_setfcap,!cap_mac_override,!cap_mac_admin,!cap_syslog,!cap_wake_alarm,!cap_block_suspend
Securebits: 00/0x0/1'b0 (no-new-privs=0)
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=0(root) euid=0(root)
gid=0(root)
groups=
Guessed mode: UNCERTAIN (0)

It has cap_chown, and works correctly.

rhatdan · 2022-12-21T16:09:40Z

rhatdan
Dec 21, 2022
Maintainer

Let's simplify, this might be something that was fixed in newer versions of podman.

$ podman -v
podman version 4.3.1
$ podman run --rm --user bin --cap-add cap_chown fedora capsh --print
Current: cap_chown=eip
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap
Ambient set =cap_chown
Current IAB: ^cap_chown,!cap_dac_read_search,!cap_linux_immutable,!cap_net_broadcast,!cap_net_admin,!cap_net_raw,!cap_ipc_lock,!cap_ipc_owner,!cap_sys_module,!cap_sys_rawio,!cap_sys_ptrace,!cap_sys_pacct,!cap_sys_admin,!cap_sys_boot,!cap_sys_nice,!cap_sys_resource,!cap_sys_time,!cap_sys_tty_config,!cap_mknod,!cap_lease,!cap_audit_write,!cap_audit_control,!cap_mac_override,!cap_mac_admin,!cap_syslog,!cap_wake_alarm,!cap_block_suspend,!cap_audit_read,!cap_perfmon,!cap_bpf,!cap_checkpoint_restore
Securebits: 00/0x0/1'b0 (no-new-privs=0)
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=1(bin) euid=1(bin)
gid=1(bin)
groups=1(bin)
Guessed mode: UNCERTAIN (0)

3 replies

wilee8 Dec 21, 2022
Author

From the RHEL7 system I've been using:

# podman run --rm --user bin --cap-add cap_chown fedora capsh --print
Current: =
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Ambient set =
Current IAB: !cap_dac_read_search,!cap_linux_immutable,!cap_net_broadcast,!cap_net_admin,!cap_ipc_lock,!cap_ipc_owner,!cap_sys_module,!cap_sys_rawio,!cap_sys_ptrace,!cap_sys_pacct,!cap_sys_admin,!cap_sys_boot,!cap_sys_nice,!cap_sys_resource,!cap_sys_time,!cap_sys_tty_config,!cap_lease,!cap_audit_control,!cap_mac_override,!cap_mac_admin,!cap_syslog,!cap_wake_alarm,!cap_block_suspend
Securebits: 00/0x0/1'b0 (no-new-privs=0)
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=1(bin) euid=1(bin)
gid=1(bin)
groups=
Guessed mode: UNCERTAIN (0)
# podman --version
podman version 1.6.4

But I tried it on a RHEL8 system I have access to:

# podman run --rm --user bin --cap-add cap_chown fedora capsh --print
Current: cap_chown=eip
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_setfcap                                                                                                                    
Ambient set =cap_chown
Current IAB: ^cap_chown,!cap_dac_read_search,!cap_linux_immutable,!cap_net_broadcast,!cap_net_admin,!cap_ipc_lock,!cap_ipc_owner,!cap_sys_module,!cap_sys_rawio,!cap_sys_ptrace,!cap_sys_pacct,!cap_sys_admin,!cap_sys_boot,!cap_sys_nice,!cap_sys_resource,!cap_sys_time,!cap_sys_tty_config,!cap_mknod,!cap_lease,!cap_audit_write,!cap_audit_control,!cap_mac_override,!cap_mac_admin,!cap_syslog,!cap_wake_alarm,!cap_block_suspend,!cap_audit_read,!cap_perfmon,!cap_bpf,!cap_checkpoint_restore                                                                              
Securebits: 00/0x0/1'b0 (no-new-privs=0)
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=1(bin) euid=1(bin)
gid=1(bin)
groups=1(bin)
Guessed mode: UNCERTAIN (0)
# podman --version
podman version 4.2.0

So cap_chown is contained in the Ambient set when running in 4.2.0, but not when running in 1.6.4. I think the solution here is going to be "run as root until the prod systems get updated to RHEL8".

rhatdan Dec 21, 2022
Maintainer

SGTM, Better yet, get it to RHEL9. :^)

wilee8 Dec 21, 2022
Author

I wish. Thanks for the help.

Run as non-root user with CAP_CHOWN on bind mount directories #16905

Uh oh!

wilee8 Dec 20, 2022

Replies: 3 comments · 7 replies

Uh oh!

rhatdan Dec 20, 2022 Maintainer

Uh oh!

rhatdan Dec 20, 2022 Maintainer

Uh oh!

wilee8 Dec 20, 2022 Author

Uh oh!

rhatdan Dec 21, 2022 Maintainer

Uh oh!

wilee8 Dec 21, 2022 Author

Uh oh!

Uh oh!

wilee8 Dec 21, 2022 Author

Uh oh!

rhatdan Dec 21, 2022 Maintainer

Uh oh!

wilee8 Dec 21, 2022 Author

Uh oh!

rhatdan Dec 21, 2022 Maintainer

Uh oh!

wilee8 Dec 21, 2022 Author

wilee8
Dec 20, 2022

Replies: 3 comments 7 replies

rhatdan
Dec 20, 2022
Maintainer

rhatdan Dec 20, 2022
Maintainer

wilee8 Dec 20, 2022
Author

rhatdan
Dec 21, 2022
Maintainer

wilee8 Dec 21, 2022
Author

wilee8 Dec 21, 2022
Author

rhatdan
Dec 21, 2022
Maintainer

wilee8 Dec 21, 2022
Author

rhatdan Dec 21, 2022
Maintainer

wilee8 Dec 21, 2022
Author