Unable to access /var/log on host

[thyeestes]

Issue Description

I'm unable to access the log files in /var/log of the host, in my podman container.

I'm running rootless podman, but the service in the container (Crowdsec) is running as root. As I understood root in the container maps to my user outside the container (uid 1000)

From the host itself I can access the log files adn my user account is member of the adm group, but in the container I get permission denied

Steps to reproduce the issue

Steps to reproduce the issue

1. add volume via - /var/log:/var/log:ro
2. try to read a log file located in /var/log from the container

Describe the results you received

Permission denied and ls -la shows:
-rw-r----- 1 nobody nobody 2940399 May 16 15:26 ufw.log
-rw-r----- 1 nobody nobody 4151577 May 14 00:00 ufw.log.1

Describe the results you expected

I expected to be able to read the logs inside the container as I am able to read it outside the container and my uid is mapped to the root user inside the container

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2:2.1.7-0ubuntu22.04+obs15.24_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 76.86
    systemPercent: 6.62
    userPercent: 16.52
  cpus: 8
  databaseBackend: boltdb
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  hostname: NUC11PAHi7
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.19.0-41-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 632045568
  memTotal: 33269129216
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_101:1.8.4-0ubuntu22.04+obs55.10_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.4
      commit: 5a8fa99a5e41facba2eda4af12fa26313918805b
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-0ubuntu22.04+obs10.63_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 1672736768
  swapTotal: 2147479552
  uptime: 92h 48m 42.00s (Approximately 3.83 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/thyestes/.config/containers/storage.conf
  containerStore:
    number: 31
    paused: 0
    running: 31
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/thyestes/.local/share/containers/storage
  graphRootAllocated: 982820896768
  graphRootUsed: 369075519488
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 32
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/thyestes/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

What are the permissions on the host? It looks like "others" have no read access to this dir so only root and the group can read it.

[thyeestes]

This are the host permissions:
-rw-r----- 1 syslog adm 2,9M mei 16 15:39 ufw.log
-rw-r----- 1 syslog adm 4,0M mei 14 00:00 ufw.log.1

[Luap99]

I assume you are part of the adm group?
Containers by default will drop all supplementary groups so you will loose access to anything that is not mapped into the userns. You can use --group-add keep-groups to work around that but keep in mind that the users/group in the container will still show up as nobody because it is not mapped.
https://docs.podman.io/en/latest/markdown/podman-create.1.html#group-add-group-keep-groups

[thyeestes]

Compose cannot use keep-groups as far as I know, at least I get an error saying it is unable to find that group.

[Luap99]

If compose can use annotations you can set run.oci.keep_original_groups=1 which should result in the same, I convert this to a discussion as this is not a podman bug,

[rhatdan]

Only add run.oci.keep_original_groups=1 annotation to the particular container that needs access.

[thyeestes]

That makes sense, but where in the yml file do I have to declare it, that it only applies to 1 container in my pod?

I tried moving it underneath container like this, but that doesn't work.
spec:
container:
- metadata:
annotations:
run.oci.keep_original_groups: "1"

[rhatdan]

@umohnani8 PTAL

[umohnani8]

annotations can only be set at the pod or deployment level and not for a specific container. Move the annotation under the Pod or Deployment spec and it should work, but it will apply it to all the containers in the pod I believe.

If we want to support adding this to only one container, we could extend the annotation to take a container name so it knows to only apply the annotation to that specific container.

[thyeestes]

Thanks. IMHO a useful feature to have the possibility for per container annotations. Curious to hear your opinion.

[thyeestes]

I honestly don't know, but thanks for your quick replies.