Ryuk container can not be started on podman 4.6.2

[fedinskiy]

Issue Description

Ryuk container[1] which is essential for testcontainers library can not be started when using podman.

[1] https://hub.docker.com/r/testcontainers/ryuk

Steps to reproduce the issue

This works(I use custom docker socket for clarity): docker -H unix:///var/opt/custom/docker.sock run -v /var/opt/custom/docker.sock:/var/run/docker.sock -e RYUK_PORT=8080 -p 8080:8080 docker.io/testcontainers/ryuk:0.5.1
This does not: podman -H unix:///run/user/1000/podman/podman.sock run -v /run/user/1000/podman/podman.sock:/var/run/docker.sock -e RYUK_PORT=8080 -p 8080:8080 docker.io/testcontainers/ryuk:0.5.1

Describe the results you received

2023/09/29 13:02:07 Pinging Docker...
panic: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock: connect: permission denied

goroutine 1 [running]:
main.main()
	/app/main.go:131 +0x431

Describe the results you expected

$ docker  -H unix:///var/opt/custom/docker.sock run -v /var/opt/custom/docker.sock:/var/run/docker.sock -e RYUK_PORT=8080 -p 8080:8080 docker.io/testcontainers/ryuk:0.5.1
2023/09/29 13:05:37 Pinging Docker...
2023/09/29 13:05:37 Docker daemon is available!
2023/09/29 13:05:37 Starting on port 8080...
2023/09/29 13:05:37 Started!

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 94.29
    systemPercent: 1
    userPercent: 4.71
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "38"
  eventLogger: journald
  freeLocks: 1973
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.5.5-200.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 11582849024
  memTotal: 33378213888
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.9-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.9
      commit: a538ac4ea1ff319bcfe2bf81cb5c6f687e2dc9d3
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230908.g05627dc-1.fc38.x86_64
    version: |
      pasta 0^20230908.g05627dc-1.fc38.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc38.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 6h 18m 23.00s (Approximately 0.25 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  registry.access.redhat.com:
    Blocked: false
    Insecure: true
    Location: registry.access.redhat.com
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: registry.access.redhat.com
    PullFromMirror: ""
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - quay.io
store:
  configFile: /home/<...>/.config/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 0
    stopped: 7
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/<...>/.local/share/containers/storage
  graphRootAllocated: 510965841920
  graphRootUsed: 350122725376
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 341
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/<...>/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 1694549242
  BuiltTime: Tue Sep 12 22:07:22 2023
  GitCommit: ""
  GoVersion: go1.20.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Podman runs on an ordinary physical machine (laptop)

Additional information

Docker is run on a cloud VM, docker info ($ docker -H unix:///var/opt/custom/docker.sock info) results:

Client: Docker Engine - Community
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 6
  Running: 0
  Paused: 0
  Stopped: 6
 Images: 59
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 4.18.0-477.21.1.el8_8.x86_64
 Operating System: Red Hat Enterprise Linux 8.8 (Ootpa)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.515GiB
 Name: <...>
 ID: <...>
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: <...>
 Experimental: false
 Insecure Registries:
  <...>
 Registry Mirrors:
<...>
 Live Restore Enabled: false

[Luap99]

You would need to disable selinux for the container if you want to leak the podman socket, i.e. --security-opt label=disable to the podman run command.

[fedinskiy]

The thing is, ryuk container is not usually started directly from CLI, but via podman/docker socket by test-containers library. Is there an option to disable selinux for the socket generally? And why original docker is not affected by this?

[fedinskiy]

A-ha, the answer ('label=false' in containers.conf) can be found here: #9860 (reply in thread)

[fedinskiy]

Interesting. When I add this setting, the ryuk container can be started manually, but it still fails when working through the library:

1. git clone [email protected]:fedinskiy/reproducer.git -b reproducer/podman-testcontainers
2. mvn clean install
   Fails with:

Caused by: com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"cause":"netavark (exit code 1): code: 4, msg: Fatal: can't open lock file /run/xtables.lock: Permission denied\n","message":"netavark (exit code 1): code: 4, msg: Fatal: can't open lock file /run/xtables.lock: Permission denied\n","response":500}

There[1] is a recommendation to disable SELinux (despite it is already disabled in containers.conf) .
After running sudo setenforce 0, I see another error:

Caused by: com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"cause":"OCI permission denied","message":"crun: write to `/proc/self/oom_score_adj`: Permission denied: OCI permission denied","response":500}

[1] moby/moby#41230

[rhatdan]

Yes SELinux will block communications between a container and any socket leaked into the container. Whether this is Docker.sock or podman.sock or any other socket for that matter.

[rhatdan]

Docker is probably not enforcing SELinux while Podman is.
If the docker daemon is started with the --selinux-enabled flag then the containers would run as container_t and not be allowed to use the docker socket.