Permission denied when starting .container file with quadlet on WSL2 on Windows 11

[thyeestes]

Issue Description

I have a .container file that I would like to start using systemctl --user start filename, but it fails with a permission denied error.

Steps to reproduce the issue

Steps to reproduce the issue

1. Install Ubuntu/Debian/Mint in WSL2
2. Install Podman
3. Create a container file
4. run systemctl --user daemon-reload
5. run systemctl --user start filename

Describe the results you received

Describe the results you receivedError: mkdir /sys/fs/cgroup/pids/user.slice/user-1000.slice/[email protected]/runtime: permission denied

Describe the results you expected

To start the container successfully

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon_2:2.1.8-0ubuntu22.04+obs16.2_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 99.74
    systemPercent: 0.09
    userPercent: 0.16
  cpus: 8
  databaseBackend: boltdb
  distribution:
    codename: victoria
    distribution: linuxmint
    version: "21.2"
  eventLogger: journald
  freeLocks: 2048
  hostname: NUC11PAHi7
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 100
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.90.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 14804717568
  memTotal: 16580014080
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.6.0-0ubuntu22.04+obs34.25_amd64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.6.0
    package: netavark_1.3.0-0ubuntu22.04+obs22.9_amd64
    path: /usr/libexec/podman/netavark
    version: netavark 1.3.0
  ociRuntime:
    name: crun
    package: crun_101:1.8.7-0ubuntu22.04+obs58.2_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.7
      commit: 53a9996ce82d1ee818349bdcc64797a1fa0433c4
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-0ubuntu22.04+obs11.2_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 4294967296
  swapTotal: 4294967296
  uptime: 0h 28m 12.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/thyestes/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/thyestes/.local/share/containers/storage
  graphRootAllocated: 1081101176832
  graphRootUsed: 2857050112
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/thyestes/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

WSL version: 1.2.5.0
Kernel version: 5.15.90.1
WSLg version: 1.0.51
MSRDC version: 1.2.3770
Direct3D version: 1.608.2-61064218
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22621.1702

Additional information

No response

[mheon]

Can you try with the systemd cgroup manager, instead of cgroupfs?

[thyeestes]

Sorry I don't know what you mean with that exactly?

[mheon]

In containers.conf on the Ubuntu WSL machine, please set cgroup_manager="systemd" - you should be able to find it at /etc/containers/containers.conf or /usr/share/containers/containers.conf

[thyeestes]

Check.

Changed the setting as suggested to systemd, however I think that is default, rebooted, same error

[vrothberg]

It seems WSL2 is using cgroups v1 where rootless Quadlet won't work. Can you retry running as root?

[wmaisonet]

Hello! Is there absolutely no way of making Quadlet work with a rootless user without cgroups v2? I have this exact problem in a RHEL 8 server, but I don't have root privileges to it.

[rhatdan]

If podman works the Quadlet should work I would think. Look at the podman generated command via /usr/libexec/quadlet --dryrun

[wmaisonet]

The problem is the generated systemd script has some flags in the ExecStart that are requiring elevated privileges. I don't know how to tell the .container file to not use those flags. I can start the container fine if I use podman run

ExecStart=/usr/bin/podman run --name=systemd-%N --cidfile=%t/%N.cid --replace --rm --log-driver passthrough --runtime /usr/bin/crun --cgroups=split --sdnotify=conmon -d

It is trying to create a file in root owned directories, that is why it fails...

Error: mkdir /sys/fs/cgroup/pids/user.slice/user-1000.slice/[email protected]/runtime: permission denied

[wmaisonet]

Found a fix in case anyone stumbles with this thread in the future...

in the Quadlet service-name.container file add the following below the [Container] options:
PodmanArgs=--cgroups=enabled --log-driver=journald

That got the service started without errors with the rootless user and cgroups v1

[thyeestes]

Running as root it works. No errors

[rhatdan]

I guess you need to convert your ubuntu/wsl to use cgroups V2 (Something I would advise) or run in rootful mode.

[thyeestes]

I switched to cgroupsV2 by adding [wsl2] kernelCommandLine = cgroup_no_v1=allI to .wslconfig, but then ended up with this error: Error: OCI runtime error: crun: cgroups in hybrid mode not supported, drop all controllers from cgroupv2

[thyeestes]

Okay tried a couple of other things and now it seems to work. What I did:
[wsl2] kernelCommandLine=systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all to .wslconfig

mount -t cgroup2 cgroup2 /sys/fs/cgroup -o rw,nosuid,nodev,noexec,relatime,nsdelegate to /etc/fstab

[rhatdan]

Great.

[thyeestes]

I need to do some further testing but Microsoft latest update to WSL2 seems to have fixed this issue