"bind mounts cannot have any filesystem-specific options applied" error

[andreaippo]

The cache mount I have used until today suddenly stopped working after a podman system prune (needed to free up some space).

This is the error I'm getting:

[1/2] STEP 6/6: RUN --mount=type=cache,target=/root/.m2 mvn clean install -Dmaven.test.skip=true
error running container: from /usr/bin/runc creating container for [/bin/sh -c mvn clean install -Dmaven.test.skip=true]: time="2025-04-16T09:57:25+02:00" level=error msg="runc create failed: invalid mount &{Source:/var/tmp/buildah3018145645/mnt/buildah-bind-target-9 Destination:/root/.m2 Device:bind Flags:20480 ClearedFlags:1 PropagationFlags:[1048576] Data:z Relabel: RecAttr:<nil> Extensions:0 IDMapping:<nil>}: bind mounts cannot have any filesystem-specific options applied"

This is the sample repo: https://github.com/andreaippo/demo-tracing

Once cloned I just run podman-compose up -d --build --force-recreate

I'm using SELinux enforcing:

❯ sestatus                                                                             
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      34

podman-info:

host:
  arch: amd64
  buildahVersion: 1.39.4
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.2.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: unknown'
  cpuUtilization:
    idlePercent: 92.89
    systemPercent: 2.21
    userPercent: 4.9
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: opensuse-tumbleweed
    version: "20250414"
  eventLogger: journald
  freeLocks: 2011
  hostname: andromeda
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.14.1-1-default
  linkmode: dynamic
  logDriver: journald
  memFree: 714014720
  memTotal: 32915935232
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.14.0-1.2.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark-1.14.1-1.1.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.1
  ociRuntime:
    name: runc
    package: runc-1.2.6-1.1.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.2.6
      commit: v1.2.6-0-ge89a29929c77
      spec: 1.2.0
      go: go1.24.2
      libseccomp: 2.6.0
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-20250320.32f6212-1.1.x86_64
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 34313084928
  swapTotal: 34359734272
  uptime: 1h 23m 43.00s (Approximately 0.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - registry.suse.com
  - docker.io
store:
  configFile: /home/andrea/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 5
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/andrea/.local/share/containers/storage
  graphRootAllocated: 826623201280
  graphRootUsed: 149394800640
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 35
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/andrea/.local/share/containers/storage/volumes
version:
  APIVersion: 5.4.2
  Built: 1743695904
  BuiltTime: Thu Apr  3 17:58:24 2025
  GitCommit: ""
  GoVersion: go1.24.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.2

Basically what I want to achieve is that repeated executions of the podman-compose command WON'T result in the first stage of my build to pull again all the maven dependencies from the Internet, every time.

I had found this --mount=type=cache tip on the web at some point, and it worked just fine.

Any other approach that achieves the same goal would be perfect.

I have tried the following, but I get the same error:

extra step: create cache dir: mkdir build-cache
then use type=bind in Dockerfile
RUN --mount=type=bind,src=build-cache,target=/root/.m2 mvn clean install -Dmaven.test.skip=true

but as I said, to no avail. Still same error.

Thanks

[andreaippo]

Just tried in a test VM running Fedora 42 and it works just fine:

host:
  arch: amd64
  buildahVersion: 1.39.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 92.48
    systemPercent: 2.83
    userPercent: 4.69
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: kde
    version: "42"
  eventLogger: journald
  freeLocks: 2037
  hostname: localhost-live
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.14.0-63.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 244072448
  memTotal: 8313331712
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.14.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark-1.14.1-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.1
  ociRuntime:
    name: crun
    package: crun-1.20-2.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.20
      commit: 9c9a76ac11994701dd666c4f0b869ceffb599a66
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250320.g32f6212-2.fc42.x86_64
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 8312598528
  swapTotal: 8313106432
  uptime: 0h 14m 1.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/andrea/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 5
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/andrea/.local/share/containers/storage
  graphRootAllocated: 41872785408
  graphRootUsed: 7856402432
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 23
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/andrea/.local/share/containers/storage/volumes
version:
  APIVersion: 5.4.1
  BuildOrigin: Fedora Project
  Built: 1741651200
  BuiltTime: Tue Mar 11 01:00:00 2025
  GitCommit: b79bc8afe796cba51dd906270a7e1056ccdfcf9e
  GoVersion: go1.24.0
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.1

Slightly different versions

[andreaippo]

Just tried on fedora 42 inside a VM, and it works just fine:

host:
  arch: amd64
  buildahVersion: 1.39.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 92.48
    systemPercent: 2.83
    userPercent: 4.69
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: kde
    version: "42"
  eventLogger: journald
  freeLocks: 2037
  hostname: localhost-live
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.14.0-63.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 244072448
  memTotal: 8313331712
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.14.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark-1.14.1-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.14.1
  ociRuntime:
    name: crun
    package: crun-1.20-2.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.20
      commit: 9c9a76ac11994701dd666c4f0b869ceffb599a66
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250320.g32f6212-2.fc42.x86_64
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 8312598528
  swapTotal: 8313106432
  uptime: 0h 14m 1.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/andrea/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 5
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/andrea/.local/share/containers/storage
  graphRootAllocated: 41872785408
  graphRootUsed: 7856402432
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 23
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/andrea/.local/share/containers/storage/volumes
version:
  APIVersion: 5.4.1
  BuildOrigin: Fedora Project
  Built: 1741651200
  BuiltTime: Tue Mar 11 01:00:00 2025
  GitCommit: b79bc8afe796cba51dd906270a7e1056ccdfcf9e
  GoVersion: go1.24.0
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.1

This caught my attention, opensuse (right) uses runc (which is also the executable throwing the error) while fedora (left) uses crun

[andreaippo]

Ok so maybe that's a problem with my distro's choice to go for runc, will check with maintainers if there was a recent switch from crun. Otherwise it might be a regression in that specific version of runc?

In any case now it works also on my daily OS (not VM), which is tumbleweed, by overriding the OCI runtime:

vi ~/.config/containers/containers.conf

Paste the following content:

[engine]
runtime = "/usr/bin/crun"

(found here).

man containers.conf will explain that the user/non root setting is at $XDG_CONFIG_HOME/containers/containers.conf

The value for the runtime prop should be set to the location provided by: whereis crun

[miry]

Thank you very much for providing the workaround. It really helps other folks.

[andreaippo]

I just built from source runc (version 1.3.0-rc.1+dev):

https://github.com/opencontainers/runc

And then used that executable for my ~/.config/containers/containers.conf:

[engine]
runtime = "/usr/local/sbin/runc"

And it works again.

So this is probably an issue with runc 1.2.6

[Luap99]

sounds like containers/buildah#6071

[andreaippo]

thanks, looks like it