how to debug podman cannot run, Error: remount /data/containers/storage/overlay, flags: 0x40000: invalid argument

[LeoQuote]

After some time of running , podman will failed to execute any command, including podman version and podman info.

sudo podman info
Error: remount /data/containers/storage/overlay, flags: 0x40000: invalid argument

however, the rootless podman works well

podman version
Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.14
Built:        Thu Jan  1 08:00:00 1970
OS/Arch:      linux/amd64

I found a issue comment maybe related to this issue , this guy has exactly same issue with me #8647 (comment) but I see no solutions

after a machine reboot, this can be fixed, It is really strange to see this error message , what should I do to troubleshoot this ?

uname -a

Linux nob66 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64 GNU/Linux

podman info

host:
  arch: amd64
  buildahVersion: 1.19.4
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 64
  distribution:
    distribution: debian
    version: "10"
  eventLogger: file
  hostname: nob111
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.19.0-18-amd64
  linkmode: dynamic
  memFree: 17626902528
  memTotal: 134384787456
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.18.1-7931a-dirty
      commit: 7931a1eab0590eff4041c1f74e2844b297c31cea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 626h 37m 47.8s (Approximately 26.08 days)
registries:
  search:
  - no-relevant
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /data/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /data/containers/storage
  volumePath: /data/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 08:00:00 1970
  GitCommit: ""
  GoVersion: go1.14
  OsArch: linux/amd64
  Version: 3.0.1

other info:

1. This machine has two diffrent disk , /dev/vda1 and /dev/vdb mounted at / and /data seperately .
2. /dev/vda1 is formatted as ext4 while /dev/vdb is xfs
3. in correct machine , containers mount should be like /dev/vdb on /data/containers/storage/overlay type xfs (rw,noatime,attr2,inode64,noquota) while malfunctioning machine is /dev/vda1 on /data/containers/storage/overlay type ext4 (rw,relatime,errors=remount-ro)
4. I updated runc but no work. This error still shows up

[rhatdan]

@giuseppe Thoughts?

[giuseppe]

it could be helpful to see the output of findmnt -R -o PROPAGATION,TARGET / and podman unshare findmnt -R -o PROPAGATION,TARGET /

[LeoQuote]

I can provide parts of the output of findmnt as most of it is k8s-related mount (configmap or secret)

private     |-/data/containers/storage/overlay
private     | `-/data/containers/storage/overlay/9d064045add2087ddd1487f26f934294934a2399e4a84f2a67f002290a8da275/merged
shared      |-/data/containers/storage/overlay-containers/bea9951c1bfa8dd30065f0055d59a918a20e8976c1a4881123fb575cecf518c3/userdata/shm
shared      |-/data

and podman unshare also return the same error

sudo podman unshare findmnt -R -o PROPAGATION,TARGET /
Error: remount /data/containers/storage/overlay, flags: 0x40000: invalid argument

[giuseppe]

who is creating a private mount on /data/containers/storage/overlay on the host? Is it used by another podman instance?

[LeoQuote]

What do you think I can do to determine this question ? I have deployed k8s on this specific machine.