Unmounting container while attempting to delete storage: unmounting ...: invalid argument

[jennydaman]

Hello, I'm using rootless Podman + Quadlet. I get this error:

× go-vod.service - Nextcloud Memories Transcoder
     Loaded: loaded (/home/web/.config/containers/systemd/go-vod.container; generated)
     Active: failed (Result: exit-code) since Sun 2023-09-03 13:02:51 EDT; 58s ago
       Docs: https://memories.gallery/hw-transcoding/
    Process: 9843 ExecStart=/usr/bin/podman run --name=systemd-go-vod --cidfile=/run/user/1001/go-vod.cid --replace --rm --cgroups=split --network=nextcloud-aio --sdnotify=conmon -d -v nextcloud_aio_nextcloud_data:/mnt/ncdata:ro -v /var/slow/nextcloud:/var/slow/nextcloud:ro --cpus=1 --memory=2g localhost/pulsejet/go-vod:0.1.7-nvidia (code=exited, status=125)
    Process: 9886 ExecStopPost=/usr/bin/podman rm -f -i --cidfile=/run/user/1001/go-vod.cid (code=exited, status=0/SUCCESS)
   Main PID: 9843 (code=exited, status=125)
        CPU: 118ms

Sep 03 13:02:50 pyro systemd[1351]: Starting Nextcloud Memories Transcoder...
Sep 03 13:02:50 pyro go-vod[9843]: time="2023-09-03T13:02:50-04:00" level=warning msg="Unmounting container \"systemd-go-vod\" while attempting to delete storage: unmounting \"/home/web/.local/share/containers/storage/overlay/fbaa3a120702fa8d79dc8ef94754dc9f8d74e1749469f9438fa4cca3c1964cb7/merged\": invalid argument"
Sep 03 13:02:50 pyro go-vod[9843]: Error: removing storage for container "systemd-go-vod": unmounting "/home/web/.local/share/containers/storage/overlay/fbaa3a120702fa8d79dc8ef94754dc9f8d74e1749469f9438fa4cca3c1964cb7/merged": invalid argument
Sep 03 13:02:50 pyro podman[9843]: 2023-09-03 13:02:50.90622895 -0400 EDT m=+0.029555791 image pull 3ea80c81daef1ec42c049ea2411d907831dd8b1aa986ee1a377d4a9cd86701e4 localhost/pulsejet/go-vod:0.1.7-nvidia
Sep 03 13:02:50 pyro systemd[1351]: go-vod.service: Main process exited, code=exited, status=125/n/a
Sep 03 13:02:51 pyro systemd[1351]: go-vod.service: Failed with result 'exit-code'.
Sep 03 13:02:51 pyro systemd[1351]: Failed to start Nextcloud Memories Transcoder.

I was able to solve this problem by running

podman rm fbaa3a120702fa8d79dc8ef94754dc9f8d74e1749469f9438fa4cca3c1964cb7

I wanted to share this here in case anyone else encounters the same problem. I don't know how the problem happened.

[tmzullinger]

I ran into this as well. In a pod of 6 containers, 5 of them failed to start after a recent reboot with the same error. This was on Fedora 37 with podman-4.6.2-1.fc37.

Thanks for noting a work around!

[rhatdan]

I just merged a fix for this, where podman kube play --replace will force remove any left over containers with the same name.

Will be in podman 4.8.

[tmzullinger]

Thanks Dan!

Will this also affect pods started via podman pod start? I'm not using podman kube and was affected by this.

If there's an open issue for this, I'm happy to note it there instead. I noted this in the issue referenced in your recent commit (#20025).

[rhatdan]

I think only if the containers are all created with --replace.

[tmzullinger]

Okay, that should be the case for me. The systemd container service files were last generated with podman 4.2.1 and all include the --replace option. Thanks!

[mdemoss]

Started seeing these errors after updating to podman-2:4.6.1-5.0.1 from podman-2:4.4.1-13.0.1. I'm also running with podman-generated systemd unit files.

[triggerhappy17]

i also seen this issue after upgrading to podman 4.6.1 with podman generated systemd units

[GianniBYoung]

i also seen this issue after upgrading to podman 4.6.1 with podman generated systemd units

Same here on podman version 4.6.1.

I was able to resolve it by forcefully removing the problem image