podman system migrate crashes

[shibumi]

Issue Description

Running podman system migrate crashes:

$ podman system migrate
stopped dd4ac350873cfe9d6e5c5673150af49861a512e404ab9a50dc54632596aa9a8f
WARN[0010] StopSignal SIGTERM failed to stop container vigilant_cerf in 10 seconds, resorting to SIGKILL 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1590adb]

goroutine 1 [running]:
github.com/containers/podman/libpod.(*storageService).UnmountContainerImage(0x0, {0xc000681940?, 0x1b?}, 0x0)
        github.com/containers/podman/libpod/storage.go:235 +0x3b
github.com/containers/podman/libpod.(*Container).unmount(0xc0003a9540, 0x40?)
        github.com/containers/podman/libpod/container_internal.go:2488 +0x3a
github.com/containers/podman/libpod.(*Container).cleanupStorage(0xc0003a9540)
        github.com/containers/podman/libpod/container_internal.go:2019 +0x4c5
github.com/containers/podman/libpod.(*Container).cleanup(0xc0003a9540, {0x20c0c00, 0x2f40760})
        github.com/containers/podman/libpod/container_internal.go:2114 +0x5fb
github.com/containers/podman/libpod.(*Container).waitForConmonToExitAndSave(0xc0003a9540)
        github.com/containers/podman/libpod/container_internal.go:1517 +0x4c5
github.com/containers/podman/libpod.(*Container).stop(0xc0003a9540, 0xa)
        github.com/containers/podman/libpod/container_internal.go:1459 +0x5c5
github.com/containers/podman/libpod.(*Container).StopWithTimeout(0xc0003a9540?, 0xa?)
        github.com/containers/podman/libpod/container_api.go:320 +0xce
github.com/containers/podman/libpod.(*Container).Stop(...)
        github.com/containers/podman/libpod/container_api.go:290
github.com/containers/podman/libpod.(*Runtime).Migrate(0xc000102540, {0x0, 0x0})
        github.com/containers/podman/libpod/runtime_migrate_linux.go:76 +0x2e5
github.com/containers/podman/pkg/domain/infra/abi.(*ContainerEngine).Migrate(0x0?, {0x0?, 0xcb2006ca48b37857?}, {{0x0?, 0x0?}})
        github.com/containers/podman/pkg/domain/infra/abi/system.go:307 +0x25
github.com/containers/podman/cmd/podman/system.migrate(0x2e3a000?, {0x2f40760?, 0x0?, 0x0?})
        github.com/containers/podman/cmd/podman/system/migrate.go:56 +0x45
github.com/spf13/cobra.(*Command).execute(0x2e3a000, {0xc00003ea40, 0x0, 0x0})
        github.com/spf13/cobra/command.go:989 +0xabb
github.com/spf13/cobra.(*Command).ExecuteC(0x2e22d20)
        github.com/spf13/cobra/command.go:1117 +0x44f
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        github.com/spf13/cobra/command.go:1034
main.Execute()
        github.com/containers/podman/cmd/podman/root.go:119 +0x57
main.main()
        github.com/containers/podman/cmd/podman/main.go:62 +0x4b2

Steps to reproduce the issue

I don't know how to reproduce this issue.

Describe the results you received

$ podman system migrate
stopped dd4ac350873cfe9d6e5c5673150af49861a512e404ab9a50dc54632596aa9a8f
WARN[0010] StopSignal SIGTERM failed to stop container vigilant_cerf in 10 seconds, resorting to SIGKILL 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1590adb]

goroutine 1 [running]:
github.com/containers/podman/libpod.(*storageService).UnmountContainerImage(0x0, {0xc000681940?, 0x1b?}, 0x0)
        github.com/containers/podman/libpod/storage.go:235 +0x3b
github.com/containers/podman/libpod.(*Container).unmount(0xc0003a9540, 0x40?)
        github.com/containers/podman/libpod/container_internal.go:2488 +0x3a
github.com/containers/podman/libpod.(*Container).cleanupStorage(0xc0003a9540)
        github.com/containers/podman/libpod/container_internal.go:2019 +0x4c5
github.com/containers/podman/libpod.(*Container).cleanup(0xc0003a9540, {0x20c0c00, 0x2f40760})
        github.com/containers/podman/libpod/container_internal.go:2114 +0x5fb
github.com/containers/podman/libpod.(*Container).waitForConmonToExitAndSave(0xc0003a9540)
        github.com/containers/podman/libpod/container_internal.go:1517 +0x4c5
github.com/containers/podman/libpod.(*Container).stop(0xc0003a9540, 0xa)
        github.com/containers/podman/libpod/container_internal.go:1459 +0x5c5
github.com/containers/podman/libpod.(*Container).StopWithTimeout(0xc0003a9540?, 0xa?)
        github.com/containers/podman/libpod/container_api.go:320 +0xce
github.com/containers/podman/libpod.(*Container).Stop(...)
        github.com/containers/podman/libpod/container_api.go:290
github.com/containers/podman/libpod.(*Runtime).Migrate(0xc000102540, {0x0, 0x0})
        github.com/containers/podman/libpod/runtime_migrate_linux.go:76 +0x2e5
github.com/containers/podman/pkg/domain/infra/abi.(*ContainerEngine).Migrate(0x0?, {0x0?, 0xcb2006ca48b37857?}, {{0x0?, 0x0?}})
        github.com/containers/podman/pkg/domain/infra/abi/system.go:307 +0x25
github.com/containers/podman/cmd/podman/system.migrate(0x2e3a000?, {0x2f40760?, 0x0?, 0x0?})
        github.com/containers/podman/cmd/podman/system/migrate.go:56 +0x45
github.com/spf13/cobra.(*Command).execute(0x2e3a000, {0xc00003ea40, 0x0, 0x0})
        github.com/spf13/cobra/command.go:989 +0xabb
github.com/spf13/cobra.(*Command).ExecuteC(0x2e22d20)
        github.com/spf13/cobra/command.go:1117 +0x44f
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        github.com/spf13/cobra/command.go:1034
main.Execute()
        github.com/containers/podman/cmd/podman/root.go:119 +0x57
main.main()
        github.com/containers/podman/cmd/podman/main.go:62 +0x4b2

Describe the results you expected

I expected that podman system migrate succeeds.

podman info output

host:
  arch: amd64
  buildahVersion: 1.39.3
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.12-4_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: unknown'
  cpuUtilization:
    idlePercent: 65.73
    systemPercent: 9.55
    userPercent: 24.72
  cpus: 4
  databaseBackend: sqlite
  distribution:
    codename: trixie
    distribution: debian
    version: "13"
  eventLogger: journald
  freeLocks: 2040
  hostname: <redacted>
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 231072
      size: 65536
  kernel: 6.12.48+deb13-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 11455889408
  memTotal: 16773206016
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.14.0-3_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark_1.14.0-2_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.14.0
  ociRuntime:
    name: crun
    package: crun_1.21-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.21
      commit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88
      rundir: /run/user/1002/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20250503.587980c-2_amd64
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1002/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1.1_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.6.0
  swapFree: 0
  swapTotal: 0
  uptime: 2h 2m 48.00s (Approximately 0.08 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/ha_proxy/.config/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 5
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/ha_proxy/.local/share/containers/storage
  graphRootAllocated: 77061185536
  graphRootUsed: 12793802752
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 5
  runRoot: /run/user/1002/containers
  transientStore: false
  volumePath: /home/ha_proxy/.local/share/containers/storage/volumes
version:
  APIVersion: 5.4.2
  BuildOrigin: Debian
  Built: 1753478586
  BuiltTime: Fri Jul 25 21:23:06 2025
  GitCommit: ""
  GoVersion: go1.24.4
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[baude]

ideally you would be reporting this on main branch or 5.7. are you in a position to do that ?

[baude]

also, any chance you could capture a core ?

[mheon]

Is this reproducible? I took a quick look, and it seems like this is caused by runtime.store being nil, which shouldn't be possible.

[shibumi]

@mheon @baude

We are experimenting right now a lot with systemd service hardening and cgroup management. I think you are able to reproduce this issue with a systemd service file like this one:

[Unit]
Description=Nginx

[Service]
Type=notify
NotifyAccess=all
ExecStart=podman kube play --cgroup-manager=cgroupfs --replace  --service-container=true nginx.kube
User=<your own user>
Group=<your own user>
Delegate=true


[Install]
WantedBy=default.target

And:

---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  containers:
    - name: server
      image: docker.io/library/nginx:alpine
      ports:
        - name: http
          containerPort: 12345

I am very confident it has to do with changing the cgroup manager.

[Luap99]

@shibumi Like I mentioned in #27499 already we do not support such User/Group= setups and most of these systemd hardening options are fundamentally incompatible with podman since you block podman from doing what it has to do.

Now of course podman commands should not panic so there is certainly something wrong in the code that should be fixed here. But in general don't' expect this stuff to work or our help debugging issues caused by such units.

I am not sure if mixing the systemd cgroup unit management with the podman cgroups driver is a good idea at all. Having two process trying to configure the cgroups can certainly cause issues.

[shibumi]

@shibumi Like I mentioned in #27499 already we do not support such User/Group= setups and most of these systemd hardening options are fundamentally incompatible with podman since you block podman from doing what it has to do.

Hi,

I am aware of this and the intention of this issue here is mainly to report the panic and nil dereference.

I'm running into a bit of a documentation gap and wanted to share a user's perspective. I've found that I often have to search through GitHub issues to learn whether features like X or Y are supported.

To help make this clearer for everyone, what do you think about adding a section to the man page? Having a clear list of what is supported and what's explicitly not would be a huge help and would also lead to less toil in the form of Github issues.

As a separate idea, I've been exploring systemd service hardening. It would be fantastic if the podman team could share some best practices for this. As a stretch goal, perhaps there's a way we could allow users to inject hardening options directly into the systemd podman generator or Quadlets?

If you know about any hardening options that are incompatible with podman itself, can we get these documented somewhere? For example, what are file paths that podman requires to work?

[Luap99]

Can you verify if #27604 fixes the issue for you. I guess the issue is that your systemd unit kills the pause process which is supposed to outlive the unit normally but I guess it didn't got moved because of the cgroup driver argument.