Pre-existing host mounts are not restored despite mount propagation is active

[specht478]

Hi guys,

if I start a Podman container with a bind-mount of the type:

mount=type=bind,src=/tmp/origin,destination=/MNT,bind-propagation=rslave

and inside /tmp/origin/ there is something mounted: /tmp/origin/test_mountpoint

I can see all mounted files inside the container under /MNT/test_mountpoint/...

But if I checkpoint and restore that same container, /MNT/test_mountpoint/... will be empty if the mount /tmp/origin/test_mountpoint is present before restore.

Here are the commands to test this:

#Preparation:
dd if=/dev/zero of=/tmp/testfs.img bs=1M count=10 
sudo mkfs.ext4 /tmp/testfs.img


# Mount testfs before container creation
mkdir -p /tmp/origin/testfs
sudo mount -o loop /tmp/testfs.img /tmp/origin/testfs

sudo podman run \
    --name testc \
    --detach \
    --replace \
    --mount=type=bind,src=/tmp/origin,destination=/MNT,bind-propagation=rslave \
    --network=host \
    alpine \
    sleep infinity

# Mount in mount is visible from inside the container
sudo podman exec testc tree /MNT
#/MNT
#└── testfs
#    └── lost+found
#
# 2 directories, 0 files

# Unmount to allow checkpoint. If not: mnt: Mount 359 ./MNT/testfs (master_id: 232 shared_id: 0) has unreachable sharing
sudo umount /tmp/origin/testfs

# Checkpoint container
sudo podman container checkpoint testc

# Mount testfs again before restoring the container
sudo mount -o loop /tmp/testfs.img /tmp/origin/testfs

# Restore
sudo podman container restore testc

### Submount is not visible anymore
sudo podman exec testc tree /MNT
#/MNT
#└── testfs
#
#1 directories, 0 files

Is this expected behavior? Is there any way so that the mounted files are accessible from inside the container after restore?

Thanks in advance!

Regards
specht478

[adrianreber]

This sounds like a Podman problem. CRIU restores all mount points which runc tells CRIU about. runc only knows about mounts from config.json. Can you check if this mount shows up in config.json?

[specht478]

config.json

Shows the mount from /tmp/origin -> /MNT

   "mounts": [
      {
         "destination": "/MNT",
         "type": "bind",
         "source": "/tmp/origin",
         "options": [
            "rslave",
            "rw",
            "nosuid",
            "nodev",
            "rbind"
         ]
      },

Before starting the container, we do on the host:

mkdir -p /tmp/origin/testfs
sudo mount -o loop /tmp/testfs.img /tmp/origin/testfs

Podman knows nothing about this mount. But after starting the container i can see the contents of testf inside the container. This is expected, since it was mounted with rslave.

Before i checkpoint the container, i perform an umount of /tmp/origin/testfs, to avoid the error has unreachable sharing

After restore, /MN/testfs is empty for the container.

Podman did not know anything about /tmp/origin/testfs in any case. But after a fresh start of the container the content is visible to the container, and after restore, it is not.

[adrianreber]

Maybe somewhere in the stack the mount propagation is not correctly restored, not sure. I guess the main reason this does not work, nobody tested it so far. I would say this is either a runc or Podman problem. I recommend looking at those.

[specht478]

The thing is: after restore, if the mount /tmp/origin/testfs is performed, the container sees its content under /MN/testfs. So I guess the propagation is working. But only for mounts that appear afterwords...

[adrianreber]

runc is trying to mount certain things before calling CRIU, maybe something is not happening correctly in runc.

[specht478]

Hi, thanks for the input. But isn't CRIU the one creating the mounts here? Here are some log outputs.

Before checkpoint, there is a mount /tmp/origin -> /MNT. /tmp/origin is empty.

CRIU sees this mount:

(00.211101)     type tmpfs source tmpfs mnt_id 455 s_dev 0x22 /origin @ ./MNT flags 0x1080006 options size=1588504k,nr_inodes=1048576
[...]
(00.211944) mnt:                Working on 455->454
[...]
(00.212203) mnt:        Resorting children of 455 in mount order
[...]
(00.212704) mnt:  [./MNT](455->454)
[...]
(00.213232) mnt: Inspecting sharing on 455 shared_id 0 master_id 18 (@./MNT)
[...]
(00.213245) mnt: Detected external slavery for 455 via 455
[...]
(00.369980) mnt:        455: 22:/origin @ ./MNT

On restore, in the host, /tmp/origin is not empty, it contains another mount in it /tmp/origin/testfs

CRIU says:

(00.006279) mnt:        Read 457 mp @ /proc
(00.006296) mnt:                Will mount 455 from /tmp/origin (E)
(00.006312) mnt:                Will mount 455 @ /tmp/.criu.mntns.5trmeN/mnt-0000000455 /MNT
[...]
(00.006759) mnt:        Resorting children of 455 in mount order
[...]
(00.007107) mnt:  [/MNT](455->454)
[...]
(00.007675) mnt-v2: Inspecting sharing on 455 shared_id 0 master_id 18 (@/MNT)
[...]
(00.153662)      1: mnt-v2: can_mount_now_v2: true as 455 is external
(00.153714)      1: mnt-v2: Mount 455 is detected as dir-mount
(00.153728)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.5trmeN/mnt-0000000455 for 455
(00.153753)      1: mnt-v2:     Mounting tmpfs @455 (0)
(00.153768)      1: mnt-v2:     Bind /tmp/origin to /tmp/.criu.mntns.5trmeN/mnt-0000000455
(00.153852)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.5trmeN/mnt-0000000455 for 455
[...]
(00.179575)      1: mnt-v2: Move mount 455 from /tmp/.criu.mntns.5trmeN/mnt-0000000455 to /tmp/.criu.mntns.5trmeN/12-0000000000/MNT

Maybe one could instruct CRIU to perform the binds for /MNT in a way that the sub-mounts are also visible?

[adrianreber]

Hi, thanks for the input. But isn't CRIU the one creating the mounts here?

Not always. If you look at runc https://github.com/opencontainers/runc/blob/64c3c8eea6019e15d77ce2dc3aa0eda9307799ae/libcontainer/criu_linux.go#L555 you can see that we go through a couple of steps to replicate the set up of mount points just like runc during container start. So I still think it might be related to this, but it could also be that CRIU is not doing something completely correct. I just keep on mentioning runc, because we tried to set up mount points correctly there. There used to be a difference between starting a container and restoring a container when it comes to mount points. Maybe we just missed something in runc. Maybe... Maybe there is also something missing in CRIU.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.