mountall hangs if datasets have duplicate mountpoint properties

[nedbass]

I accidentally got my workstation into a state where the boot would hang after the "Running init-bottom...done". The cause turned out to be two datasets that had identical mountpoint properties. This seemed to prevent mountall from mounting either dataset and it would never emit the virtual-filesystems event, blocking udev and other services. If I start a shell on tty2 and mount one of the filesystems manually, mountall completes and the boot continues normally.

I need to verify this, but I think the first dataset with the duplicate mountpoint to show up in zfs list must be non-empty to reproduce this issue. However, as another test, I created a dataset with a non-empty underlying mountpoint, but this did not cause a boot problem. So the duplicate mountpoint properties seem to cause an issue beyond just the mountpoint being non-empty.

For example, this reproduces the issue:

zfs create tank/fish
touch /tank/fish/foo
zfs create -o mountpoint=/tank/fish frog

This does not reproduce it:

zfs create tank/fish
umount /tank/fish
touch /tank/fish/foo

This reproducer is admittedly a configuration error (I introduced it by receiving a replication stream into a test pool) but I thought we should evaluate the impact and how it could be handled better. I could collect more debug data, but I wanted to start by just reporting what happened since @dajhorn may have some insight about it.

$ apt-cache policy mountall ubuntu-zfs
mountall:
  Installed: 2.36.4-zfs1
  Candidate: 2.36.4-zfs1
  Version table:
 *** 2.36.4-zfs1 0
       1001 http://ppa.launchpad.net/zfs-native/daily/ubuntu/ precise/main amd64 Packages
        100 /var/lib/dpkg/status
     2.36 0
        500 http://archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
ubuntu-zfs:
  Installed: 7~precise
  Candidate: 7~precise
  Version table:
 *** 7~precise 0
       1001 http://ppa.launchpad.net/zfs-native/daily/ubuntu/ precise/main amd64 Packages
        100 /var/lib/dpkg/status

$strings /sbin/mountall | grep ZFS
%s: parsing ZFS list

$ grep ZFS_MOUNT /etc/default/zfs
ZFS_MOUNT='no'

[nedbass]

Here is a simplified test case. It does not matter if the datasets are empty or not.

root@precise-vm:~# truncate -s 128m /tmp/tank
root@precise-vm:~# zpool create tank /tmp/tank
root@precise-vm:~# zfs create tank/fish
root@precise-vm:~# zfs umount -a
root@precise-vm:~# zfs create -o mountpoint=/tank/fish tank/frog
root@precise-vm:~# zfs umount -a
root@precise-vm:~# mountall --verbose
/ is local
/proc is virtual
/sys is virtual
/sys/fs/fuse/connections is virtual
/sys/kernel/debug is virtual
/sys/kernel/security is virtual
/dev is virtual
/dev/pts is virtual
/tank/fish is virtual
/tank/fish is virtual
/var/lib/lightdm/.gvfs is local
mounting event sent for /tmp
mounting event sent for /tank
mounting event sent for swap /dev/disk/by-uuid/34efd22b-4736-46e0-a2f1-60e693317683
mounted event handled for /
mount / [3993] exited normally
mount /proc [4008] exited normally
mount /sys [4014] exited normally
mount /sys/fs/fuse/connections [4018] exited normally
mount /sys/kernel/debug [4019] exited normally
mount /sys/kernel/security [4020] exited normally
mount /dev [4022] exited normally
mount /dev/pts [4025] exited normally
mount /run [4026] exited normally
mount /run/lock [4028] exited normally
mount /run/shm [4031] exited normally
/bin/sh: 1: gvfs-fuse-daemon: not found
mountall: mount /var/lib/lightdm/.gvfs [4034] terminated with status 127
local 1/3 remote 0/0 virtual 0/13 swap 0/1
mounted event handled for /proc
local 1/3 remote 0/0 virtual 1/13 swap 0/1
mounted event handled for /sys
local 1/3 remote 0/0 virtual 2/13 swap 0/1
mounted event handled for /sys/fs/fuse/connections
local 1/3 remote 0/0 virtual 3/13 swap 0/1
mounted event handled for /sys/kernel/debug
local 1/3 remote 0/0 virtual 4/13 swap 0/1
mounted event handled for /sys/kernel/security
local 1/3 remote 0/0 virtual 5/13 swap 0/1
mounted event handled for /dev/pts
local 1/3 remote 0/0 virtual 6/13 swap 0/1
mounted event handled for /run/lock
local 1/3 remote 0/0 virtual 7/13 swap 0/1
mounted event handled for /run/shm
local 1/3 remote 0/0 virtual 8/13 swap 0/1
mounted event handled for /var/lib/lightdm/.gvfs
local 2/3 remote 0/0 virtual 8/13 swap 0/1
mounted event handled for /dev
local 2/3 remote 0/0 virtual 9/13 swap 0/1
mounting event handled for /tmp
mounting event handled for /tank
mounting /tank
mount /tank [4048] exited normally
mounting event handled for swap /dev/disk/by-uuid/34efd22b-4736-46e0-a2f1-60e693317683
activating /dev/disk/by-uuid/34efd22b-4736-46e0-a2f1-60e693317683
swapon: /dev/disk/by-uuid/34efd22b-4736-46e0-a2f1-60e693317683: swapon failed: Device or resource busy
mountall: swapon /dev/disk/by-uuid/34efd22b-4736-46e0-a2f1-60e693317683 [4109] terminated with status 255
mountall: Problem activating swap: /dev/disk/by-uuid/34efd22b-4736-46e0-a2f1-60e693317683
mounted event handled for /run
local 2/3 remote 0/0 virtual 10/13 swap 0/1
mounted event handled for /tmp
local 3/3 remote 0/0 virtual 10/13 swap 0/1
mounted event handled for /tank
local 3/3 remote 0/0 virtual 11/13 swap 0/1
mounted event handled for swap /dev/disk/by-uuid/34efd22b-4736-46e0-a2f1-60e693317683
swap finished
local 3/3 remote 0/0 virtual 11/13 swap 1/1
mounting event sent for swap /dev/disk/by-uuid/34efd22b-4736-46e0-a2f1-60e693317683
mounting event handled for swap /dev/disk/by-uuid/34efd22b-4736-46e0-a2f1-60e693317683
[hangs]

[dajhorn]

@nedbass

I can reproduce this, but I noticed that the second zfs umount -a call fails silently. If I manually dismount all ZFS datasets and run mountall again, then I get a slightly different failure:

try_mount: UUID=a5566bef-a07d-4720-a045-ec2d935cfbcc waiting for device
try_mount: /tank/fish waiting for /tank/fish
try_mount: /tank/fish waiting for /tank/fish
mounted event handled for /tmp
local 4/4 remote 0/0 virtual 14/22 swap 0/1
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
try_mount: UUID=a5566bef-a07d-4720-a045-ec2d935cfbcc waiting for device
try_mount: /tank/fish waiting for /tank/fish
try_mount: /tank/fish waiting for /tank/fish

[dajhorn]

The bug is caused by try_mounts() getting an incorrect dep->mounted value, which seems to be caused by mount_policy using the /tank/fish directory as the parent for both datasets instead of just /tank.

[nedbass]

This patch avoids the bug for me by eliminating duplicates from the mounts list. parse_fstab() does something similar.

diff --git a/src/mountall.c b/src/mountall.c
index a23f407..e97d97a 100644
--- a/src/mountall.c
+++ b/src/mountall.c
@@ -783,6 +783,7 @@ parse_zfs_list (void)
                char *saveptr;
                char *zfs_name, *zfs_mountpoint, *zfs_canmount, *zfs_optatime, *zfs_optronly;
                nih_local char *zfs_mntoptions = NULL;
+               Mount *         mnt;

                /* If the line didn't fit, then enlarge the buffer and retry. */
                while ((! strchr (buf, '\n')) && (! feof (zfs_stream))) {
@@ -824,7 +825,14 @@ parse_zfs_list (void)
                        NIH_MUST (nih_strcat (&zfs_mntoptions, NULL, ",noatime"));
                }

-               new_mount (zfs_mountpoint, zfs_name, FALSE, "zfs", zfs_mntoptions);
+               mnt = find_mount (zfs_mountpoint);
+               if (mnt) {
+                       update_mount (mnt, zfs_name, FALSE, "zfs",
+                                     zfs_mntoptions);
+               } else {
+                       new_mount (zfs_mountpoint, zfs_name, FALSE, "zfs",
+                                  zfs_mntoptions);
+               }
        }

        zfs_result = pclose (zfs_stream);

[dajhorn]

@nedbass, perhaps you found an upstream bug here. With ZFS disabled, I get an incomplete result from mountall if this is in the /etc/fstab file:

/var/tmp/alfa   /mnt/charlie    ext2    loop    0   0
/var/tmp/bravo  /mnt/charlie    ext2    loop    0   0

But a regular mount -a behaves as expected.

[nedbass]

Strange, I performed an equivalent test and didn't have a problem. Maybe we tested different versions. Based on my understanding of the bug, this code in parse_fstab should prevent the bug for duplicate mount points in the fstab file. It ensures only one of the entries exists in the mounts list when mount_policy is called. Do you know if the version you tested contains that logic?

[dajhorn]

Strange, I performed an equivalent test and didn't have a problem.

Lexical order and fstab order seems to affect the result.

Do you know if the version you tested contains that logic?

I tried the current releases for Precise and Quantal.

I get mountall behavior that is the same as mount -a and zfs mount -a with this patch:

--- a/src/mountall.c
+++ b/src/mountall.c
@@ -146,6 +146,7 @@
 };

 #define MOUNT_NAME(_mnt) (strcmp ((_mnt)->type, "swap")            \
+             && strcmp ((_mnt)->type, "zfs")       \
              && strcmp ((_mnt)->mountpoint, "none")    \
              ? (_mnt)->mountpoint : (_mnt)->device)

@@ -1131,6 +1132,17 @@
    nih_assert (root != NULL);
    nih_assert (path != NULL);

+   /*
+    * A path cannot be a root of itself.
+    *
+    * Without this test, `mountall` does not preserve `mount -a` behavior for
+    * fstab entries that share a mount point, and hangs on ZFS datasets and
+    * other virtual filesystems that are not backed by a real block device.
+    */
+
+   if (! strcmp (root, path))
+       return FALSE;
+
    len = strlen (root);
    if ((! strncmp (path, root, len))
        && ((path[len] == '\0')

[nedbass]

It is unclear to me what the original intention of that (path[len] == '\0') test was, but it seems to be there to specifically allow a path to be its own parent. With your proposed change, it may as well be removed.

[dajhorn]

@nedbass, I used the concise alternative that you suggested. After checking the code history and pondering the issue, I can't think of a circumstance where the existing mountall behavior is beneficial or necessary. Regardless, I do think that bug compatibility with util-linux and other platforms is worthwhile.

This is also a complete solution for the hang that @mk01 found earlier, so I dropped the solves-dependencies-problem-endless-loop.patch from the queue.