zfs and aufs not playing nice

[mrdrmmn]

mountall needs to make sure that all aufs branches have been mounted before mounting the aufs mountpoint. This bug was apparently fixed in the standard mountall in ubuntu 9.10. (https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/433537) However, it seems to be affecting the zfs mountall.

[dajhorn]

Could you post the output of mountall --debug for this failure?

[mrdrmmn]

Sure thing. Just so you don't wonder too much about the data, I unmounted /z, /mnt/vol3, and /mnt/vol5 prior to capturing the output so that the aufs mount (/z) and several of the mount it should be dependent on actually went through the mount process. You'll notice that this the aufs (/z) filesystem mounted prior to zfs filesystem (/mnt/vol3) while apparently after the btrfs filesystem (/mnt/vol5).

Let me know if there is anything else I can provide that may help. And without further ado, the output of mountall --debug:

plymouth_connect: Failed to connect to Plymouth: Connection refused
parse_filesystems: reading filesystems
parse_filesystems: sysfs (nodev)
parse_filesystems: rootfs (nodev)
parse_filesystems: bdev (nodev)
parse_filesystems: proc (nodev)
parse_filesystems: cgroup (nodev)
parse_filesystems: cpuset (nodev)
parse_filesystems: tmpfs (nodev)
parse_filesystems: devtmpfs (nodev)
parse_filesystems: debugfs (nodev)
parse_filesystems: securityfs (nodev)
parse_filesystems: sockfs (nodev)
parse_filesystems: pipefs (nodev)
parse_filesystems: anon_inodefs (nodev)
parse_filesystems: devpts (nodev)
parse_filesystems: ext3
parse_filesystems: ext4
parse_filesystems: ramfs (nodev)
parse_filesystems: hugetlbfs (nodev)
parse_filesystems: ecryptfs (nodev)
parse_filesystems: fuseblk
parse_filesystems: fuse (nodev)
parse_filesystems: fusectl (nodev)
parse_filesystems: pstore (nodev)
parse_filesystems: mqueue (nodev)
parse_filesystems: btrfs
parse_filesystems: zfs (nodev)
parse_filesystems: binfmt_misc (nodev)
parse_filesystems: aufs (nodev)
parse_fstab: updating from /lib/init/fstab
update_mount: /: / /dev/root rootfs defaults check
update_mount: /proc: /proc none proc nodev,noexec,nosuid
update_mount: /proc/sys/fs/binfmt_misc: /proc/sys/fs/binfmt_misc none binfmt_misc nodev,noexec,nosuid,optional
update_mount: /sys: /sys none sysfs nodev,noexec,nosuid
update_mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections none fusectl optional
update_mount: /sys/kernel/debug: /sys/kernel/debug none debugfs optional
update_mount: /sys/kernel/security: /sys/kernel/security none securityfs optional
update_mount: /sys/firmware/efi/efivars: /sys/firmware/efi/efivars none efivarfs optional
update_mount: /spu: /spu none spufs gid=spu,optional
update_mount: /dev: /dev none devtmpfs,tmpfs mode=0755
update_mount: /dev/pts: /dev/pts none devpts noexec,nosuid,gid=tty,mode=0620
update_mount: /tmp: /tmp none none defaults
update_mount: /run: /run none tmpfs noexec,nosuid,size=10%,mode=0755
update_mount: /run/lock: /run/lock none tmpfs nodev,noexec,nosuid,size=5242880
update_mount: /run/shm: /run/shm none tmpfs nosuid,nodev
parse_fstab: updating from /etc/fstab
update_mount: /boot/grub: /boot/grub LABEL=/boot/grub auto defaults check
update_mount: /mnt/vol5: /mnt/vol5 LABEL=vol5 btrfs device=/dev/disk/by-id/scsi-SATA_WDC_WD20EARS-00_WD-WCAZA1018148,device=/dev/disk/by-id/scsi-SATA_WDC_WD20EARS-00_WD-WCAYY0229139,compress=zlib
update_mount: /z: /z none aufs create=pmfs,sum,xino=/run/.aufs.xino,br:/mnt/vol5=rw:/mnt/vol3=rw:/mnt/vol2=rw:/mnt/vol1=rw:/mnt/vol0=ro
parse_zfs_list: parsing ZFS list
update_mount: /: / rpool/ROOT/ubuntu-1 zfs zfsutil
update_mount: /home: /home rpool/home zfs zfsutil
update_mount: /mnt/vol0: /mnt/vol0 vol0 zfs zfsutil
update_mount: /mnt/vol1: /mnt/vol1 vol1 zfs zfsutil
update_mount: /mnt/vol2: /mnt/vol2 vol2 zfs zfsutil
update_mount: /mnt/vol3: /mnt/vol3 vol3 zfs zfsutil
update_mount: /rpool: /rpool rpool zfs zfsutil
update_mount: /rpool/ROOT: /rpool/ROOT rpool/ROOT zfs zfsutil
parse_mountinfo_file: updating mounts
update_mount: /sys: /sys sysfs sysfs nodev,noexec,nosuid
update_mount: /proc: /proc proc proc nodev,noexec,nosuid
update_mount: /dev: /dev udev devtmpfs mode=0755
update_mount: /dev/pts: /dev/pts devpts devpts noexec,nosuid,gid=tty,mode=0620
update_mount: /run: /run tmpfs tmpfs noexec,nosuid,size=10%,mode=0755
update_mount: /: / rpool/ROOT/ubuntu-1 zfs defaults check
update_mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections none fusectl optional
update_mount: /sys/fs/cgroup: /sys/fs/cgroup cgroup tmpfs rw,relatime,rw,mode=755
update_mount: /sys/kernel/debug: /sys/kernel/debug none debugfs optional
update_mount: /sys/kernel/security: /sys/kernel/security none securityfs optional
update_mount: /run/lock: /run/lock none tmpfs nodev,noexec,nosuid,size=5242880
update_mount: /run/shm: /run/shm none tmpfs nosuid,nodev
update_mount: /sys/fs/cgroup/cpuset: /sys/fs/cgroup/cpuset cgroup cgroup rw,relatime,rw,cpuset
update_mount: /sys/fs/cgroup/cpu: /sys/fs/cgroup/cpu cgroup cgroup rw,relatime,rw,cpu
update_mount: /sys/fs/cgroup/cpuacct: /sys/fs/cgroup/cpuacct cgroup cgroup rw,relatime,rw,cpuacct
update_mount: /sys/fs/cgroup/memory: /sys/fs/cgroup/memory cgroup cgroup rw,relatime,rw,memory
update_mount: /boot/grub: /boot/grub /dev/sda1 ext3 defaults check
update_mount: /sys/fs/cgroup/devices: /sys/fs/cgroup/devices cgroup cgroup rw,relatime,rw,devices
update_mount: /sys/fs/cgroup/freezer: /sys/fs/cgroup/freezer cgroup cgroup rw,relatime,rw,freezer
update_mount: /sys/fs/cgroup/blkio: /sys/fs/cgroup/blkio cgroup cgroup rw,relatime,rw,blkio
update_mount: /sys/fs/cgroup/perf_event: /sys/fs/cgroup/perf_event cgroup cgroup rw,relatime,rw,perf_event
update_mount: /home: /home rpool/home zfs zfsutil
update_mount: /mnt/vol0: /mnt/vol0 vol0 zfs zfsutil
update_mount: /rpool: /rpool rpool zfs zfsutil
update_mount: /rpool/ROOT: /rpool/ROOT rpool/ROOT zfs zfsutil
update_mount: /proc/sys/fs/binfmt_misc: /proc/sys/fs/binfmt_misc binfmt_misc binfmt_misc nodev,noexec,nosuid,optional
update_mount: /mnt/vol1: /mnt/vol1 vol1 zfs zfsutil
update_mount: /mnt/vol2: /mnt/vol2 vol2 zfs zfsutil
mount_policy: /sys/firmware/efi/efivars: dropping unknown filesystem
mount_policy: /spu: dropping unknown filesystem
mount_policy: /: dropping filesystem because it has an already mounted child (/)

tag_mount: / is root filesystem
mount_policy: /proc can be mounted while root readonly
mount_policy: /proc/sys/fs/binfmt_misc parent is /proc
mount_policy: /sys can be mounted while root readonly
mount_policy: /sys/fs/fuse/connections parent is /sys
mount_policy: /sys/kernel/debug parent is /sys
mount_policy: /sys/kernel/security parent is /sys
mount_policy: /dev can be mounted while root readonly
mount_policy: /dev/pts parent is /dev
mount_policy: /tmp parent is /
tag_mount: /tmp is not virtual, inherited local
mount_policy: /run can be mounted while root readonly
mount_policy: /run/lock parent is /run
mount_policy: /run/shm parent is /run
mount_policy: /boot/grub parent is /
mount_policy: /mnt/vol5 parent is /
mount_policy: /z can be mounted while root readonly
mount_policy: /home can be mounted while root readonly
mount_policy: /mnt/vol0 can be mounted while root readonly
mount_policy: /mnt/vol0 prior fstab entry /home
mount_policy: /mnt/vol1 can be mounted while root readonly
mount_policy: /mnt/vol1 prior fstab entry /mnt/vol0
mount_policy: /mnt/vol2 can be mounted while root readonly
mount_policy: /mnt/vol2 prior fstab entry /mnt/vol1
mount_policy: /mnt/vol3 can be mounted while root readonly
mount_policy: /mnt/vol3 prior fstab entry /mnt/vol2
mount_policy: /rpool can be mounted while root readonly
mount_policy: /rpool prior fstab entry /mnt/vol3
mount_policy: /rpool/ROOT parent is /rpool
mount_policy: /rpool/ROOT prior fstab entry /rpool
mount_policy: /sys/fs/cgroup parent is /sys
mount_policy: /sys/fs/cgroup prior fstab entry /rpool/ROOT
mount_policy: /sys/fs/cgroup/cpuset parent is /sys/fs/cgroup
mount_policy: /sys/fs/cgroup/cpu parent is /sys/fs/cgroup
mount_policy: /sys/fs/cgroup/cpuacct parent is /sys/fs/cgroup
mount_policy: /sys/fs/cgroup/memory parent is /sys/fs/cgroup
mount_policy: /sys/fs/cgroup/devices parent is /sys/fs/cgroup
mount_policy: /sys/fs/cgroup/freezer parent is /sys/fs/cgroup
mount_policy: /sys/fs/cgroup/blkio parent is /sys/fs/cgroup
mount_policy: /sys/fs/cgroup/perf_event parent is /sys/fs/cgroup
/ is local
/proc is virtual
/proc/sys/fs/binfmt_misc is virtual
/sys is virtual
/sys/fs/fuse/connections is virtual
/sys/kernel/debug is virtual
/sys/kernel/security is virtual
/dev is virtual
/dev/pts is virtual
/tmp is local
/run is virtual
/run/lock is virtual
/run/shm is virtual
/boot/grub is local
/mnt/vol5 is local
/z is virtual
/home is virtual
/mnt/vol0 is virtual
/mnt/vol1 is virtual
/mnt/vol2 is virtual
/mnt/vol3 is virtual
/rpool is virtual
/rpool/ROOT is virtual
/sys/fs/cgroup is virtual
/sys/fs/cgroup/cpuset is virtual
/sys/fs/cgroup/cpu is virtual
/sys/fs/cgroup/cpuacct is virtual
/sys/fs/cgroup/memory is virtual
/sys/fs/cgroup/devices is virtual
/sys/fs/cgroup/freezer is virtual
/sys/fs/cgroup/blkio is virtual
/sys/fs/cgroup/perf_event is virtual
mounted: /
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /proc
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /proc/sys/fs/binfmt_misc
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys/fs/fuse/connections
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys/kernel/debug
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys/kernel/security
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /dev
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /dev/pts
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /run
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /run/lock
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /run/shm
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /boot/grub
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /home
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /mnt/vol0
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /mnt/vol1
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /mnt/vol2
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /rpool
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /rpool/ROOT
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys/fs/cgroup
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys/fs/cgroup/cpuset
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys/fs/cgroup/cpu
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys/fs/cgroup/cpuacct
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys/fs/cgroup/memory
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys/fs/cgroup/devices
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys/fs/cgroup/freezer
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys/fs/cgroup/blkio
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted: /sys/fs/cgroup/perf_event
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
control_server_open: Mountall0_1.Server started at address: unix:abstract=/com/ubuntu/mountall/server/
run_fsck: /tmp: no check required
mounting event sent for /tmp
try_mount: /mnt/vol5 waiting for device
run_fsck: /z: no check required
mounting event sent for /z
run_fsck: /mnt/vol3: no check required
mounting event sent for /mnt/vol3
udev_catchup: catching up
try_udev_device: block /dev/sda (null) (null)
try_udev_device: block /dev/sda1 9b9010d1-7d2d-4a67-a81d-0b37d3a1c6b1 /boot/grub
try_udev_device: /boot/grub by name
run_fsck: /boot/grub: mounted filesystem
try_udev_device: block /dev/sda2 6493392249542836328 rpool
try_udev_device: block /dev/sdb (null) (null)
try_udev_device: block /dev/sdb1 6add1c63-57f4-4e52-9e99-0f16ca278d2d /boot/grub
try_udev_device: block /dev/sdb2 6493392249542836328 rpool
try_udev_device: block /dev/sdc1 12512439444215740404 vol1
try_udev_device: block /dev/sdd1 12512439444215740404 vol1
try_udev_device: block /dev/sde1 10812058813165327808 vol0
try_udev_device: block /dev/sdf1 1076757845528739222 vol2
try_udev_device: block /dev/sdg1 3132473998697227028 vol3
try_udev_device: block /dev/sdh1 11680632253170806410 vol4
try_udev_device: block /dev/sdi1 10812058813165327808 vol0
try_udev_device: block /dev/sdj1 11680632253170806410 vol4
try_udev_device: block /dev/sdk1 3132473998697227028 vol3
try_udev_device: block /dev/sdl 0ca33395-2414-4559-b922-2ae5129d2b19 vol5
try_udev_device: /mnt/vol5 by label
update_mount: /mnt/vol5: /mnt/vol5 /dev/disk/by-label/vol5 btrfs device=/dev/disk/by-id/scsi-SATA_WDC_WD20EARS-00_WD-WCAZA1018148,device=/dev/disk/by-id/scsi-SATA_WDC_WD20EARS-00_WD-WCAYY0229139,compress=zlib
run_fsck: /mnt/vol5: no check required
mounting event sent for /mnt/vol5
try_udev_device: block /dev/sdm 0ca33395-2414-4559-b922-2ae5129d2b19 vol5
try_udev_device: /mnt/vol5 by link /dev/disk/by-label/vol5
run_fsck: /mnt/vol5: already ready
try_udev_device: block /dev/sdn1 1076757845528739222 vol2
mounted event handled for /
swap finished
local 1/4 remote 0/0 virtual 0/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /proc/sys/fs/binfmt_misc
local 1/4 remote 0/0 virtual 1/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /sys
local 1/4 remote 0/0 virtual 2/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /sys/fs/fuse/connections
local 1/4 remote 0/0 virtual 3/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /sys/kernel/security
local 1/4 remote 0/0 virtual 4/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /proc
local 1/4 remote 0/0 virtual 5/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /sys/kernel/debug
local 1/4 remote 0/0 virtual 6/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /dev/pts
local 1/4 remote 0/0 virtual 7/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /run/lock
local 1/4 remote 0/0 virtual 8/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /run/shm
local 1/4 remote 0/0 virtual 9/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /home
local 1/4 remote 0/0 virtual 10/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /mnt/vol0
local 1/4 remote 0/0 virtual 11/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /mnt/vol1
local 1/4 remote 0/0 virtual 12/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /mnt/vol2
local 1/4 remote 0/0 virtual 13/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /rpool
local 1/4 remote 0/0 virtual 14/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /rpool/ROOT
local 1/4 remote 0/0 virtual 15/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /sys/fs/cgroup
local 1/4 remote 0/0 virtual 16/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /sys/fs/cgroup/cpuset
local 1/4 remote 0/0 virtual 17/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /sys/fs/cgroup/cpu
local 1/4 remote 0/0 virtual 18/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /sys/fs/cgroup/cpuacct
local 1/4 remote 0/0 virtual 19/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /sys/fs/cgroup/memory
local 1/4 remote 0/0 virtual 20/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /sys/fs/cgroup/devices
local 1/4 remote 0/0 virtual 21/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /sys/fs/cgroup/freezer
local 1/4 remote 0/0 virtual 22/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /sys/fs/cgroup/blkio
local 1/4 remote 0/0 virtual 23/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /sys/fs/cgroup/perf_event
local 1/4 remote 0/0 virtual 24/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /dev
local 1/4 remote 0/0 virtual 25/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounting event handled for /tmp
run_mount: /tmp: placeholder
mounted: /tmp
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounting event handled for /z
mounting /z
spawn: mount -n -t aufs -o create=pmfs,sum,xino=/run/.aufs.xino,br:/mnt/vol5=rw:/mnt/vol3=rw:/mnt/vol2=rw:/mnt/vol1=rw:/mnt/vol0=ro none /z
spawn: mount /z [28537]
mount /z [28537] exited normally
parse_mountinfo_file: updating mounts
update_mount: /sys: /sys sysfs sysfs nodev,noexec,nosuid
update_mount: /proc: /proc proc proc nodev,noexec,nosuid
update_mount: /dev: /dev udev devtmpfs mode=0755
update_mount: /dev/pts: /dev/pts devpts devpts noexec,nosuid,gid=tty,mode=0620
update_mount: /run: /run tmpfs tmpfs noexec,nosuid,size=10%,mode=0755
update_mount: /: / rpool/ROOT/ubuntu-1 zfs defaults check
update_mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections none fusectl optional
update_mount: /sys/fs/cgroup: /sys/fs/cgroup cgroup tmpfs rw,relatime,rw,mode=755
update_mount: /sys/kernel/debug: /sys/kernel/debug none debugfs optional
update_mount: /sys/kernel/security: /sys/kernel/security none securityfs optional
update_mount: /run/lock: /run/lock none tmpfs nodev,noexec,nosuid,size=5242880
update_mount: /run/shm: /run/shm none tmpfs nosuid,nodev
update_mount: /sys/fs/cgroup/cpuset: /sys/fs/cgroup/cpuset cgroup cgroup rw,relatime,rw,cpuset
update_mount: /sys/fs/cgroup/cpu: /sys/fs/cgroup/cpu cgroup cgroup rw,relatime,rw,cpu
update_mount: /sys/fs/cgroup/cpuacct: /sys/fs/cgroup/cpuacct cgroup cgroup rw,relatime,rw,cpuacct
update_mount: /sys/fs/cgroup/memory: /sys/fs/cgroup/memory cgroup cgroup rw,relatime,rw,memory
update_mount: /boot/grub: /boot/grub /dev/sda1 ext3 defaults check
update_mount: /sys/fs/cgroup/devices: /sys/fs/cgroup/devices cgroup cgroup rw,relatime,rw,devices
update_mount: /sys/fs/cgroup/freezer: /sys/fs/cgroup/freezer cgroup cgroup rw,relatime,rw,freezer
update_mount: /sys/fs/cgroup/blkio: /sys/fs/cgroup/blkio cgroup cgroup rw,relatime,rw,blkio
update_mount: /sys/fs/cgroup/perf_event: /sys/fs/cgroup/perf_event cgroup cgroup rw,relatime,rw,perf_event
update_mount: /home: /home rpool/home zfs zfsutil
update_mount: /mnt/vol0: /mnt/vol0 vol0 zfs zfsutil
update_mount: /rpool: /rpool rpool zfs zfsutil
update_mount: /rpool/ROOT: /rpool/ROOT rpool/ROOT zfs zfsutil
update_mount: /proc/sys/fs/binfmt_misc: /proc/sys/fs/binfmt_misc binfmt_misc binfmt_misc nodev,noexec,nosuid,optional
update_mount: /mnt/vol1: /mnt/vol1 vol1 zfs zfsutil
update_mount: /mnt/vol2: /mnt/vol2 vol2 zfs zfsutil
update_mount: /z: /z none aufs create=pmfs,sum,xino=/run/.aufs.xino,br:/mnt/vol5=rw:/mnt/vol3=rw:/mnt/vol2=rw:/mnt/vol1=rw:/mnt/vol0=ro
mounted: /z
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounting event handled for /mnt/vol3
mounting /mnt/vol3
spawn: mount -n -t zfs -o zfsutil vol3 /mnt/vol3
spawn: mount /mnt/vol3 [28550]
mount /mnt/vol3 [28550] exited normally
parse_mountinfo_file: updating mounts
update_mount: /sys: /sys sysfs sysfs nodev,noexec,nosuid
update_mount: /proc: /proc proc proc nodev,noexec,nosuid
update_mount: /dev: /dev udev devtmpfs mode=0755
update_mount: /dev/pts: /dev/pts devpts devpts noexec,nosuid,gid=tty,mode=0620
update_mount: /run: /run tmpfs tmpfs noexec,nosuid,size=10%,mode=0755
update_mount: /: / rpool/ROOT/ubuntu-1 zfs defaults check
update_mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections none fusectl optional
update_mount: /sys/fs/cgroup: /sys/fs/cgroup cgroup tmpfs rw,relatime,rw,mode=755
update_mount: /sys/kernel/debug: /sys/kernel/debug none debugfs optional
update_mount: /sys/kernel/security: /sys/kernel/security none securityfs optional
update_mount: /run/lock: /run/lock none tmpfs nodev,noexec,nosuid,size=5242880
update_mount: /run/shm: /run/shm none tmpfs nosuid,nodev
update_mount: /sys/fs/cgroup/cpuset: /sys/fs/cgroup/cpuset cgroup cgroup rw,relatime,rw,cpuset
update_mount: /sys/fs/cgroup/cpu: /sys/fs/cgroup/cpu cgroup cgroup rw,relatime,rw,cpu
update_mount: /sys/fs/cgroup/cpuacct: /sys/fs/cgroup/cpuacct cgroup cgroup rw,relatime,rw,cpuacct
update_mount: /sys/fs/cgroup/memory: /sys/fs/cgroup/memory cgroup cgroup rw,relatime,rw,memory
update_mount: /boot/grub: /boot/grub /dev/sda1 ext3 defaults check
update_mount: /sys/fs/cgroup/devices: /sys/fs/cgroup/devices cgroup cgroup rw,relatime,rw,devices
update_mount: /sys/fs/cgroup/freezer: /sys/fs/cgroup/freezer cgroup cgroup rw,relatime,rw,freezer
update_mount: /sys/fs/cgroup/blkio: /sys/fs/cgroup/blkio cgroup cgroup rw,relatime,rw,blkio
update_mount: /sys/fs/cgroup/perf_event: /sys/fs/cgroup/perf_event cgroup cgroup rw,relatime,rw,perf_event
update_mount: /home: /home rpool/home zfs zfsutil
update_mount: /mnt/vol0: /mnt/vol0 vol0 zfs zfsutil
update_mount: /rpool: /rpool rpool zfs zfsutil
update_mount: /rpool/ROOT: /rpool/ROOT rpool/ROOT zfs zfsutil
update_mount: /proc/sys/fs/binfmt_misc: /proc/sys/fs/binfmt_misc binfmt_misc binfmt_misc nodev,noexec,nosuid,optional
update_mount: /mnt/vol1: /mnt/vol1 vol1 zfs zfsutil
update_mount: /mnt/vol2: /mnt/vol2 vol2 zfs zfsutil
update_mount: /z: /z none aufs create=pmfs,sum,xino=/run/.aufs.xino,br:/mnt/vol5=rw:/mnt/vol3=rw:/mnt/vol2=rw:/mnt/vol1=rw:/mnt/vol0=ro
update_mount: /mnt/vol3: /mnt/vol3 vol3 zfs zfsutil
mounted: /mnt/vol3
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /boot/grub
local 2/4 remote 0/0 virtual 25/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /run
local 2/4 remote 0/0 virtual 26/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounting event handled for /mnt/vol5
mounting /mnt/vol5
spawn: mount -n -t btrfs -o device=/dev/disk/by-id/scsi-SATA_WDC_WD20EARS-00_WD-WCAZA1018148,device=/dev/disk/by-id/scsi-SATA_WDC_WD20EARS-00_WD-WCAYY0229139,compress=zlib /dev/disk/by-label/vol5 /mnt/vol5
spawn: mount /mnt/vol5 [28556]
mount /mnt/vol5 [28556] exited normally
parse_mountinfo_file: updating mounts
update_mount: /sys: /sys sysfs sysfs nodev,noexec,nosuid
update_mount: /proc: /proc proc proc nodev,noexec,nosuid
update_mount: /dev: /dev udev devtmpfs mode=0755
update_mount: /dev/pts: /dev/pts devpts devpts noexec,nosuid,gid=tty,mode=0620
update_mount: /run: /run tmpfs tmpfs noexec,nosuid,size=10%,mode=0755
update_mount: /: / rpool/ROOT/ubuntu-1 zfs defaults check
update_mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections none fusectl optional
update_mount: /sys/fs/cgroup: /sys/fs/cgroup cgroup tmpfs rw,relatime,rw,mode=755
update_mount: /sys/kernel/debug: /sys/kernel/debug none debugfs optional
update_mount: /sys/kernel/security: /sys/kernel/security none securityfs optional
update_mount: /run/lock: /run/lock none tmpfs nodev,noexec,nosuid,size=5242880
update_mount: /run/shm: /run/shm none tmpfs nosuid,nodev
update_mount: /sys/fs/cgroup/cpuset: /sys/fs/cgroup/cpuset cgroup cgroup rw,relatime,rw,cpuset
update_mount: /sys/fs/cgroup/cpu: /sys/fs/cgroup/cpu cgroup cgroup rw,relatime,rw,cpu
update_mount: /sys/fs/cgroup/cpuacct: /sys/fs/cgroup/cpuacct cgroup cgroup rw,relatime,rw,cpuacct
update_mount: /sys/fs/cgroup/memory: /sys/fs/cgroup/memory cgroup cgroup rw,relatime,rw,memory
update_mount: /boot/grub: /boot/grub /dev/sda1 ext3 defaults check
update_mount: /sys/fs/cgroup/devices: /sys/fs/cgroup/devices cgroup cgroup rw,relatime,rw,devices
update_mount: /sys/fs/cgroup/freezer: /sys/fs/cgroup/freezer cgroup cgroup rw,relatime,rw,freezer
update_mount: /sys/fs/cgroup/blkio: /sys/fs/cgroup/blkio cgroup cgroup rw,relatime,rw,blkio
update_mount: /sys/fs/cgroup/perf_event: /sys/fs/cgroup/perf_event cgroup cgroup rw,relatime,rw,perf_event
update_mount: /home: /home rpool/home zfs zfsutil
update_mount: /mnt/vol0: /mnt/vol0 vol0 zfs zfsutil
update_mount: /rpool: /rpool rpool zfs zfsutil
update_mount: /rpool/ROOT: /rpool/ROOT rpool/ROOT zfs zfsutil
update_mount: /proc/sys/fs/binfmt_misc: /proc/sys/fs/binfmt_misc binfmt_misc binfmt_misc nodev,noexec,nosuid,optional
update_mount: /mnt/vol1: /mnt/vol1 vol1 zfs zfsutil
update_mount: /mnt/vol2: /mnt/vol2 vol2 zfs zfsutil
update_mount: /z: /z none aufs create=pmfs,sum,xino=/run/.aufs.xino,br:/mnt/vol5=rw:/mnt/vol3=rw:/mnt/vol2=rw:/mnt/vol1=rw:/mnt/vol0=ro
update_mount: /mnt/vol3: /mnt/vol3 vol3 zfs zfsutil
update_mount: /mnt/vol5: /mnt/vol5 /dev/sdl btrfs device=/dev/disk/by-id/scsi-SATA_WDC_WD20EARS-00_WD-WCAZA1018148,device=/dev/disk/by-id/scsi-SATA_WDC_WD20EARS-00_WD-WCAYY0229139,compress=zlib
mounted: /mnt/vol5
plymouth_connect: Failed to connect to Plymouth: Connection refused
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mountinfo_watcher: mountinfo changed, reparsing
parse_mountinfo_file: updating mounts
update_mount: /sys: /sys sysfs sysfs nodev,noexec,nosuid
update_mount: /proc: /proc proc proc nodev,noexec,nosuid
update_mount: /dev: /dev udev devtmpfs mode=0755
update_mount: /dev/pts: /dev/pts devpts devpts noexec,nosuid,gid=tty,mode=0620
update_mount: /run: /run tmpfs tmpfs noexec,nosuid,size=10%,mode=0755
update_mount: /: / rpool/ROOT/ubuntu-1 zfs defaults check
update_mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections none fusectl optional
update_mount: /sys/fs/cgroup: /sys/fs/cgroup cgroup tmpfs rw,relatime,rw,mode=755
update_mount: /sys/kernel/debug: /sys/kernel/debug none debugfs optional
update_mount: /sys/kernel/security: /sys/kernel/security none securityfs optional
update_mount: /run/lock: /run/lock none tmpfs nodev,noexec,nosuid,size=5242880
update_mount: /run/shm: /run/shm none tmpfs nosuid,nodev
update_mount: /sys/fs/cgroup/cpuset: /sys/fs/cgroup/cpuset cgroup cgroup rw,relatime,rw,cpuset
update_mount: /sys/fs/cgroup/cpu: /sys/fs/cgroup/cpu cgroup cgroup rw,relatime,rw,cpu
update_mount: /sys/fs/cgroup/cpuacct: /sys/fs/cgroup/cpuacct cgroup cgroup rw,relatime,rw,cpuacct
update_mount: /sys/fs/cgroup/memory: /sys/fs/cgroup/memory cgroup cgroup rw,relatime,rw,memory
update_mount: /boot/grub: /boot/grub /dev/sda1 ext3 defaults check
update_mount: /sys/fs/cgroup/devices: /sys/fs/cgroup/devices cgroup cgroup rw,relatime,rw,devices
update_mount: /sys/fs/cgroup/freezer: /sys/fs/cgroup/freezer cgroup cgroup rw,relatime,rw,freezer
update_mount: /sys/fs/cgroup/blkio: /sys/fs/cgroup/blkio cgroup cgroup rw,relatime,rw,blkio
update_mount: /sys/fs/cgroup/perf_event: /sys/fs/cgroup/perf_event cgroup cgroup rw,relatime,rw,perf_event
update_mount: /home: /home rpool/home zfs zfsutil
update_mount: /mnt/vol0: /mnt/vol0 vol0 zfs zfsutil
update_mount: /rpool: /rpool rpool zfs zfsutil
update_mount: /rpool/ROOT: /rpool/ROOT rpool/ROOT zfs zfsutil
update_mount: /proc/sys/fs/binfmt_misc: /proc/sys/fs/binfmt_misc binfmt_misc binfmt_misc nodev,noexec,nosuid,optional
update_mount: /mnt/vol1: /mnt/vol1 vol1 zfs zfsutil
update_mount: /mnt/vol2: /mnt/vol2 vol2 zfs zfsutil
update_mount: /z: /z none aufs create=pmfs,sum,xino=/run/.aufs.xino,br:/mnt/vol5=rw:/mnt/vol3=rw:/mnt/vol2=rw:/mnt/vol1=rw:/mnt/vol0=ro
update_mount: /mnt/vol3: /mnt/vol3 vol3 zfs zfsutil
update_mount: /mnt/vol5: /mnt/vol5 /dev/sdl btrfs device=/dev/disk/by-id/scsi-SATA_WDC_WD20EARS-00_WD-WCAZA1018148,device=/dev/disk/by-id/scsi-SATA_WDC_WD20EARS-00_WD-WCAYY0229139,compress=zlib
mounted event handled for /z
local 2/4 remote 0/0 virtual 27/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /tmp
local 3/4 remote 0/0 virtual 27/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /mnt/vol3
virtual finished
remote finished
local 3/4 remote 0/0 virtual 28/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused
mounted event handled for /mnt/vol5
local finished
All filesystems mounted
local 4/4 remote 0/0 virtual 28/28 swap 0/0
fsck_update: updating check priorities
plymouth_connect: Failed to connect to Plymouth: Connection refused

[mk01]

compile and try https://github.com/mk01/mountall

I added the changes discussed. But not the strict order dependency. This is still there.

I have not tested, not even started the binary. Just tested if still compile.

[mrdrmmn]

I did finally get a chance to compile and test and it is still results in the aufs mount occuring first and then the zfs mount occuring after the aufs mount starts preventing data on the zfs volume from being accessible via the aufs mountpoint.

[mk01]

there is a huge typo, this was intention dcc7119

give it one minute more, if it not helps, wait for the cumulative updates what @dajhorn is preparing here #5

[mrdrmmn]

Gave that a try and it also did not change the behavior. I've looked into the code for mountall and I don't think this is zfs specific at this point. There are a few reasons I believe this:

• The code does not take into consideration that a mountpoint can have multiple parents. This would be required to account for aufs mountpoints properly.
• The code does not consider the aufs mount options when looking at the parents. The is_parent function only considers the string value of the mount points.

I suppose it is possible that some other bit of code is supposed to be handling this functionality (I'm not really a c programmer nor have I looked at this code before today) but it doesn't appear to be. At the same time, it seems like to make this work, mountall would have to understand the aufs mount options (almost certainly the wrong thing to do) or it would need to be able to ask aufs in some way what paths it depends on (probably closer to the right way, but will be much more involved).

Either way, unless you believe I am incorrect in my assessment, I think it is probably best to call this an upstream bug and consider it a "won't fix" from the perspective of zfsonlinux as I am certain there is plenty to do without trying to fix upstream bugs. :)

Thanks for looking into this.

[mk01]

Although not studied actual code of mountall (upstream up-to-date version) very deeply, there were some significant changes between 2.3x and 2.4x.

Makes no sense to reinvent america again (but again I don't know how far upstream is with aufs) and put it into the official mountall version for u/precise. Darik did what he could to have it working with ZFS.

Darik has 2.46 (last time I checked) somewhere - he is maintaining the zfs-versions of the packages, so you can ask him. What I remember, the mounting code of 2.4 is completely paralleled - few (if any) of this 2.36 design flaws is valid for upstream.

But I could be wrong. Anyhow I had the feeling that exactly the test for aufs and updating the mount point record instead of creating new would solve the situation.

[dajhorn]

Either way, unless you believe I am incorrect in my assessment, I think it is probably best to call this an upstream bug and consider it a "won't fix" from the perspective of zfsonlinux as I am certain there is plenty to do without trying to fix upstream bugs.

Your assessment seems correct, but if you submit an aufs patch for mountall that is easy to understand, then I would consider adding it to the package here.

Past that, you might need to wait six or twelve months for a fix to come downstream.