scsi reservations issue on failover

[WanWizard]

Hey, followed your great instructions to the letter, but I'm left with a situation that leaves me stumped.

I have a setup with two supermicro's, each connected to 2 12-disk JBOD's with SAS disks, but without a loop, so no multipath (and multipath is not installed). Both JBOD's are used in mirrored vdevs, so I can lose an entire JBOD without much issues.

OS: CentOS Linux release 7.6.1810 (Core)
ZFS: 0.7.13, from the zfs-kmod repo

This setup works fine, until pacemaker decides there is a need to failover. It doesn't matter if that is because the active node is put into standby, because the hardware is switched off, etc.

When pacemaker fails over, the second node tries to import the pool, which fails because something on the first node has placed SCSI reservations on the disk:

[root@nas01 /]# sg_persist -r /dev/sdh
  NETAPP    X412_HVIPC560A15  NA02
  Peripheral device type: disk
  PR generation=0x1, Reservation follows:
    Key=0x666e0001
    scope: LU_SCOPE,  type: Write Exclusive, registrants only

as soon as the failover happens, the second node starts to log:

[ 5834.890588] sd 0:0:7:0: reservation conflict
[ 5834.890674] sd 0:0:7:0: reservation conflict
[ 5834.890693] sd 0:0:7:0: [sdh] Test Unit Ready failed: Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[ 5834.891274] sd 0:0:7:0: reservation conflict
[ 5834.891369] sd 0:0:7:0: reservation conflict
[ 5834.891466] sd 0:0:7:0: reservation conflict
[ 5834.891560] sd 0:0:7:0: reservation conflict
[ 5834.921402]  sdh: sdh1 sdh9
[ 5834.922452] sd 0:0:7:0: reservation conflict
[ 5834.957331] sd 0:0:7:0: reservation conflict
[ 5834.958157] sd 0:0:7:0: reservation conflict
[ 5835.052881] sd 0:0:7:0: reservation conflict

which either causes the entire import to fail, or, if the import succeeds, with disks offline due to excessive errors.

I've been pulling my hair out for about two weeks now, but no clue what sets these reservations, or how I can have them released on a cluster start or a cluster failover. There seem to be lots of people building Linux HA clusters with ZFS judging the discussions I found, but no one mentions this issue...

[ewwhite]

Hello,

This doesn't follow the spirit of what I documented.
The idea of my instructions is to use JBODs with dual-ported SAS disks and multipath cabling.

Why doesn't your design use multipath?

[WanWizard]

Don't have the space for a second HBA, servers have only a few low-profile slots. One has the HBA, the other a dual 10G card for NFS connectivity.

This is a low-budget op with second hand hardware, paid by donations, to provide storage for the build/compile servers of an open source development team I help out with some infra and admin work.

I understand that two HBA's and multipathing would provide additional availability, but unfortunately it is what it is. Until a big sponsor comes along... ;-)

[milleroff]

We had the same problems with SCSI reservations as it never worked as expected. Sometimes when a failover happened, the new active server could not import the disks.
We ended up setting the IPMI fencing. If something goes wrong, the second server shuts off the first server over the IPMI protocol and takes the control over the disks.

[WanWizard]

Good to read I'm not alone. Not good that you needed to work around it like that. I'd hoped to avoid that.

[ewwhite]

I haven't had such issues with any deployment.
You can use a single dual-port HBA in each host. Dual HBA cards are not required.

@milleroff Please make sure meta provides=unfencing, pcmk_monitor_action="metadata" and pcmk_host_list= is populated in your SCSI stonith resource. Also, this absolutely requires dual-port SAS drives everywhere.

[WanWizard]

That is indeed how I've hooked it up now, one port going to each of the enclosures.

All my disks are Hitachi HUS156060VLS600, which are dual-port SAS drives. I don't have any SCSI fencing active, removing that was my first step in trying to find the problem.

[ewwhite]

SCSI fencing is crucial to what you're doing. That's how the failover and pool import work.

[WanWizard]

So what is setting the reservations, as it's not the fence_scsi agent?

I get that in production you need it to avoid imports on both nodes (which would be utter horror), but if it doesn't work in a controlled failover (where the pools are cleanly exported and a node is cleanly shutdown to tigger a failover), I don't see how adding another layer of complexity will fix the issue.

[ewwhite]

Controlled and uncontrolled failovers work in the setup I've described and documented.

I do not know what's unique about your environment, but removing critical components of the design isn't going to help the situation. What is the output of zpool status -v and lsscsi and multipath -ll?

This high-availability design assumes:
Multipath SAS cabling
Dual-ported disks
Multipath service enabled
ZFS pool creation using dm-multipath devices (not individual /dev/sdX SCSI disks)
SCSI reservation fencing

[WanWizard]

I get that. I'd rather work with a documented (and if possible supported) environment as well. But as said, it is what it is. ;-)

[root@nas01 /]# zpool status -v
  pool: sas01
 state: ONLINE
  scan: none requested
config:

        NAME                        STATE     READ WRITE CKSUM
        sas01                       ONLINE       0     0     0
          mirror-0                  ONLINE       0     0     0
            wwn-0x5000cca01fa899a4  ONLINE       0     0     0
            wwn-0x5000cca01fa81cfc  ONLINE       0     0     0
          mirror-1                  ONLINE       0     0     0
            wwn-0x5000cca02a6a7e6c  ONLINE       0     0     0
            wwn-0x5000cca01fcf0128  ONLINE       0     0     0
          mirror-2                  ONLINE       0     0     0
            wwn-0x5000cca01f8f1394  ONLINE       0     0     0
            wwn-0x5000cca0411f4c48  ONLINE       0     0     0
          mirror-3                  ONLINE       0     0     0
            wwn-0x5000cca01fcc73ec  ONLINE       0     0     0
            wwn-0x5000cca02a018bf8  ONLINE       0     0     0
          mirror-4                  ONLINE       0     0     0
            wwn-0x5000cca01f47b644  ONLINE       0     0     0
            wwn-0x5000cca018511124  ONLINE       0     0     0
          mirror-5                  ONLINE       0     0     0
            wwn-0x5000cca01fda745c  ONLINE       0     0     0
            wwn-0x5000cca02a6b3e20  ONLINE       0     0     0
          mirror-6                  ONLINE       0     0     0
            wwn-0x5000cca01fa6e548  ONLINE       0     0     0
            wwn-0x5000cca018d80018  ONLINE       0     0     0
          mirror-7                  ONLINE       0     0     0
            wwn-0x5000cca02a67e1e0  ONLINE       0     0     0
            wwn-0x5000cca02a0070e0  ONLINE       0     0     0

errors: No known data errors

[root@nas01 /]# lsscsi
[0:0:0:0]    disk    NETAPP   X412_HVIPC560A15 NA04  /dev/sdb
[0:0:1:0]    disk    NETAPP   X412_HVIPC560A15 NA02  /dev/sdc
[0:0:2:0]    disk    NETAPP   X412_HVIPC560A15 NA02  /dev/sdd
[0:0:3:0]    disk    NETAPP   X412_HVIPC560A15 NA02  /dev/sde
[0:0:4:0]    disk    NETAPP   X412_HVIPC560A15 NA02  /dev/sdf
[0:0:5:0]    disk    NETAPP   X412_HVIPC560A15 NA04  /dev/sdg
[0:0:6:0]    disk    NETAPP   X412_HVIPC560A15 NA02  /dev/sdh
[0:0:7:0]    disk    NETAPP   X412_HVIPC560A15 NA04  /dev/sdi
[0:0:8:0]    enclosu LSI      SAS2X28          0e12  -
[0:0:9:0]    disk    NETAPP   X412_HVIPC560A15 NA02  /dev/sdj
[0:0:10:0]   disk    NETAPP   X412_HVIPC560A15 NA04  /dev/sdk
[0:0:11:0]   disk    NETAPP   X412_HVIPC560A15 NA02  /dev/sdl
[0:0:12:0]   disk    NETAPP   X412_HVIPC560A15 NA04  /dev/sdm
[0:0:13:0]   disk    NETAPP   X412_HVIPC560A15 NA02  /dev/sdn
[0:0:14:0]   disk    NETAPP   X412_HVIPC560A15 NA04  /dev/sdo
[0:0:15:0]   disk    NETAPP   X412_HVIPC560A15 NA02  /dev/sdp
[0:0:16:0]   disk    NETAPP   X412_HVIPC560A15 NA04  /dev/sdq
[0:0:17:0]   enclosu LSI      SAS2X28          0e12  -
[2:0:0:0]    disk    ATA      SAMSUNG SSD CM85 3D2Q  /dev/sda

The SSD disk is a 32GB SATA DOM the server boots from.

I don't have a multipath setup, so no multipath service installed. And therefore no multipath output, and no dm-multipath devices.

I know my setup isn't as documented, but I was still hoping someone knew where these reservations came from, so I could work with/around them. I have no issues writing a resource agent to deal with those if needed, if that is what it takes.

Thanks sofar. I have reinstated the fence_scsi agent now, will try another failover tomorrow, I'm in GMT+2, getting late here... ;-)

[ewwhite]

I'd enable the multipath daemon, re-create the pool with the resulting /dev/mapper devices and add a fencing resource containing those /dev/mapper devices.

pcs stonith create fence-vol1 fence_scsi pcmk_monitor_action="metadata" pcmk_host_list="zfs-node1,zfs-node2" devices="/dev/mapper/35000c500236061b3,/dev/mapper/35000c500236032f7,/dev/mapper/35000c5007772e5ff,/dev/mapper/35000c50023614aef,/dev/mapper/35000a7203008de44,/dev/mapper/35000c500236004a3,/dev/mapper/35000c5002362ffab,/dev/mapper/35000c500236031ab,/dev/mapper/35000c50023605c1b,/dev/mapper/35000c500544508b7,/dev/mapper/35000c5002362f347" meta provides=unfencing

Your NetApp shelves should allow you to do this.

[WanWizard]

Ok, will do that tomorrow evening. Thanks for the help sofar.

[WanWizard]

Decided to restart the project completely. Formatted and cleared everything, reinstalled CentOS and ZFS, following the wiki, virtually to the letter.

Difference is I decided not to use multipath, I've had a chat with some redhat DC guys today, and they all advised me not to use it when there is no multipath in use, to avoid another layer of complexity. So I followed their advice, and used "/dev/disk/by-id" instead of "/dev/mapper". Seems to work fine sofar.

Just tested a few failovers by switching nodes to standby and back and faking network issues, and that seems to work fine now, including the SCSI fencing. Happy days.

Only one issue left: when ZFS fails over, the shares aren't activated after the failover, and I need to do a zfs share -a to get them active again. Put one node in standby, and restarted the active node, after which the same issue occured. /etc/default/zfs' has ZFS_SHARE='yes', and the zfs-share.service is enabled.

No idea where to look next. I didn't have this problem before, so I seem to roll from one issue to the next...

[ewwhite]

I don't understand what you are trying to do by avoiding multipath, as it is a key element of this design.

I understand you're seeking assistance, but you have not clearly articulated the reasoning behind not using multipath devices. If there's an architectural issue preventing multipath cabling, please explain.

[WanWizard]

I don't have multipaths to my storage, so it totally pointless to install and use multipath, every device only has one path.

I get it is a key element of your design, but I don't have the hardware to match, I have already explained that to you (I only have one HBA per server, and only place for one).