Host HA not working properly

[cdfgallo]

ISSUE TYPE

• Bug Report

COMPONENT NAME

NFS Primary Storage

CLOUDSTACK VERSION

4.18.1.0

CONFIGURATION

advanced networking
primary storage on NFSv3
host ha enabled on the cluster with a specific host dedicated to HA

OS / ENVIRONMENT

management server with OS centOS7
hosts with OS Ubuntu 22.04

SUMMARY

If I shutdown a host, it keeps a lock on NFS mount that prevent all VMs that were running on it to automatically restart on the dedicated HA host.
Also the lock prevents to start these VMs until the host is online again and able to release the lock.

STEPS TO REPRODUCE

In a cluster with primary storage NFSv3, shutdown a host with VMs running on it

EXPECTED RESULTS

The lock is released as soon as the host is down, or after a short delay.

ACTUAL RESULTS

The lock is not released until the host is back online

[DaanHoogland]

@cdfgallo this sounds like an issue with nfs where locks are kept until a proper umount has been done. I would expect this to happen if a host gracefully shuts down. if a host crashes it would not happen, at least not in older nfs versions afairc.
we might have to add a shutdown script to take care of that, but that will only help on gracefull shutdowns.

[cdfgallo]

@DaanHoogland this issue happens when a host is not gracefully shutdown (lost power) or it lose network connections.
I haven't tried to shutdown gracefully the host as I would put it into maintenance mode before doing so.

[DaanHoogland]

@DaanHoogland this issue happens when a host is not gracefully shutdown (lost power) or it lose network connections. I haven't tried to shutdown gracefully the host as I would put it into maintenance mode before doing so.

I don't think that test would make a lot of sense in real life. Can you test this setup with NFSv4

[cdfgallo]

So it is an expected behaviour that a host being down keeps the lock on VMs file on NFSv3?
In this case we need for sure to try NFSv4

[DaanHoogland]

So it is an expected behaviour that a host being down keeps the lock on VMs file on NFSv3? In this case we need for sure to try NFSv4

this is a downside of NFSv3, yes.

[cdfgallo]

On NFS4 I have no issues. I'm closing this issue. Thank you @DaanHoogland !

[NuxRo]

For those coming here from search engines and other sites, the solution is to use NFSv4.

BTW, since CloudStack 4.19.1 it is possible to specify the version per NFS primary storage:
#8947

Another way of doing it (the bad way) is to mount the NFS share with "nolock" option, but this is impractical and dangerous.