Improve 'cluster health sbd' command

[liangxin1300]

Problems

This PR is going to address the issues below brought by #1932:

• Cluster members are SSH unreachable
  Functionalities are aborted due to SSH unreachable

# crm cluster health sbd
ERROR: cluster.health: Failed on root@sle16-2: Cannot create SSH connection to root@sle16-2: ssh: connect to host sle16-2 port 22: Connection refused
Connection closed

• Failed to configure crashdump-watchdog-timeout

# crm sbd configure crashdump-watchdog-timeout=30
INFO: Set crashdump option for fence_sbd resource
INFO: Set msgwait-timeout to 2*watchdog-timeout + crashdump-watchdog-timeout: 60
INFO: Update SBD_TIMEOUT_ACTION in /etc/sysconfig/sbd: flush,crashdump
INFO: Configuring disk-based SBD
INFO: Initializing SBD device /dev/sda5
INFO: Update SBD_OPTS in /etc/sysconfig/sbd: -C 30
INFO: Sync file /etc/sysconfig/sbd to sle16-2
INFO: Adjusting sbd msgwait to 90, watchdog timeout to 15
ERROR: sbd.configure: Failed to run 'crm sbd configure msgwait-timeout=90 watchdog-timeout=15': ERROR: sbd.configure: Failed to run 'crm sbd configure msgwait-timeout=120 watchdog-timeout=15':

• The current minimum value of watchdog timeout is always 5

Changes

• Cluster members are SSH unreachable
  • Give a warning to skip the consistency check
  • Perform checking functionalities, raise warning/error when checking fails
  • Give an error that the issue can't be fixed due to SSH unreachable

# crm cluster health sbd
WARNING: Skip configuration consistency check due to unreachable nodes
ERROR: It's recommended that SBD_DELAY_START is set to 71, now is no
ERROR: Cannot fix SBD_DELAY_START issue: There are nodes whose SSH ports are unreachable: sle16-2.
Please check the network connectivity before check and fix SBD timeout configurations.
ERROR: SBD: Check sbd timeout configuration: FAIL.

• sbd: Calculate expected msgwait timeout correctly with crashdump timeout

# crm sbd configure crashdump-watchdog-timeout=30
INFO: Set crashdump option for fence_sbd resource
INFO: Set msgwait-timeout to 2*watchdog-timeout + crashdump-watchdog-timeout: 60
INFO: Update SBD_TIMEOUT_ACTION in /etc/sysconfig/sbd: flush,crashdump
INFO: Configuring disk-based SBD
INFO: Initializing SBD device /dev/sda5
INFO: Update SBD_OPTS in /etc/sysconfig/sbd: -C 30
INFO: Sync file /etc/sysconfig/sbd to sle16-2
INFO: Update SBD_DELAY_START in /etc/sysconfig/sbd: 101
INFO: Sync file /etc/sysconfig/sbd to sle16-2
INFO: Adjusting systemd start timeout for sbd.service to 121s
INFO: Sync directory /etc/systemd/system/sbd.service.d to sle16-2
INFO: Adjusting stonith-timeout to 83
WARNING: "stonith-timeout" in crm_config is set to 83, it was 71
INFO: Restarting cluster service
INFO: BEGIN Waiting for cluster
.......                                                                                                                                                                                                                      
INFO: END Waiting for cluster

• ui_sbd: Get minimum timeout value dynamically

# minimum watchdog timeout is from profiles.yml
# crm sbd configure watchdog-timeout=1
ERROR: sbd.configure: The minimum value for watchdog-timeout is 15
# crm sbd configure msgwait-timeout=2
ERROR: sbd.configure: The minimum value for msgwait-timeout is 30

# when no watchdog timeout is defined in profiles.yml
# crm sbd configure watchdog-timeout=1
ERROR: sbd.configure: The minimum value for watchdog-timeout is 5
# crm sbd configure msgwait-timeout=2
ERROR: sbd.configure: The minimum value for msgwait-timeout is 10

• sbd: Check and fix drop-in file which to unset SBD_DELAY_START
  Fix: crmsh could aggressively populate all nodes with /run/systemd/system/sbd.service.d/sbd_delay_start_disabled.conf #2001

# crm cluster health sbd
WARNING: Runtime drop-in file /run/systemd/system/sbd.service.d/sbd_delay_start_disabled.conf to unset SBD_DELAY_START is missing
INFO: Please run "crm cluster health sbd --fix" to fix the above warning
INFO: SBD: Check sbd timeout configuration: OK.

# crm cluster health sbd --fix
INFO: Createing runtime drop-in file /run/systemd/system/sbd.service.d/sbd_delay_start_disabled.conf to unset SBD_DELAY_START
INFO: SBD: Check sbd timeout configuration: OK.

• Show multi errors or warnings at once if detected

# crm sbd configure show
...
ERROR: It's recommended that SBD_DELAY_START is set to 131, now is 31
ERROR: It's recommended that stonith-timeout is set to 119, now is 19
WARNING: Runtime drop-in file /run/systemd/system/sbd.service.d/sbd_delay_start_disabled.conf to unset SBD_DELAY_START is missing
INFO: Please run "crm cluster health sbd --fix" to fix the above error

• Now the check items include:
  • /etc/corosync/corosync.conf consistency
  • /etc/sysconfig/sbd consistency
  • SBD disk metadata
  • SBD devices metadata consistency
  • SBD_WATCHDOG_TIMEOUT
  • SBD_DELAY_START
  • systemd start timeout for sbd.service
  • stonith-watchdog-timeout property
  • stonith-timeout property
  • unset SBD_DELAY_START in drop-in file
  • sbd.service should be enabled
  • fence_sbd agent

[codecov]

Codecov Report

❌ Patch coverage is 72.36025% with 89 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.33%. Comparing base (0de9df8) to head (040914e).

Files with missing lines	Patch %	Lines
crmsh/sbd.py	71.49%	65 Missing ⚠️
crmsh/utils.py	64.58%	17 Missing ⚠️
crmsh/ui_cluster.py	0.00%	3 Missing ⚠️
crmsh/ui_sbd.py	92.59%	2 Missing ⚠️
crmsh/xmlutil.py	50.00%	2 Missing ⚠️

Additional details and impacted files

Flag	Coverage Δ
integration	55.20% <24.22%> (-0.22%)	⬇️
unit	52.57% <68.63%> (+0.10%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crmsh/bootstrap.py	87.93% <100.00%> (+0.05%)	⬆️
crmsh/ui_sbd.py	83.72% <92.59%> (+0.35%)	⬆️
crmsh/xmlutil.py	70.20% <50.00%> (-0.09%)	⬇️
crmsh/ui_cluster.py	71.32% <0.00%> (+0.49%)	⬆️
crmsh/utils.py	66.94% <64.58%> (+0.04%)	⬆️
crmsh/sbd.py	80.12% <71.49%> (-2.98%)	⬇️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[zzhou1]

Very good UX improvement!