HEP: System robustness enhancement [CI SKIP]

[w13915984028]

IMPORTANT: Please do not create a Pull Request without creating an issue first.

HEP: System robustness enhancement

Signed-off-by: Jian Wang <w13915984028@gmail.com>

Problem:

How to gracefully shutdown an running Harvester cluster? no clear answer yet.

Solution:

Add to a new feature to accomplish this.

Related Issue:
#3261

One previous DOC PR: docs: add cluster shutdown and restart

Test plan:

[w13915984028]

BTW: all review comments are welcome, thanks.

[vtrenton]

Brainstorming:

1. Shut down VMs using kube-virt graceful shutdown process: https://github.com/kubevirt/kubevirt/blob/main/docs/graceful-shutdown.md
2. signal k8s to drain/cordon harvester node.
3. assure all workloads like longhorn have been removed from the node
4. shutdown node

oversimplifying a bit - but might be a good start.

[github-actions]

This repository uses an automated workflow to automatically label pull requests which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community pull requests better. If the pull request is still relevant, please add a comment to the pull request so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the pull request in 14 days. Thank you for your contributions.

[markhillgit]

reopening, this is still meant to be active

[github-actions]

This repository uses an automated workflow to automatically label pull requests which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community pull requests better. If the pull request is still relevant, please add a comment to the pull request so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the pull request in 14 days. Thank you for your contributions.

[markhillgit]

ping, keeping this PR active

[github-actions]

This repository uses an automated workflow to automatically label pull requests which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community pull requests better. If the pull request is still relevant, please add a comment to the pull request so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the pull request in 14 days. Thank you for your contributions.