Handling cleanup after hardware failures

[lentzi90]

I want to discuss procedures and experiences with hardware failures and how to clean up after them.

Scenario:

A host fails catastrophically and cannot be fixed. The host was registered as a BareMetalHost and provisioned when it happened.
There is now no way to deprovision the BMH, so we are stuck with a BMH object that cannot be deleted without resorting to things like manually deleting the finalizers.

Additionally, if the BMH was consumed by a Metal3Machine as part of a CAPI Cluster, the Machine object will also be stuck waiting for the Metal3Machine, which is waiting on the BMH to be deprovisioned. (This could be an issue even for temporary issues that can be fixed. A user may not want to wait a week for a spare part with a cluster stuck with 2/3 control-plane nodes healthy.)

What I am after in this discussion is to figure out how such scenarios "should" be handled. I also want to probe the interest for new features to handle this in a more automatic way. To the best of my knowledge there isn't really any documentation or automation around this currently in Metal3. The user would have to figure out which resources are stuck and manually remove them.

As a first step, let's figure out what needs cleaning up. I know about these:

1. BareMetalHost
2. BMC credentials secret

Both of these have finalizers and will get stuck if BMO cannot deprovision. Do you know of more? How about HardwareData?

For CAPM3/IPAM objects, I believe that once the BMH is gone, they will automatically be cleaned up, but I am not completely sure about this.

My first attempt at documentation for this:

1. Remove the finalizer from the BareMetalHost after confirming that the underlying host is truly gone and data destroyed.
2. Remove the finalizer from the BMC credentials secret.
3. Delete the Ironic pod to clear the node from the database (I am sure there are better ways to do this, please enlighten me).

[pratikrath126]

Hey! Good topic. We've run into this in prod too. Beyond BMH and BMC secret, you'll typically also hit:

• Metal3Machine (if the BMH was part of a CAPI cluster) – it has a finalizer from the Metal3Machine controller.
• HardwareData (CR) associated with the BMH.
• IPAMClaim/IPAMHandle if dynamic IP allocation was used.
• Node object (if Kubernetes node was created) and possibly related cloud-config secrets.

For Ironic DB cleanup: you don't need to delete the pod; you can clean up the node via ironic node-delete <uuid> if you know the BMC address. Alternatively, just removing the BMH finalizers and letting the operator reconcile usually removes the ironic node eventually.

A scriptable pattern we use:

1. Annotate BMH with metal3.io/force-deprovision: true (if BMO supports it) to skip waiting for BMC.
2. Remove finalizers from BMH, Metal3Machine, HardwareData, IPAMClaim.
3. Delete Node (if present) and related secrets.
4. Optionally run kubectl delete pod -l app=ironic -n baremetal-operator-system to force a DB re-sync after cleanup.

Would be great to see BMO add a force-delete flag that does this automatically.

[dtantsur]

Alternatively, just removing the BMH finalizers and letting the operator reconcile usually removes the ironic node eventually

This is far from guaranteed. Ironic itself does not remove "stale" nodes, and BMO cannot reconcile something that does not exist.

[dtantsur]

Some time ago, I proposed adding forced decomissioning to Ironic: https://bugs.launchpad.net/ironic/+bug/2133499. The feedback was mixed, but maybe input from this group can boost it?