[RFE] Auto hot swapping non validating systems

[grafuls]

Is your feature request related to a problem? Please describe.
Bare metal is hard and more often than not some systems get stuck on provisioning or validation. This is a problem particularly with self scheduling as the automation is relying on getting those systems through.

Describe the solution you'd like
Add a timeout for validation that would find the problematic hosts after a set timeout and replace with a system of the same model type for the process to hopefully complete. Add some notification to inform admins of the problematic host for a hands on approach.

[sadsfae]

I love this idea, we'll want this to be an enable/disable feature and have some level of intelligence to ensure the hardware matches between problem host vs potential replacement host beyond the model.

I think it would be useful for all types of scheduling so we might want an option in both selfservice.yml and quads.yml to cover self-scheduled and deliberate assignments.

Some things to Discuss

Do we want to auto mark broken problematic hosts?

We certainly want a postman notification for any swaps and thus a notification flag to set / send notification / reset should there be more than one that needs a swap. Do we want to have a set limit to number of times a host can get swapped out of a cloud before we hard fail it and stop and flag it for a person to look at?

We'll likely want a cooldown period and a mechanism to check/set a threshold for number of failed validation attempts for each host. If validate_env runs every 20 minutes then for example it could try auto-replacing a problem host after 3 x failures or one hour.

auto_replace_failed: true
auto_replace_failed_count: 3

[grafuls]

Initially I wouldn't mark the system as broken straight away as we might run out of non broken systems pretty rapidly.
Most of the times those systems need just a human nudge to get going so I believe the notification to admins would suffice.

[sadsfae]

Initially I wouldn't mark the system as broken straight away as we might run out of non broken systems pretty rapidly. Most of the times those systems need just a human nudge to get going so I believe the notification to admins would suffice.

Right, but mention this so we don't try to swap an additional host with the same one we swapped out prior so marking broken would keep that from happening. For an MVP initial implementation we can just try to swap the first occurrence of a problematic host after N number of validate_env failures. This would keep the notification and assignment flags / reset simple.