flux assumes down nodes can come back, but this is only true for bootstrap from configuration

[grondo]

This is an issue related to resilient batch jobs, first mentioned in #6304. If a batch job has configured -o exit-timeout=none, and non-critical ranks are lost up to the point where there are not enough available nodes to run any pending job, the instance will hang until timeout. This is because Flux is currently designed to assume that down nodes may eventually return to service, but this not supported except when bootstrap uses a config file, so does not apply to jobs.

I'm not exactly sure how to address this, since the down node assumption is currently fundamental. Note that this cannot just be handled in a special submission feasibility plugin because jobs could already be pending when a node is lost.

[garlick]

As I recall the scheduler acquisition protocol doesn't provide a way to shrink the resource set - it just marks newly excluded nodes down. If it could, then jobs could fail the runtime feasibility check at least, which would be better than a hang.

Actually if we could make the excluded node set dynamically configurable then we could

• add something to automatically exclude down nodes in a batch instance
• potentially provide a mechanism to exclude spare nodes and activate them later

Also, the FLUB stuff was a first step toward making it possible to replace down nodes in a batch job. More to do to make that useful, but perhaps worth mentioning here

• broker: enable brokers to be added to running instances #5184

[grondo]

Making the excluded node set dynamically configurable seems like a good idea. However, I'm struggling with whether that's the best conceptual approach long term. It seems like this is just the old shrink (and for the case of spare nodes, grow) concepts we've had for a long time. Once a node is lost from a batch job, it is gone, perhaps replaced by a new node in the future (I don't actually see why this has to be a spare node allocated by the job).

Fluxion already supports grow/shrink and this could be trivially done in sched-simple, at least for librlist based on Rv1. What remains to be done here? My recollection of the issue with FLUB is that it requires pre-configuring a set of "fake" nodes that can join an instance in the future. So is the final piece here making it so the overlay can have a runtime adjustable size?

Anyway, it feels like we'll need to solve this one in two stages:

There are users using or at least experimenting with resilient batch jobs now, and the original issue described here seems like a critical flaw, so we need a short term solution soon. Perhaps making the exclude set dynamic is the easiest path forward here. I think this will require an addition to the resource acquisition protocol? (I do still think we consider if this should be a "shrink" operation and not necessarily "exclude", but up for debate)

Next, we should open a longer term issue on generic grow/shrink, where I think the schedulers are ready to handle this but flux-core is not.

[garlick]

Agreed, let's not expose "resource exclusion" to the scheduler in RFC 28. Let's just ad a "shrink by execution target" response.

We can open a medium term issue on what we think "full shrink" requires. Probably more than is required to resolve this issue. For example, recursively shrinking sub-instances, partial release, and any overlay considerations.

And yeah, a slightly longer term one for grow.

Thanks for keeping us on track for the grand design!

[grondo]

Now that this has been addressed in flux-core and Fluxion, let's close this specific issue.

There still exists one issue here - users need a way to easily determine if a job is failing due to a node failure, or a nonzero job exit status, or because the job is now unsatisfiable. I'll open a separate issue on that.