Remove/revise Slurm launch integration

[rhc54]

I've been getting reports lately of problems launching OMPI and PRRTE jobs under Slurm. Apparently, Slurm is now attempting to modify the ORTE/PRRTE internal srun command line used to launch the mpirun daemons by injecting an MCA parameter into the environment when the allocation is created.

This isn't the first time we have encountered problems with either discovering the Slurm allocation (where we rely on specific envars) or launching the daemons (where we rely on a stable srun cmd line). It has been a recurring problem that rears its head every couple of years. The other two MPIs that support the Slurm environment (MPICH and IntelMPI) both follow our same integration approach, so this is something that has impacted us all for some time.

In talking with SchedMD, the feeling on their side is that the various MPIs are interfering with their ability to develop Slurm, particularly when trying to add features. The requirement that specific envars and the srun cmd line remain stable is simply no longer going to be acceptable - some of their new features introduced last year just cannot accommodate it.

What they were trying to do was use the MCA param system to add a flag to the internal srun command that instructed srun to fallback to a "legacy" mode. However, that really doesn't work in a reliable manner. They can only inject it at the envar level, but we read values from default param files (which then get silently overwritten by their value) and take the final value of the param from the cmd line - which then overwrites the Slurm setting. So either the user unwittingly removes the new flag (and subsequently gets bizarre behavior as the daemons don't properly launch), or sys admins and users wind up confused when their flags simply "disappear".

One could argue that the param in question (plm_slurm_args) isn't that heavily used. However, that may not be universally true - and is impossible to really validate. My initial report came from an AWS user who couldn't get PRRTE to launch. Wenduo and I tried to advise and help debug the situation, to no avail. It was only after the user gave up that I finally figured out the root cause of the problem - aided by getting other reports of similar issues.

I have tried to propose alternative approaches based on what other environments have done. For example, LSF and PBS provide us with a launch API that is stable, but leaves them free to modify the underlying implementation at will to conform to their own changes. For resource discovery, pretty much everyone else provides a hostfile that we can read.

Unfortunately, SchedMD has rejected those proposals. Their basic position at the moment is that (a) we shouldn't be constraining their development path, and (b) they shouldn't have to provide us with alternative integration methods. I'm still pressing them for a positive proposal, but not getting very far at the moment. My sense is that they are frustrated by the problems over the years and are basically just giving up.

Assuming their stance doesn't change, the only alternative I can see is to remove the RAS and PLM Slurm components, and drop back to using hostfiles and ssh for starting the job. Since SchedMD is refusing to provide a hostfile upon making the allocation, it would place the burden on the user to generate it (note that executing srun hostname may not result in the correct hostfile due to recent changes in the srun behavior).

Beyond the issue of generating the hostfile, a move to using ssh means the loss of fine-grained accounting since Slurm will not see the processes launched by mpirun. When we launch the daemons via srun, the accounting system "sees" the daemon and therefore tracks all resources used by the daemon and its children (i.e., the app procs). When launched by ssh, this doesn't happen. So the overall job still gets accounted as the system "sees" mpirun itself and the use of resources on the nodes - but it doesn't provide per-proc tracking.

So we should discuss how we want to resolve this. The release of Slurm 23.11 is where the breakage occurs, so it is still early in the adoption process - but it breaks all existing OMPI releases, which is a problem we cannot control.

[wenduwan]

Related openpmix/prrte#1960

[bosilca]

A fair question is what is SchedMD seeing as a potential development path in which none of the parties constraint the development of the others ?

[rhc54]

That is a fair question - unfortunately, they haven't provided one other than insisting we stop using their cmd line and envars as they will not assure stability.

[hppritcha]

Problem for removing srun based launch of prrted's on systems of some interest like ORNL frontier and NERSC perlmutter is that we will lose support for using the OFI libfabric CXI provider, as it was via the slurmd that we acquire (or more precisely the CXI provider acquires) the VNI information needed for initialization.

[rhc54]

True, though that is readily remedied. Cray was contractually obligated to provide a method for doing that as part of Aurora, but refused to do so at the last minute. So the "hooks" are all present in PMIx - we are just missing the last step of inserting it into the CXI server on the compute nodes.

FWIW: many organizations that use Slingshot simply turn the VNI "off" as it actually does virtually nothing for you. It simply enforces that only procs operating in the same allocation can communicate with each other - but it takes little effort to demonstrate that attempting to decipher what an unknown application in a different allocation is doing by "sniffing" raw MPI exchanges is effectively impossible.

Obviously, policy overrides reality in some situations, so that may not be an option for some. Still, there are some fairly simple ways of solving this if someone cares to provide access to an appropriate machine, or if someone with such access is willing to collaborate on the solution.

[rhc54]

I had a very long (sometimes heated) discussion with SchedMD this afternoon, and I think we have hit upon a solution that is acceptable to both sides. I'll lay it out in more detail at the meeting next week, but basically we have agreed that:

• what is out there today is not something we can call back - it is done and gone. The changes introduced by Slurm 23.11 were well intentioned, but flawed - and we are going to have to live with them. Fortunately, they should hopefully not impact many people.
• we need to focus on the long-term solution for Slurm integration to try and break this continual cycle of problems, both in discovering the allocation and launching the job. We also need to define a method for this that provides adequate isolation between the two projects so Slurm can do what it wants/needs, without being hindered by OMPI's installed base.

We spent a fair amount of time kicking over possible approaches before finally agreeing on creating a separate Slurm integration library that would provide at least two APIs - one to retrieve allocated resources and the other to launch the job. Ideas were exchanged on how to make those APIs future-proof, how to detect which methods were supported by the Slurm version we are operating under, etc. What Slurm does inside the library to implement those APIs will be invisible to us - and frankly, we don't care. SchedMD is free to modify that to meet their evolving needs.

At the end, I think we are on a good path. Tim went off to pursue this more and will circle back with me next week for a more detailed look at the design. Initial goal is to (a) introduce a slight modification to the current envar method in Slurm 24.05 which will require a mod to PRRTE to support, and (b) include the new library-based method in Slurm 24.11 so it can be used in OMPI v6 and PRRTE v4 (or v5, depending on release schedules).

[jsquyres]

I think we need to update the Open MPI docs to explicitly state that SLURM >= 23.11 is silently injecting (and/or modifying?) the OMPI_MCA_plm_slurm_args environment variable. This may cause unexpected behavior in all versions of Open MPI (because the plm_slurm_args MCA param goes back many years / many Open MPI versions).

What else do we need to tell Open MPI users?

[rhc54]

A few more things for you folks to consider - just a "heads up" to help seed the discussion next week:

1. The modification SchedMD is requesting is that we (a) add the new --external-launcher flag to our Slurm plm plugin, and (b) extend that option to include the OMPI version (i.e., --external-launcher=ompi5.0.4). In order to do that, we will have to runtime detect the Slurm version we are operating under so we know that we can add that flag, and know that we can/should add the extension. SchedMD would provide a sentinel envar for this purpose - if that isn't sufficient, then we would have to execute srun -v to obtain the version prior to launch. They would like us to include this in the next OMPI and PRRTE releases.
2. SchedMD proposes that we use JSON as the go-between for the new library interfaces. It was clear that JSON would be used to return the allocation - not quite clear how JSON would be used in the launch API. This will require that we add a JSON parser to OMPI. I link to and use an external JSON library (libjansson) in PMIx for the PALS support (since everything in PALS is a web service accessed via curl), but I don't know how widely installed that library is or how willing admins will be to add it. It isn't a GPL library, so we could consider absorbing it or replicating just the parts we need - either way, this is something to consider.

Personally, I have to confess to being amused at the first request since it represents another modification of our hard-coded srun cmd line, coming almost immediately after being promised that --external-launcher would be the "forever" solution. We would of course have to check the value of the MCA param to ensure we don't also add it to the cmd line since that envar will still be out there. A little tricky, but doable if you want to agree to the request.

JSON for the allocation is workable, although I'm not wild about it. I'll have to wait to see their proposal next week to understand why we would pass JSON to the launch API - that one isn't intuitive to me.

Please come prepared to offer your advice and questions next week. If some of you want to be involved in this process with SchedMD, let me know - there is no intent to restrict involvement.

[rhc54]

So we have a bit of a problem. Slurm 24.05 has been released with the --external-launcher option now taking the optional argument. However, if we pass that argument to an earlier version of srun, it will error out - so we have to somehow determine when we can/should provide the option and when we should not.

SchedMD has proposed that we test-run srun to make the determination instead of trying to parse the srun --version output. So we would modify the slurm launcher to first try srun --external-launcher=foo --version to see if that fails. If it fails, then try srun --external-launcher --version to see if the --external-launcher was supported at all. Then we would go ahead and launch the daemons with the correct version of the cmd line.

My concern here is that we are potentially executing multiple srun calls to launch the daemons. Do folks find this of concern? Any other suggestions on how we might determine what cmd line to use?

[naughtont3]

I wonder if we should add some configure stage checks for this and set a flag internally to avoid having to deal with this at runtime. I guess one possible problem could be if you end up executing on a system different from what you used during configure/build time (e.g., binary packages), but maybe that case has better version checking across the binary packages it mixes.

[rhc54]

We do have a prte_check_slurm.m4 that checks for srun in the PATH, but doesn't attempt to execute anything. However, that only happens at build time, and it is entirely possible someone would execute this under a different Slurm version. So I suspect this does have to be a runtime check of some type.

Tim had talked about maybe adding an envar to help steer us, but apparently that didn't happen. So we're kinda stuck with having to directly test srun, I think. We could, I suppose, just try to launch with what we think is the most likely environment and then fallback down the chain until the daemon launch succeeds. After all, srun will immediately fail if it doesn't recognize the option, so we wouldn't have to worry about daemons hanging around - I don't see what doing an initial test buys us.

On a separate note: the Slurm support will eventually add a dependency to it - namely, you will need to have a JSON library available. The API library will be separate, so we can check for it too...but building Slurm support is going to undergo some changes. At the very least, we probably need to advise people to add --with-slurm to the configure line to ensure the build fails if all the dependencies aren't found - otherwise, we will silently not build the Slurm components even if srun is found, which is a new behavior.

Tim has suggested creating new slurm2 components to handle the API library - might help, need to see. Would allow the current components to build if JSON isn't around - but then we are back to the confused cmd line and envars again, so we could well have malfunctioning components. Not sure that's any better.

[hppritcha]

could use sinfo to get version info. not sure that's really any different though.

[rhc54]

Maybe the best answer is to simply defer this until we get the API? Yeah, it causes problems - but I'm not sure that the solution isn't going to be flaky as well. Trying to adapt across multiple Slurm variants feels kinda iffy. Perhaps we just add it to the docs and advise people to move off those Slurm releases, either downwards or (once released) up to 24.11 and above? Of course, moving up implies that OMPI also has to move up to integrate to those changes.

Doesn't feel like there is any really good answer 🤷‍♂️

[rhc54]

Thought about this some more and I believe there really are only three options at this point:

• detect the Slurm version and error out if it is 23.11 or 24.05. Give them a cmd line option to override if they know what they are doing.
• output a warning if it is 23.11 or 24.05 and run. Give them a cmd line option to disable the warning.
• just document that these are bad Slurm versions and recommend people avoid them.

I'm inclined to implement the 3rd option as it is the easiest to do and this hasn't been a total mess - at least, so far. Definitely don't want to try and mess with the srun cmd line to chase Slurm options - that seems a futile exercise.