Neither `--launch-agent` nor `--mca plm rsh` is respected if OpenMPI thinks the host in question is `localhost`

[timber-schroeder]

Background information

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

v4.1.6

---

Details of the problem

I am using mpirun on a cluster with several nodes in it, with ssh/rsh as the expected transport for job starts.

I can't seem to get mpirun to respect the --launch-agent nor --mca plm rsh flags if it thinks that the host target in question is the local machine, even if it is not due to namespace isolation. This logic appears to be hardcoded based on hostname matching.

The command I am trying to make work, with the actual hostnames and commands replaced:

host1 $ mpirun --mca plm rsh --mca plm_rsh_assume_same_shell false --noprefix --host 'host1,host2,host3' --npernode 1 --launch-agent 'enroot start ${image_sqfs} orted' $command
<mpirun>
<!snip>
<stderr>--------------------------------------------------------------------------&#010;</stderr>
<stderr>mpirun was unable to find the specified executable file, and therefore&#010;</stderr>
<stderr>did not launch the job.  This error was first reported for process&#010;</stderr>
<stderr>rank 0; it may have occurred for other processes as well.&#010;</stderr>
<stderr>&#010;</stderr>
<stderr>NOTE: A common cause for this error is misspelling a mpirun command&#010;</stderr>
<stderr>      line parameter option (remember that mpirun interprets the first&#010;</stderr>
<stderr>      unrecognized command line token as the executable).&#010;</stderr>
<stderr>&#010;</stderr>
<stderr>Node:       $host1&#010;</stderr>
<stderr>Executable: $command&#010;</stderr>
<stderr>--------------------------------------------------------------------------&#010;</stderr>
<!snip>
</mpirun>

$command is an executable that the process running mpirun has no access to, which resides in ${image_sqfs}. This is intended. What is not intended is for mpirun to ignore the --launch-agent and --mca plm rsh commands, which run orted in the correct namespace with the required executables.

Running mpirun on host4 instead, which is not part of the --host argument list, works without issue.
Setting --mca plm_rsh_no_tree_spawn true changes the behavior and causes mpirun to hang and never attempt to launch any process at all.

I am effectively using enroot to launch orted in a mount namespace that contains the executables that I want mpirun to run. This works as expected, unless the parent mpirun call is also on one of the hosts specified in --hosts, at which point mpirun appears to attempt to directly fork/exec the executable in question, which fails as mpirun has no access to the same file that orted does.

In my opinion, the expected behavior here should be for the launch agent to always be used for job execution if specified, regardless of whether mpirun (incorrectly) thinks it is on localhost or not. Even setting --mca plm rsh appears to be completely ignored in this case as openmpi does not actually attempt to use ssh/rsh.

This appears to have come up a few times before, in #10151 and #7120.

[rhc54]

In my opinion, the expected behavior here should be for the launch agent to always be used for job execution if specified, regardless of whether mpirun (incorrectly) thinks it is on localhost or not.

I'm afraid that isn't the behavior the system employs. Local procs are fork/exec'd under mpirun itself for a variety of reasons as explained in the cited references. There are no plans to change that behavior.

[timber-schroeder]

Is the assumption that this is being done based on hostname alone accurate at least? It is unfortunate that such assumptions exist in openmpi, but I am not opposed to building a patch for local usage to avoid this (again imo buggy regardless of historic reasons) behavior, but I am not familiar with the source layout at the moment.

Other workarounds may exist such as putting openmpi into a bogus UTS namespace to force it to always run remote commands regardless of the target hosts.

[rhc54]

It isn't an "assumption" - it is a "requirement" of the design. As was stated in the citations, you can play around with it to find a possible workaround for launch, but other parts of the system will likely fail. Still, you are welcome to try 🤷‍♂ .

What I do is simply setup a set of containers on my laptop. This works fine and simulates multiple nodes on a single machine. I can then enter a container and launch mpirun as if it were a multi-node system. Perhaps you could see if that meets your needs? If you don't want the procs under mpirun for some reason, then exclude that node from the hostfile (or dash-host option) and just add a container to just hold mpirun.

[timber-schroeder]

Unfortunately the issue here isn't the lack of a test environment but rather the lack of being able to run the commands in question reliably on a head node in various clusters. I am running mpirun directly on bare metal compute hosts.

Were it just a development environment I would be doing as you listed, but do note that containers are namespaces like the UTS one I mentioned above. I tried a naive UTS mount with a different hostname but unfortunately mpirun still tried to run a local process, so there must be some attempt at name resolution going on.

Closing since this was clearly communicated as a wontfix due to design decisions. Should this ever come up in the future for major releases, the localhost assumption purely for job execution going away would be nice, in conjunction with --launch-agent and --mca plm rsh not being ignored. This would actually be useful in HPC clusters, quite a few of which often use enroot as I listed above. The only reason it isn't noticed nearly as often is because the container image usually has the target binaries in addition to the mpi runtime.