mpiexec 4.1.3, how to avoid transfer PMIX_INSTALL_PREFIX to the child process of orted

[gzm55]

Background information

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

v4.1.4

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

installed via conda package, make the xcopy to the cluster.

Please describe the system on which you are running

• Operating system/version: CentOS 7
• Computer hardware: x64_64
• Network type: localhost

---

Details of the problem

In our cluster, the mpi prefix are different on different nodes. For example on master node, the opal_prefix is /path/master/prefix, while on the nodes, the opal_prefix is `/path/node-/prefix

The job is run like this

export OPAL_PREFIX=/path/master/prefix
unset PMIX_INSTALL_PREFIX
shell$ mpiexec -n 1 --mca plm_rsh_agent /usr/bin/echo sh -c 'echo $PMIX_INSTALL_PREFIX'

and run in another shell the orted command:

export OPAL_PREFIX=/path/node-<id>/prefix
export PMIX_INSTALL_PREFIX=$OPAL_PREFIX
orted -mca ess "env" ...

Then the result of master shell is always /path/master/prefix, not the nodes specific path, even if I exported a correct value for PMIX_INSTALL_PREFIX before orted started.

How to avoid transfer PMIX_INSTALL_PREFIX from mpiexec to the child process of orted?

[rhc54]

Afraid I don't really understand what you are doing. mpiexec starts its own orted processes - you cannot run orted by hand and expect it to do anything. PMIX_INSTALL_PREFIX is an internal value that simply records the location of the original installation - you use PMIX_PREFIX to relocate it.

Are you using the PMIx embedded in OMPI? If so, then just set OPAL_PREFIX to relocate things. However, if you truly have put OMPI in a different place on every node - well, then you are pretty much out-of-luck. You could perhaps make it work, but it is hard and not something one should do for production (maybe attempted as an experiment). I'd suggest instead spending some time to cleanup the cluster into a more sane configuration.

[gzm55]

Afraid I don't really understand what you are doing. mpiexec starts its own orted processes - you cannot run orted by hand and expect it to do anything. PMIX_INSTALL_PREFIX is an internal value that simply records the location of the original installation - you use PMIX_PREFIX to relocate it.

Are you using the PMIx embedded in OMPI? If so, then just set OPAL_PREFIX to relocate things. However, if you truly have put OMPI in a different place on every node - well, then you are pretty much out-of-luck. You could perhaps make it work, but it is hard and not something one should do for production (maybe attempted as an experiment). I'd suggest instead spending some time to cleanup the cluster into a more sane configuration.

@rhc54 our situation is that before the application is launched by the mpiexec, the ompi folder is copied to a sandbox temporary path on every remote nodes, and orted could be found from PATH. our app is linked to ompi and MPI_init seems to search pmix library via PMIX_INSTALL_PREFIX environment. so i need a correct PMIX_INSTALL_PREFIX setting on every slave node.

[ggouaillardet]

you can use your own agents to modify the environment:

mpirun --mca orte_launch_agent launch_agent.sh --mca orte_fork_agent fork_agent.sh a.out

instead of remote executing the orted daemon, it will remote execute launch_agent.sh,
and instead of fork&exec a.out, it will fork&exec fork_agent.sh a.out

Note building Open MPI with --disable-dlopen might mitigate these issues.
An other idea is after the Open MPI folder is copied to the sanbox, you can chrpath the libraries so they find their dependencies in the sandbox.

[gzm55]

mpirun --mca orte_launch_agent launch_agent.sh --mca orte_fork_agent fork_agent.sh a.out

thanks, i will try both methods.

[rhc54]

Like I said, PMIX_INSTALL_PREFIX does nothing - you need to set PMIX_PREFIX, which is set automatically by setting OPAL_PREFIX. You could also simply put the sandbox in the same place on every node, making the problem infinitely simpler. Further simplification is to build static, as @ggouaillardet pointed out, and completely eliminate the "finding the library" problem.