Possible Regression in OpenMPI v5.x – Missing rendezvous File since Feb 21

[ericch1]

Hi,

We have been testing the ompi/v5.x branch nightly against our code.

Since February 21, when testing against the following commit:

commit c1d3071
Merge: beee369 7f4ddab
Author: Tommy Janjusic janjust@users.noreply.github.com
Date: Fri Feb 21 09:23:45 2025 -0600

Merge pull request #13100 from janjust/v5.0.x

we have consistently encountered the following error message in the output of all jobs:

--------------------------------------------------------------------------
There was an error when attempting to access the specified server
rendezvous file:

  Filename:  /tmp/TV_2025.02.26.18h16m08s.QBF/pmix.sys.dms3
  Error:     could not be found

Please correct the error and try again.
--------------------------------------------------------------------------

For reference, here are the logs for the configuration and build:

• http://www.giref.ulaval.ca/~cmpgiref/ompi_5.0.x/2025.02.21.18h16m08s_ompi_config.log
• http://www.giref.ulaval.ca/~cmpgiref/ompi_5.0.x/2025.02.21.18h16m08s_ompi_make.log
• http://www.giref.ulaval.ca/~cmpgiref/ompi_5.0.x/2025.02.21.18h16m08s_ompi_make_install.log
• http://www.giref.ulaval.ca/~cmpgiref/ompi_5.0.x/2025.02.21.18h16m08s_ompi_ompi_info_all.txt

This issue was not present before this commit, and everything works fine with the ompi/v4.1.x branch.

We would appreciate any insights you might have.

Thanks,
Eric

[rhc54]

Can you give some example cmd line that generated that error? It isn't something one should ordinarily encounter (a tool might, but not an application).

[ericch1]

Hi @rhc54,

I forgot to mention an important information:

it occurs on only if the code it is launched for 1 process. On parallel runs, with mpiexec -n 2 or more, this message is not there...

May it be also related to the fact that we configure with debug activated?

./configure --prefix=... --enable-debug --enable-picky --with-cma

Thanks,

Eric

[rhc54]

it occurs on only if the code it is launched for 1 process.

Do you perhaps mean "run as a singleton"? Or do you mean "mpirun -n 1 ..."?

May it be also related to the fact that we configure with debug activated?

Not really - you'd have to turn some verbosity on to see anything. Configure option just enables some of those in case you do turn on the verbosity.

[ericch1]

Do you perhaps mean "run as a singleton"? Or do you mean "mpirun -n 1 ..."?

I mean "run as a singleton".

I didn't test "mpirun -n 1 ...", I'll do it today if you want..

[ericch1]

Ok, if I run a simple "Hello World" example as a "singleton" the error is there, but with "mpirun -n 1" it vanishes... :

mpicc -o hw hw.cc 
bash-5.2$ ./hw
--------------------------------------------------------------------------
There was an error when attempting to access the specified server
rendezvous file:

  Filename:  /tmp/pmix.sys.dms3
  Error:     could not be found

Please correct the error and try again.
--------------------------------------------------------------------------
Hello world from processor dms3, rank 0 out of 1 processors

bash-5.2$ mpiexec -n 1 ./hw
Hello world from processor dms3, rank 0 out of 1 processors

[ericch1]

Some important variables in our environment:

env |grep "^OMPI"
OMPI_MCA_btl_vader_single_copy_mechanism=none
OMPI_MCA_fs_ufs_lock_algorithm=1
OMPI_MCA_hwloc_base_binding_policy=none
OMPI_MCA_mpi_yield_when_idle=1
OMPI_MCA_btl_base_warn_component_unused=0

[rhc54]

Okay, it's just a message that is uncovered when running as a singleton. Easy enough to fix. For now, it won't harm anything - i.e., the singleton will run just fine.

[ericch1]

Okay, it's just a message that is uncovered when running as a singleton. Easy enough to fix.

Okay, good news then.

For now, it won't harm anything - i.e., the singleton will run just fine.

I confirm: all tests results are ok, except for the ones for which we also verify the stdout/stderr... which made us discover this "regression"...

Thanks for your insights!