Using ESMPy within srun fails with launcher not compatible with PMI1 client

[schlunma]

Requirements

• Reviewed ESMPy Documentation
• Searched GitHub Discussions

Affiliation(s)

DLR

ESMF Version

v8.8.1

Issue

Using ESMPy within Slurm's srun command, i.e.

$ srun --partition=interactive -- python -c "import esmpy; esmpy.Manager()"

gives the following error:

[unset]: launcher not compatible with PMI1 client
Abort(538569231): Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71): MPI_Init_thread(argc=(nil), argv=(nil), required=3, provided=0x7fffd5b31d88) failed
MPII_Init_thread(203)...:
MPIR_pmi_init(150)......:
pmi1_init(14)...........: PMI_Init returned -1

The following work fine:

$ python -c "import esmpy; esmpy.Manager()"
$ srun --mpi=cray_shasta --partition=interactive -- python -c "import esmpy; esmpy.Manager()"

The error does not appear with ESMPy v8.4.2, but shows in v8.6.1.

I am wondering if this is a bug or a feature. Any help would on this would be appreciated! Thanks!

System information

$ mamba list esm

  Name              Version  Build                   Channel
──────────────────────────────────────────────────────────────────
  esmf              8.8.1    mpi_mpich_h7cf99a1_100  conda-forge
  esmpy             8.8.1    pyhecae5ae_0            conda-forge
  esmvalcore        2.12.0   pyhd8ed1ab_1            conda-forge
  iris-esmf-regrid  0.12.1   pyhd8ed1ab_0            conda-forge
  xesmf             0.8.10   pyhd8ed1ab_0            conda-forge

$ srun --version
slurm 24.05.1-BullSequana.1.1

$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.9 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.9"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.9 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.9
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.9"

Autotag

@billsacks

[billsacks]

The ESMF team's sense is that this is a feature rather than a bug. Our suspicion is that the conda-forge ESMF versions you're using were built with the cray mpich MPI library, which may then require --mpi=cray_shasta for compatibility (similar to what's mentioned in https://slurm.schedmd.com/mpi_guide.html#hpe_cray_pmi). Perhaps there was a difference in the MPI library used in the ESMF 8.4.2 build from conda-forge? Or there may have been a difference between ESMF 8.4 and ESMF 8.6 in how ESMF detects / handles the cray MPI library in the build. We have also seen somewhat similar errors when the MPI library used in the mpi4py dependency differs from the system mpi library.

[schlunma]

Great, thanks for your answer @billsacks, that sounds very reasonable! The ESMF build I use is indeed mpi_mpich_h10ba997_101. A colleague of mine tested this with nompi_hc4cf7a7_1 without any problems.