Running GCHP in a container, but still editing the source code?

[sjacker2]

Your name

Summer Acker

Your affiliation

University of Illinois Urbana-Champaign

Please provide a clear and concise description of your question or discussion topic.

Hi! My research group (Hannah Horowitz's group) tried to get GCHP running on our campus cluster a couple of years ago, but were having some troubles with spack. See the issue here for reference:

#320

Given that it's been a couple of years, I decided to try again and ran into similar issues, even with an updated spack. I was able to get GCHP to build and compile this time, but once I ran it, it would hang or fail due to incompatibility between the spack OpenMPI and the campus cluster slurm. I then tried creating a new spack environment which used the OpenMPI from the campus cluster and building other necessary modules (i.e. netcdf) off of that in the hopes of it being more compatible. After much other debugging, I'm now at a point where it keeps seg faulting, saying "Program received signal SIGSEGV: Segmentation fault - invalid memory reference." I think there are still some problems with the campus cluster and my environment not agreeing on resource allocation or something similar. We're starting to come to the conclusion that GCHP just isn't compatible with our compute environment still.

I was wondering if anyone had any suggestions for getting GCHP running on a different campus cluster (outside of Harvard or WashU). I know that using singularity is an option, but I heard that it might not be possible to edit the source code of the model while using a container, which my study may require. Has anyone ever had success with editing the source code within a container? Also, is there a specific image that has proved to work well for others (even if not editing source code)?

Any suggestions and guidance would be extremely helpful.

Thank you so much for your advice!

[sjacker2]

Adding an update:

I am using the 14.2.0 GCHP image with singularity now. The version of OpenMPI on our cluster is openmpi@5.0.1%gcc@13.3.0, which is what I am loading into the container with gchp.env. Whenever I try to run GCHP, I get the following error:

The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM support. This usually happens
when OMPI was not configured --with-slurm and we weren't able
to discover a SLURM installation in the usual places.

Please configure as appropriate and try again.

*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
[ccc0269.campuscluster.illinois.edu:1058139] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[ccc0269.campuscluster.illinois.edu:1058131] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

Here is the full log:

current_error.txt

Our campus cluster requires slurm since that is our job scheduler, but it looks like the image OpenMPI may have been built without slurm capabilities. This error could also be arising from the fact that the image has OpenMPI version 3.0.5 and I'm using 5.0.1. I'm not really sure how to fix this incompatibility, so any guidance would be appreciated!

Thank you so much!

[yantosca]

Thanks @sjacker2 for the update. I believe you would need OpenMPI 4 for the current GCHP / MAPL combination. I am pretty sure OpenMPI 5 won't work yet (at least until MAPL3). Tagging @lizziel who may know more.

For reference, these are the packages in our GCHP environment at Harvard.

Currently Loaded Modules:
  1) gmp/6.2.1    5) openmpi/4.1.4   9) netcdf-c/4.9.2
  2) mpfr/4.2.0   6) zlib/1.2.13    10) netcdf-fortran/4.6.0
  3) mpc/1.3.1    7) szip/2.1.1     11) flex/2.6.4
  4) gcc/12.2.0   8) hdf5/1.14.0    12) cmake/3.25.2

[sjacker2]

Thank you so much @yantosca ! I'll try mimicking the same environment and see if I can get that working.

The problem is that our campus cluster only has OpenMPI 5 and spack has problems with adding slurm integration (which is needed on our cluster) when I try to build a spack environment with a spack OpenMPI module. I'll still give it another go with these specific versions and see if that helps! Thank you!

[sjacker2]

Hi Bob @yantosca! I think I've successfully replicated your GCHP environment by manually building all of the packages. I am in the process of compiling and realized I don't have ESMF built yet. Do you know the specific version of ESMF that is used in your environment? Thank you so much!

[yantosca]

Hello Summer! For GEOS-Chem 14.7.0, you need at least ESMF 8.6.1. For earlier versions we used ESMF 8.4.2.

[yantosca]

@sjacker: although I think you may be able to use ESMF 8.6.1 with e.g. GEOS-Chem 14.6.3 just fine. Just that for the version of MAPL in GC 14.7.0 you need at least ESMF 8.6.1.

[sjacker2]

Got it. Thank you so much!

[lizziel]

I ran into this same issue on AWS this week when installing OpenMPI4 via spack:

The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM support. This usually happens
when OMPI was not configured --with-slurm and we weren't able
to discover a SLURM installation in the usual places.

I did not build it with slurm support and assume that is the problem. @sjacker2, could you share the versions and settings you ended up using successfully? I am curious if we are fixing it the same way.

[sjacker2]

Hi @lizziel and @yantosca! I ended up building everything without spack using the versions that Bob sent above that the Harvard GCHP environment has, but I have yet to have a successful run. Here is my OpenMPI configuration that I am currently using (with slurm turned on), but I think it's currently wrong (see below):

Open MPI configuration:

Version: 4.1.4
Build MPI C bindings: yes
Build MPI C++ bindings (deprecated): no
Build MPI Fortran bindings: mpif.h, use mpi, use mpi_f08
MPI Build Java bindings (experimental): no
Build Open SHMEM support: yes
Debug build: no
Platform file: (none)

Miscellaneous

CUDA support: no
HWLOC support: internal
Libevent support: external
Open UCC: no
PMIx support: Internal

Transports

Cisco usNIC: no
Cray uGNI (Gemini/Aries): no
Intel Omnipath (PSM2): no
Intel TrueScale (PSM): no
Mellanox MXM: no
Open UCX: yes
OpenFabrics OFI Libfabric: no
OpenFabrics Verbs: yes
Portals4: no
Shared memory/copy in+copy out: yes
Shared memory/Linux CMA: yes
Shared memory/Linux KNEM: no
Shared memory/XPMEM: yes
TCP: yes

Resource Managers

Cray Alps: no
Grid Engine: no
LSF: no
Moab: no
Slurm: yes
ssh/rsh: yes
Torque: no

OMPIO File Systems

DDN Infinite Memory Engine: no
Generic Unix FS: yes
IBM Spectrum Scale/GPFS: no
Lustre: no
PVFS2/OrangeFS: no

I got GCHP to build successfully without errors, but ran into a couple off issues when running it. The first was that it would hang when I used srun -n 48 -N 2 --mpi=PMIx ./gchp. I changed this to mpi=none in the hopes that it would just automatically assign what I have available to me but then it got this error:

[ccc0269.campuscluster.illinois.edu:4147881] OPAL ERROR: Unreachable in file pmix3x_client.c at line 112
--------------------------------------------------------------------------
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:

  version 16.05 or later: you can use SLURM's PMIx support. This
  requires that you configure and build SLURM --with-pmix.

  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
  install PMI-2. You must then build Open MPI using --with-pmi pointing
  to the SLURM PMI library location.

Please configure as appropriate and try again.

I think this means my OpenMPI configuration is incorrect. I'm going to try switching the "PMIx support: Internal" to Slurm's PMI2 and see if that works.