Cray MPICH/ROCm compatibility for GPU-aware MPI (#4729)

## Summary

Implements a check in OLCF makefile to compare Cray MPICH supported ROCm
version with that in current environment. If the major versions differ,
do not link library for GPU-aware MPI as this will cause an error at
runtime.

## Additional background

* If the ROCm versions are incompatible, a user will see an error like
the following at runtime: `error while loading shared libraries:
libamdhip64.so.6: cannot open shared object file: No such file or
directory`.
* Supported major ROCm version for Cray MPICH is determined by parsing
`ldd` output for `libmpi_gtl_hsa.so`.
* The user's ROCm version is determined by parsing the output of
`hipconfig --version`.
* The user must also make sure the `craype-accel-amd-gfx90a` module (on
Frontier) is not loaded, as this will automatically try to link GTL.
* If running with a ROCm version that is not supported by the Cray MPICH
version, `CRAY_LD_LIBRARY_PATH` must also be prepended to
`LD_LIBRARY_PATH` and disable GPU-aware MPI with `export
MPICH_GPU_SUPPORT_ENABLED=0`.

## Checklist

The proposed changes:
- [x] fix a bug or incorrect behavior in AMReX
- [ ] add new capabilities to AMReX
- [ ] changes answers in the test suite to more than roundoff level
- [ ] are likely to significantly affect the results of downstream AMReX
users
- [ ] include documentation in the code and/or rst files, if appropriate

Co-authored-by: Harris, Austin <[email protected]>

Author: jaharris87

Tools/GNUMake/sites/Make.olcf

@@ -43,7 +43,11 @@ ifeq ($(which_computer),frontier)
     endif
     # for gpu aware mpi
     ifeq ($(USE_HIP),TRUE)
-      LIBRARIES += $(PE_MPICH_GTL_DIR_amd_gfx90a) -lmpi_gtl_hsa
+      ROCM_MAJOR_VERSION = $(shell hipconfig --version | cut -d. -f1)
+      MPICH_ROCM_VERSION = $(shell ldd ${CRAY_MPICH_ROOTDIR}/gtl/lib/libmpi_gtl_hsa.so | grep libamdhip64.so | cut -d" " -f1 | cut -d. -f3 )
+      ifeq ($(ROCM_MAJOR_VERSION),$(MPICH_ROCM_VERSION))
+        LIBRARIES += $(PE_MPICH_GTL_DIR_amd_gfx90a) -lmpi_gtl_hsa
+      endif
     endif
   endif
 endif