MLPoisson on AMD GPUs crashes - kernel not starting (?)

[ElloiseFangelLloyd]

Hi all

I have been using the MLPoisson functionality to solve Poisson's equation, and it runs with no issues on CPUs and on GPUs with the CUDA backend. Having migrated to a new system (LUMI), I need to run the same code on the AMD GPUs, and this is crashing during runtime. (MRE attached).

AMReX was compiled and installed (with cmake, which is the preferred approach at LUMI) using the amdclang compiler and the HIP backend, and has been working without problems thus far. I have had no issues setting up MultiFabs and using ParallelFor
and MFIter, however, the MLPoisson instantiation (line 96 in the MRE) fails.

I have attached the MRE, the error occurs in line 96: MLPoisson Poisson_equation({geom}, {grids}, {dmap}, info). This is a single-level solver and the code is almost entirely taken from the ABecLaplacianC example.

It is my understanding (not an expert in reading traces...) that the kernel has not been created and the template instantiation has not worked.

I am usually compiling AMReX code with amdclang for uniformity, however, crayclang was also tried. I contacted support at LUMI, who also tried varying verions of ROCm (6.0 and 6.2) and two AMReX versions (devel and v2024).

Any advice on this issue is massively appreciated! Thanks so much in advance for your time.

Backtrace:

:0:/long_pathname_so_that_rpms_can_package_the_debug_info/src/external/clr/hipamd/src/hip_global.cpp:114
: 99912799288 us: [pid:84989 tid:0x155537ffd700] Cannot find Symbol with name:
_ZN5amrex13launch_globalILi256EZNS_11ParallelForILi256EiZNS_7FabSetTINS_8MultiFabEE6setValEdEUliiiiE_vEENSt9enable_ifIXsr19MaybeDeviceRunnableIT1_EE5valueEvE4typeERKNS_3Gpu10KernelInfoERKNS_3BoxET0_OS7_EUlvE_EEvSH_.intern.3a49fd332233d7b0

#0  0x0000155501f12d2b in raise () from /lib64/libc.so.6
#1  0x0000155501f143e5 in abort () from /lib64/libc.so.6
#2  0x0000155553df71e9 in ?? () from /opt/rocm-6.0.3/lib/libamdhip64.so.6
#3  0x0000155553df748c in ?? () from /opt/rocm-6.0.3/lib/libamdhip64.so.6
#4  0x0000155553db0b4b in ?? () from /opt/rocm-6.0.3/lib/libamdhip64.so.6
#5  0x0000155553f1ac98 in ?? () from /opt/rocm-6.0.3/lib/libamdhip64.so.6
#6  0x0000155553ef4ae2 in ?? () from /opt/rocm-6.0.3/lib/libamdhip64.so.6
#7  0x0000155553eff657 in hipLaunchKernel () from
/opt/rocm-6.0.3/lib/libamdhip64.so.6
#8  0x0000155550793779 in amrex::ParallelFor<256, int,
amrex::FabSetT<amrex::MultiFab>::setVal(double)::{lambda(int, int, int,
int)#1}, 3, void>(amrex::Gpu::KernelInfo const&, amrex::BoxND<3> const&, int,
amrex::FabSetT<amrex::MultiFab>::setVal(double)::{lambda(int, int, int,
int)#1} const&) (box=...,
    ncomp=<optimized out>, f=...) at
/pfs/lustrep1/scratch/project_462000008/evitali/5568/amrex/Src/Base/AMReX_GpuLaunchFunctsG.H:848
#9  amrex::ParallelFor<int,
amrex::FabSetT<amrex::MultiFab>::setVal(double)::{lambda(int, int, int,
int)#1}, 3, void>(amrex::BoxND<3> const&, int,
amrex::FabSetT<amrex::MultiFab>::setVal(double)::{lambda(int, int, int,
int)#1}&&) (box=..., ncomp=<optimized out>, f=...)
    at
/pfs/lustrep1/scratch/project_462000008/evitali/5568/amrex/Src/Base/AMReX_GpuLaunchFunctsG.H:1280
#10 .omp_outlined._debug__ (this=<optimized out>, .global_tid.=<optimized
out>, .bound_tid.=<optimized out>, ncomp=<optimized out>, val=<optimized out>)
at
/pfs/lustrep1/scratch/project_462000008/evitali/5568/amrex/Src/Boundary/AMReX_FabSet.H:263
#11 .omp_outlined.(void) const (.global_tid.=<optimized out>,
.bound_tid.=<optimized out>, this=<optimized out>, this@entry=0xc804a0,
ncomp=<optimized out>, val=<optimized out>) at
/pfs/lustrep1/scratch/project_462000008/evitali/5568/amrex/Src/Boundary/AMReX_FabSet.H:258
#12 0x00001555507a67b4 in amrex::FabSetT<amrex::MultiFab>::setVal
(this=this@entry=0xc804a0, val=nan(0x8000000000000)) at
/pfs/lustrep1/scratch/project_462000008/evitali/5568/amrex/Src/Boundary/AMReX_FabSet.H:258
#13 0x00001555507a6650 in amrex::BndryRegisterT<amrex::MultiFab>::define
(this=0xc804a0, _face=..., _typ=..., _in_rad=<optimized out>,
_out_rad=<optimized out>, _extent_rad=<optimized out>, _ncomp=1, dmap=...) at
/pfs/lustrep1/scratch/project_462000008/evitali/5568/amrex/Src/Boundary/AMReX_BndryRegister.H:188
#14 0x00001555508e8e84 in amrex::BndryRegisterT<amrex::MultiFab>::define
(this=0xc804a0, grids_=..., dmap=..., in_rad=in_rad@entry=1,
out_rad=out_rad@entry=0, extent_rad=extent_rad@entry=0, ncomp=<optimized out>)
at
/pfs/lustrep1/scratch/project_462000008/evitali/5568/amrex/Src/Boundary/AMReX_BndryRegister.H:155
#15 0x00001555508e7f1f in amrex::MLCellLinOpT<amrex::MultiFab>::defineAuxData
(this=this@entry=0x7fffffff3588) at
/pfs/lustrep1/scratch/project_462000008/evitali/5568/amrex/Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H:384
#16 0x00001555508f8256 in amrex::MLCellLinOpT<amrex::MultiFab>::define
(this=this@entry=0x7fffffff3588, a_geom=..., a_grids=..., a_dmap=...,
a_info=..., a_factory=...) at
/pfs/lustrep1/scratch/project_462000008/evitali/5568/amrex/Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H:362
#17 amrex::MLCellABecLapT<amrex::MultiFab>::define
(this=this@entry=0x7fffffff3588, a_geom=..., a_grids=..., a_dmap=...,
a_info=..., a_factory=...) at
/pfs/lustrep1/scratch/project_462000008/evitali/5568/amrex/Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H:100
#18 0x000015555090aa02 in amrex::MLPoissonT<amrex::MultiFab>::define
(this=0x7fffffff3588, a_geom=..., a_grids=..., a_dmap=..., a_info=...,
a_factory=...) at
/pfs/lustrep1/scratch/project_462000008/evitali/5568/amrex/Src/LinearSolvers/MLMG/AMReX_MLPoisson.H:119
#19 amrex::MLPoissonT<amrex::MultiFab>::MLPoissonT (this=0x7fffffff3588,
a_geom=..., a_grids=..., a_dmap=..., a_info=..., a_factory=...) at
/pfs/lustrep1/scratch/project_462000008/evitali/5568/amrex/Src/LinearSolvers/MLMG/AMReX_MLPoisson.H:96
#20 0x000000000025b00f in makePoisson () at
/pfs/lustrep1/scratch/project_462000008/evitali/5843_amrexagain/main.cpp:96
#21 0x000000000025a929 in main (argc=1, argv=0x7fffffff4508) at
/pfs/lustrep1/scratch/project_462000008/evitali/5843_amrexagain/main.cpp:27

amrex_poisson.zip

Edit: put line 96 into code format.

[WeiqunZhang]

Could you try hipcc? You might also want to take a look at https://warpx.readthedocs.io/en/latest/install/hpc/lumi.html and #2311.

[WeiqunZhang]

Could you run the test at https://github.com/AMReX-Codes/amrex/tree/development/Tests/LinearSolvers/ABecLaplacian_C? You can try to compile it with make USE_HIP=TRUE XTRALIBS=${PE_MPICH_GTL_DIR_amd_gfx90a} -lmpi_gtl_hsa. The XTRALIBS is for gpu-aware mpi. You can also try without that.

[ElloiseFangelLloyd]

Edit:
Compiling with export MPICH_GPU_SUPPORT_ENABLED=1 as recommended in the WarpX code and by LUMI required both -offload_arch=gfx90a and explicitly calling module load craype-accel-amd-gfx90a in order to properly link GTL libraries. The error shared object initialization failed !!! described below persists.

---

back.txt
Compiled both with and without the XTRALIBS, results in this error either way (from branch development as in the link).

./main3d.hip.x86-trento.MPI.HIP.ex
Initializing AMReX (25.02-17-g28a1e19b3b27)...
MPI initialized with 1 MPI processes
MPI initialized with thread support level 0
Initializing HIP...
HIP initialized with 1 device.
amrex::Abort::0::GPU last error detected in file ../../../Src/Base/AMReX_GpuLaunchFunctsG.H line 813: shared object initialization failed !!!
SIGABRT
See Backtrace.0 file for details
MPICH ERROR [Rank 0] [job id ] [Thu Feb 20 11:56:27 2025] [nid005026] - Abort(6) (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 6) - process 0

[WeiqunZhang]

Sorry I forgot about one has to set AMD arch. Could you try to set export AMREX_AMD_ARCH=gfx90a as environment variable instead of passing -offload_arch to make?

[ElloiseFangelLloyd]

I have tested this both in the initial MRE and in the ABecLaplacian_C test, it did not make a difference in either case. It is worth noting that I absolutely still had to pass -offload_arch, or else I got errors with GTL library not linked. Is it possible they are arguing somehow?
My colleague is running on the same systems, and gets the kernel launch errors which I got from my MRE from the function AmrCoreAdv::MakeNewGrids (Real time) - is it possible that there is something problematic with the grids somewhere?

[WeiqunZhang]

I don't have access to LUMI. So it's hard for me to know what's going on. I think the issue is the environment, not the source code, because it runs on other AMD machines like Frontier just fine. I will be in office in about half hour. If you want to have a zoom meeting to share your screen with me, send me an email, [email protected].

[ElloiseFangelLloyd]

The fix for this was to recompile AMReX without OpenMP, so without USE_OMP=TRUE. Then I can run both the tests and my MRE without problems.
Why this fixed the problem is somewhat beyond me, but this worked for me, in any case. :)