Error compiling for AMD ROCM

[LutzRohlfing]

Issue:

I tried compiling AMR-Wind for an AMD W7900 Pro GPU and got the following issue:

 50      Use AMD's 'clang++'/'amdclang++' compiler, or on Cray the CC compiler
 51      wrapper instead.
 52    Call Stack (most recent call first):
 53      submods/amrex/Src/CMakeLists.txt:40 (include)
 54    
 55    
 56    CMake Error at submods/amrex/Tools/CMake/AMReXParallelBackends.cmake:311 (find_package):
 57      Could not find a package configuration file provided by "hiprand" with any
 58      of the following names:
 59    
 60        hiprandConfig.cmake
 61        hiprand-config.cmake
 62  

System: Ubuntu 24.04
AMR-Wind Version: 3.8.1

Error.txt

I attached the full spack build log in the error.txt.

[jrood-nrel]

Ultimately the Spack compiler configuration for your machine is incorrect. I believe it's not correct to use the hipcc compilers. Here is an example configuration for an AMD machine we use: https://github.com/Exawind/exawind-manager/blob/b62157eb5c601edd7336d025d3f0959791f19737/configs/frontier/packages.yaml#L8-L10

[LutzRohlfing]

Thanks for the feedback, i tried using a different compiler, but got the same issue. Can you take a look at my spack.yaml?

spack.yaml

[jrood-nrel]

Your configuration is missing all the AMD packages. It's best to list the AMD packages explicitly: https://github.com/Exawind/exawind-manager/blob/8b1e095c8014a5c04f09126bfbd5fb51365ab456/configs/frontier/packages.yaml#L37-L126

[michaelasprague]

@LutzRohlfing Were you successful here? May we close this issue?

[LutzRohlfing]

Yes it works now thanks!

[LutzRohlfing]

Ok, i am sorry, but i do have to reopen this issue. I managed to compile and AMR-Wind runs, but when doing some testing i realised i have really bad convergence in my solution. I ran the same case on my machine with AMD and ROCM and on another machine with NVIDIA and CUDA. On the machine with a NVIDIA GPU it works fine. I attached both logs of the run here. I am aware that it is not the same Version of AMR-Wind. The CUDA AMR-Wind Version is 3.6.0 and the ROCM AMR-Wind Version should be 3.8.1 (even though it does not say that in the header?). Is this an issue between versions and i should revert to a previous version of AMR-Wind or did i mess something up during compiling?

Log_Nvidia.txt
Log_AMD.txt

[jrood-nrel]

You should start by using the same version between machines, and you would probably want it to be the latest version.