Compilation on lumi

[pittlerf]

Here we want to fix the problem occuring during the compilation of PLEGMA on lumi using the branch hip_compile_lumi.

[pittlerf]

We compiled using rocm.5.2 and rocm.5.3 as well, in both cases having the same issue

[pittlerf]

i compile with the following modules:
`module purge
module load CrayEnv

module load buildtools/22.08

module load PrgEnv-gnu
craype-accel-amd-gfx90a

export CMAKE_PREFIX_PATH=/appl/lumi/SW/CrayEnv/EB/buildtools/22.08
module use /pfs/lustrep2/projappl/project_462000125/samantao-public/mymodules
module load rocm/5.2.3
module load cray-hdf5-parallel/1.12.1.5
`

[pittlerf]

The compilation script that I use:
source ./load_modules.sh cmake -DCMAKE_BUILD_TYPE="DEVEL" \ -DCMAKE_C_COMPILER=$(which cc) \ -DCMAKE_CXX_COMPILER=$(which hipcc) \ -DCMAKE_HIP_COMPILER=$(which clang++) \ -DPLEGMA_BUILD_SHAREDLIB=ON \ -DPLEGMA_TARGET_TYPE=HIP \ -DPLEGMA_NUCLEON_3PF_FIX_SINK=ON \ -DPLEGMA_TARGET=HIP \ -DPLEGMA_LIGHT_BARYONS=ON \ -DPLEGMA_UDSC_BARYONS=OFF \ -PLEGMA_OPENBLASHOME=/users/pittlerf/code/openBLAS/OpenBLAS/build \ -PLEGMA_OPENBLAS_LIB=/users/pittlerf/code/openBLAS/OpenBLAS/build/lib/libopenblas.a \ -PLEGMA_OPENBLAS_LIBDIR=/users/pittlerf/code/openBLAS/OpenBLAS/build/lib \ -DPLEGMA_SCATTERING_CONTRACTIONS=ON \ -DMULTI_GPU=ON \ -DPLEGMA_TEXTURE=OFF \ -DQUDA_HOME=/users/pittlerf/code/quda/build \ -DQUDA_SRC=/users/pittlerf/code/quda \ -DPLEGMA_OPENBLAS=ON \ -DPLEGMA_LIMEHOME=/users/pittlerf/code/lime/build \ -DHDF5_LIBRARIES=${HDF5_DIR}/lib \ -DHDF5_INCLUDE_DIRS=${HDF5_DIR}/include \ -DPLEGMA_HDF5HOME=$HDF5_DIR \ -DGPU_ARCH=gfx90a \ -DMPI_HOME_MANUAL=/opt/cray/pe/mpich/8.1.18/ofi/gnu/9.1 ../

[pittlerf]

The test executable, that I am using is called Plaquette
and here is the command I use for the creating the executable:
/pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2.3/hip/bin/hipcc --amdgpu-target=gfx90a -fopenmp CMakeFiles/Plaquette.dir/Plaquette.cpp.o -o Plaquette -Wl,-rpath,/users/pittlerf/code/PLEGMA/build/lib:/users/pittlerf/code/quda/build/lib:/opt/cray/pe/hdf5-parallel/1.12.1.5/gnu/9.1/lib:/opt/cray/pe/mpich/8.1.18/ofi/gnu/9.1/lib ../lib/libplegma.so ../utils/libPLEGMAutils.a /users/pittlerf/code/quda/build/lib/libquda.so /opt/cray/pe/hdf5-parallel/1.12.1.5/gnu/9.1/lib/libhdf5_parallel.so.200 /opt/cray/pe/mpich/8.1.18/ofi/gnu/9.1/lib/libmpi.so /pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2.3/hip/lib/libamdhip64.so /opt/cray/pe/mpich/8.1.18/ofi/gnu/9.1/lib/libmpi.so /users/pittlerf/code/lime/build/lib/liblime.a /pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2.3/llvm/lib/libgomp.so /users/pittlerf/code/openBLAS/OpenBLAS/build/lib/libopenblas.a /pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2-109-sles/lib/libhiprand.so.1.1.50203 /pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2-109-sles/lib/librocrand.so.1.1.50203 --offload-arch=gfx90a /pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2-109-sles/lib/libhipblas.so.0.1.50203 /pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2-109-sles/lib/librocblas.so.0.1.50203 /pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2.3/llvm/lib/clang/14.0.0/lib/linux/libclang_rt.builtins-x86_64.a /pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2.3/hip/lib/libamdhip64.so.5.2.50203 /pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2-109-sles/lib/libhipfft.so /pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2.3/hip/lib/libamdhip64.so /users/pittlerf/code/lime/build/lib/liblime.a /pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2.3/llvm/lib/libgomp.so /users/pittlerf/code/openBLAS/OpenBLAS/build/lib/libopenblas.a /pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2.3/rccl/lib/librccl.so

[pittlerf]

I tried to port also on our benchmark code benchmark.cpp to lumi. I am using #include "roctx.h" . However compilation with clang fails due to the following error
/users/pittlerf/code/PLEGMA/plegma/benchmark.cpp:115:69: error: call to implicitly-deleted copy constructor of 'plegma::PLEGMA_Gauge<double>' (&PLEGMA_Gauge<double>::APEsmearing), "APEsmearing (1 iter)", gauge_a, 1, 0.5, 3);
The same code compiles on an nvidia machine. Do you have an idea, how to avoid this?

[pittlerf]

I tried to include some hip code before we allocate the __const__ in PLEGMA.cu in order to check whether that works:
I put the following and this still works.
int *tmpquda=(int *)device_malloc(1 * sizeof(int)); device_free(tmpquda); hipMalloc(&tmpquda, sizeof(int)); hipMemset(&tmpquda, 1, sizeof(int)); hipFree(&tmpquda);

[pittlerf]

I also checked what happens when I try to allocate __constant__ , and I got the same error as before,

This is the chunk of code that I was including:
`constant int hipvariable[1];

void plegma::PLEGMA_init(int localL[4], int nProcs[4], int verbosity){

PLEGMA_printf("start PELGMA Init\n");
fflush(stdout);
HGC_hold_exit = false;
HGC_options->checkErrors();

const int teee=10022;
void **ptr = (void **) &hipvariable;
hipError_t temporary=hipMemcpyToSymbol(*ptr,&teee,sizeof(int));
if (temporary==hipSuccess){
printf("Copy is successfull\n");
}
else{
printf("Unfortunately\n");
} `

[pittlerf]

rtm8.cpp.txt
If I just put the same code in one of the HIP-Examples it works. So the problem should be really the compilation.

[pittlerf]

I encountered a similar issue in extern constant fails to link (global vars) #2249
https://github.com/ROCm-Developer-Tools/[HIP](https://github.com/ROCm-Developer-Tools/HIP/extern constant fails to link (global vars) #2249