CUDA offload fails to verify devices > SM90

[hummingtree]

The most likely reason is the line here:

llvm-project/offload/plugins-nextgen/common/src/Utils/ELF.cpp

Line 79 in f49ee00

(Header.e_flags & EF_CUDA_SM) > EF_CUDA_SM90)

[Artem-B]

@hummingtree A bit more context for the bug report would be helpful.

• What exactly do you mean by "fails to verify devices" ?
• What do you consider to be an error?
• How do we reproduce it?
• What would you expect to happen if the things were working correctly?

[hummingtree]

So I took the release/20.x branch and compile it with

cmake -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local/llvm -DLLVM_ENABLE_BACKTRACES=ON -DLLVM_ENABLE_WERROR=OFF -DBUILD_SHARED_LIBS=ON -DLLVM_ENABLE_RTTI=ON -DLLVM_TARGETS_TO_BUILD="AArch64;NVPTX" -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS="clang;compiler-rt;lld;openmp" -DLLVM_ENABLE_RUNTIMES="libcxxabi;libcxx;libunwind;offload" -DLIBOMPTARGET_DEVICE_ARCHITECTURES="sm_90;sm_100" -DLIBOMPTARGET_ENABLE_DEBUG=OFF ../llvm-project/llvm

Then use this clang++ to compile

#include <cstdio>
#include <omp.h>

int main() {
  printf("omp_get_num_devices = %d\n", omp_get_num_devices());
}

with

clang++ -fopenmp -fopenmp-targets=nvptx64 --offload-arch=sm_100 main.cpp

On B200 GPUs (sm_100) we see that omp_get_num_devices = 0, with the following debug information

omptarget --> Init offload library!
OMPT --> Entering connectLibrary
OMPT --> OMPT: Trying to load library libomp.so
OMPT --> OMPT: Trying to get address of connection routine ompt_libomp_connect
OMPT --> OMPT: Library connection handle = 0xfffff7c5bf38
OMPT --> Exiting connectLibrary
omptarget --> Loading RTLs...
omptarget --> RTLs loaded!
PluginInterface --> Failure to check validity of image 0xaaaaaab34600: Unsupported NVPTX architecturePluginInterface --> Failure to check validity of image 0xaaaaaab34600: Unsupported NVPTX architecturePluginInterface --> Failure to check validity of image 0xaaaaaab34600: Unsupported NVPTX architectureomptarget --> No RTL found for image 0x0000aaaaaaaa0b58!
omptarget --> Done registering entries!
omptarget --> Call to omp_get_num_devices returning 0
omp_get_num_devices = 0

We expect to find non-zero number of devices - and the most likely the line of code mentioned above is the cause: any devices later than SM90 are seen as "Unsupported NVPTX architecture".

[hummingtree]

@Artem-B ^^^

[Artem-B]

@jhuber6 ^^^

[jhuber6]

There isn't a higher e_flag value for NVPTX yet. That'll need to be added.

[jhuber6]

Seems NVIDIA changed their e_flags format for the stuff post sm_100. Used to just be the lower byte but it looks pretty different now. @AlexMaclean any chance of NVPTX actually publishing the ELF header format cubins use? I'm not keen to reverse engineer it again.

Right now I'm just doing cat /dev/null | clang -x c - --target=nvptx64-nvidia-cuda -march=sm_100; llvm-readelf -h a.out to check.

[llvmbot]

@llvm/issue-subscribers-offload

Author: Jiqun Tu (hummingtree)

The most likely reason is the line here: https://github.com/llvm/llvm-project/blob/f49ee00ec47f596269639987f5ea8755a358ca16/offload/plugins-nextgen/common/src/Utils/ELF.cpp#L79