Fix: Fix crash in Debug build with multi-GPU due to forced cudaSetDevice(0)

[wangtianxiang]

🐞 Bug Behavior

When compiling with CMAKE_BUILD_TYPE=Debug and running any GPU test case via mpirun -np 2 abacus using 2 GPUs, the program crashes at a cudaMemset call with cudaErrorInvalidValue: invalid argument:

Additionally, even when not using CMAKE_BUILD_TYPE=Debug, although the program runs without crashing, no device information is printed in device.log.

🔍 Root Cause

Through debugging, we found that the constructor Psi<T, Device>::Psi calls base_device::information::print_device_info, which internally invokes cudaSetDevice(0) — forcing all MPI ranks to use GPU 0.

This creates a critical inconsistency in multi-GPU runs:

• Initially, rank 0 and rank 1 are correctly bound to GPU 0 and GPU 1 respectively, and allocate device memory on their assigned GPUs.
• However, when print_device_info is called (e.g., for logging), rank 1 is forcibly switched to GPU 0 via cudaSetDevice(0).
• Later, when cudaMemcpy is called on rank 1, it attempts to set memory that resides on GPU 1, while the current device context is GPU 0 → resulting in cudaErrorInvalidValue.

The root issue: Hard-coded cudaSetDevice(0) inside a shared utility function breaks multi-GPU context isolation in MPI environments.

⚠️ To legally access memory across different GPUs (e.g., GPU 1 accessing GPU 0’s memory), Peer-to-Peer access must be explicitly enabled via cudaDeviceEnablePeerAccess.

Additionally, in non-Debug builds (Release with -O3), the program does not crash — but produces no output in device.log. This is due to an undefined behavior caused by missing template specialization declaration:

• A specialization print_device_info<DEVICE_GPU> is defined in output_device.cpp, but not declared in device.h.
• As a result, whether the compiler instantiates the empty primary template or the specialized version becomes undefined behavior.
• In Debug mode (-O0), the linker often resolves to the specialization → cudaSetDevice(0) is called → crash.
• In Release mode (-O3), the compiler aggressively inlines/optimizes and often picks the empty primary template → no cudaSetDevice(0) → no crash, but no logging output.

🛠️ Solution

To fix this issue, we apply two key changes:

1. Replace cudaSetDevice(0) with cudaGetDevice() in print_device_info
2. Declare the template specialization in device.h

[mohanchen]

Good catch! Thanks a lot for your contribution!