Help with NVLink Assesments in AMReX

[tony3601000]

Howdy,

I am attempting to do some weak/strong scaling tests with my AMReX-based CFD code using multiple NVIDIA GPUs. I have some questions:

1. This paper discusses scaling with and without GPU-aware MPI, which is similar to what we are attempting. Was the speed up in Fig.4 due to the usage of NVLink through GPU-aware MPI?
2. Using AMReX, is there a way to measure the bandwidth across NVLink for sims that use multiple NVIDIA GPUs (or at least detect the usage of NVLink)?
3. Are there any profiling tools you suggest besides NVIDIA Nsight Systems to help assess multi-GPU configurations and direct GPU-GPU communication?

Thank you for the help!

[WeiqunZhang]

Re: SC24 poster, the speedup was due to the use of GPU-aware MPI in Perlmutter. It was Cray's MPI implementation based on MPICH. I guess it uses nvlink for on-node communication. But my understanding is that, for off-node communication, it still needs to go through PCIe.