Merged
Conversation
ax3l
reviewed
Feb 14, 2025
Src/Base/AMReX_GpuDevice.cpp
Outdated
| if ((Machine::name() != "nersc.perlmutter") && | ||
| (Machine::name() != "olcf.frontier")) | ||
| { | ||
| amrex::Warning("Multiple GPUs are visible to each MPI rank. This is usually not an issue. But this may lead to incorrect or suboptimal rank-to-GPU mapping."); |
Member
There was a problem hiding this comment.
I think with the implementation below, we need to be more precise now: for the machines we implement logic for, we should post something like: "Fixing GPU assignment for Frontier according to heuristics..." or so?
For perlmutter and frontier, if there are multiple device available, we will try to map GPUs to the closest core. For an FFT test on perlmutter using 256 nodes, the correct mapping reduced the run time from 0.172 to 0.127. Note that you can achieve the similar effect with `srun ... bash -c "export CUDA_VISIBLE_DEVICES=\$((3-SLURM_LOCALID)); ..."` by manually limiting the number of visible devices. But in this commit, we are trying to do this automatically for the user. Also note that MPI appears to crash with gpu-bind=closest on perlmutter. So we need to use gpu-bind=none. For frontier, you could use gpu-bind=closest. But if your use gpu-bind=none, this commit will try to do the correct mapping for you. In this commit, we also removed the old machin stuff and added new code for machine detection.
WeiqunZhang
force-pushed
the
perlmutter_gpu_mapping
branch
from
d779130 to
947955a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For perlmutter and frontier, if there are multiple devices available, we will try to map GPUs to the closest core.
For an FFT test on perlmutter using 256 nodes, the correct mapping reduced the run time from 0.172 to 0.127. Note that you can achieve the similar effect with
srun ... bash -c "export CUDA_VISIBLE_DEVICES=\$((3-SLURM_LOCALID)); ..."by manually limiting the number of visible devices. But in this commit, we are trying to do this automatically for the user. Also note that MPI appears to crash with gpu-bind=closest on perlmutter. So we need to use gpu-bind=none.For frontier, you could use gpu-bind=closest. But if your use gpu-bind=none, this commit will try to do the correct mapping for you.
In this commit, we also removed the old machine stuff and added new code for machine detection.