v4.1.x: opal/cuda: Handle VMM pointers

[Akshay-Venkatesh]

Memory allocated using cumemcreate API with location as {CU_MEM_LOCATION_TYPE_HOST/CU_MEM_LOCATION_TYPE_HOST_NUMA/CU_MEM_LOCATION_TYPE_HOST _NUMA_CURRENT} can be detected as host memory type by pointer query API but this doesn't allow the CPU to access such memory using memcpy or other CPU load/store mechanisms unless explicitly requested with cuMemSetAccess. Without the changes in this PR, HOST_NUMA backed cumemcreate memory is detected as host by openmpi layers (opal/datatype, ompi/coll) and subsequent accesses by CPU thread leads to illegal access errors.

bot:notacherrypick

[github-actions]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

d2921b0: opal/cuda: avoid direct access to cumem host numa ...

• check_signed_off: does not contain a valid Signed-off-by line
• check_cherry_pick: does not include a cherry pick message (did you need to bot:notacherrypick?)

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

[github-actions]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

9cd2372: opal/cuda: avoid direct access to cumem host numa ...

• check_cherry_pick: does not include a cherry pick message (did you need to bot:notacherrypick?)

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

[github-actions]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

b72a410: opal/cuda: avoid direct access to cumem host numa ...

• check_cherry_pick: does not include a cherry pick message (did you need to bot:notacherrypick?)

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

[jsquyres]

@Akshay-Venkatesh @janjust So this isn't needed / doesn't exist in main/v5.0.x?

[Akshay-Venkatesh]

@Akshay-Venkatesh @janjust So this isn't needed / doesn't exist in main/v5.0.x?

Hi @jsquyres . It is needed but the changes will go into accelerator code paths that are quite different from those that exist in 4.1.x series. I'll post a PR soon.

[jsquyres]

Ok, good enough.

PR was changed after review

[jsquyres]

@Akshay-Venkatesh You just changed this PR significantly. Is it complete and fully tested?

[Akshay-Venkatesh]

@jsquyres After making changes to main branch I noticed that similar code would fit for 4.1.x and I had missed an additional check that was needed before marking memory as device vs host. I made those changes to both my branches and I've tested this extensively to make sure everything passes. Would appreciate another round of reviews to make sure I didn't miss anything.

[bosilca]

This looks not only overly complicated but also incorrect.

Regarding correctness: according to the CUDA documentation each call the cuMemRetainAllocationHandle must be matched with a call to cuMemRelease, which i don't see in this PR. This will result in the memory region referenced here not being able to be released.

What exactly do you get from the combination cuMemRetainAllocationHandle + cuMemGetAllocationPropertiesFromHandle that you could not have obtained from cuMemGetAccess ?

[jsquyres]

Put this back in Draft mode, because @bosilca's last comments on here were voicing objections (and I don't want to accidentally merge it). So let's get those objections addressed, and then this can get merged.