[Feature] Registering cuMem address without NVLS or MNNVL support

[dromniscience]

Currently, registering local cuMem-mapped addresses in RegisterMemory requires NVLS/MNNVL support (the code here explicitly requires this). Given that real-world PyTorch and Megatron workloads use the cuMem driver API to manage tensors, this causes problems when integrating MSCCL++ into PyTorch and Megatron on machines without NVLS/MNNVL support. AFAIK, cuMem should be usable without such specialized hardware. Is such coupling of the cuMem buffer registration and multicast support an intended feature or an ad-hoc constraint that will be released in a future release?

[Binyang2014]

Thanks for reporting this issue. It makes sense to me to decouple nvls and cuMem.
Could you share your dev env (GPU info and cuda version) and error info here? It would make it easier for us to solve the issue.

Just interested, I though torch use cudaMalloc by default to manage tensors (not sure about the latest releases). Not aware they change to driver API

[chhwang]

@dromniscience This is probably fixed in #704. Could you please have a try?