UCT/CUDA/CUDA_IPC: Fixed cuda_ipc_cache cleanup at process termination.

[rakhmets]

What?

Fixed cuda_ipc_cache cleanup at process termination to close opened CUDA IPC memory handles.

Why?

UCS_STATIC_CLEANUP runs in a destructor when the UCX library is unloaded. Unload order is undefined, so the CUDA Driver can already be deinitialized. Then uct_cuda_ipc_destroy_cache -> uct_cuda_ipc_cache_purge -> uct_cuda_ipc_primary_ctx_retain_and_push calls cuDevicePrimaryCtxGetState, which (as any other CUDA Driver API) returns CUDA_ERROR_DEINITIALIZED.

[yosefe]

so atexit handler would run before cuda driver is deinitialized?
and if cuda driver is deinitialized before we destroy the mem handles it causes a leak?

[rakhmets]

so atexit handler would run before cuda driver is deinitialized? and if cuda driver is deinitialized before we destroy the mem handles it causes a leak?

In most cases - yes. There are still some corner cases. E.g. if the first call to cuInit is from cuda_md.c's UCS_STATIC_INIT, and is called after atexit from cuda_ipc_cache.c's UCS_STATIC_INIT. It doesn't cause the leak, as the driver closes the mem handles at process exit. It's the best effort.