Change UCX environment variables in __init__, document in knownissues.md (#370)

Whie the RPATH issue is now fixed in UCX 1.7.0 and later, it appears that the memory cache still does not work correctly. Switching to using `UCX_MEMTYPE_CACHE=no` produces fewer warning messages.

Author: simonbyrne

docs/make.jl

@@ -45,6 +45,7 @@ makedocs(
         "index.md",
         "installation.md",
         "usage.md",
+        "knownissues.md",
         "Examples" => EXAMPLES,
         "Reference" => [
             "library.md",

docs/src/knownissues.md

@@ -0,0 +1,38 @@
+# Known issues
+
+## UCX
+
+[UCX](https://www.openucx.org/) is a communication framework used by several MPI implementations.
+
+### Memory cache
+
+When used with CUDA, UCX intercepts `cudaMalloc` so it can determine whether the pointer passed to MPI is on the host (main memory) or the device (GPU). Unfortunately, there are several known issues with how this works with Julia:
+- https://github.com/openucx/ucx/issues/5061
+- https://github.com/openucx/ucx/issues/4001 (fixed in UCX v1.7.0)
+
+By default, MPI.jl disables this by setting
+```
+ENV["UCX_MEMTYPE_CACHE"] = "no"
+```
+at `__init__` which may result in reduced performance, especially for smaller messages.
+
+### Multi-threading and signal handling
+
+When using [Julia multi-threading](https://docs.julialang.org/en/v1/manual/parallel-computing/#man-multithreading-1), the Julia garbage collector internally [uses `SIGSEGV` to synchronize threads](https://docs.julialang.org/en/v1/devdocs/debuggingtips/#Dealing-with-signals-1).
+
+By default, UCX will error if this signal is raised ([#337](https://github.com/JuliaParallel/MPI.jl/issues/337)), resulting in a message such as:
+```
+Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0xXXXXXXXX)
+```
+
+This signal interception can be controlled by setting the environment variable `UCX_ERROR_SIGNALS`: if not already defined, MPI.jl will set it as:
+```
+ENV["UCX_ERROR_SIGNALS"] = "SIGILL,SIGBUS,SIGFPE"
+```
+at `__init__`. If set externally, it should be modified to exclude `SIGSEGV` from the list.
+
+## Microsoft MPI
+
+### Custom operators on 32-bit Windows
+
+It is not possible to use [custom operators with 32-bit Microsoft MPI](https://github.com/JuliaParallel/MPI.jl/issues/246), as it uses the `stdcall` calling convention, which is not supported by [Julia's C-compatible function pointers](https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/index.html#Creating-C-Compatible-Julia-Function-Pointers-1).

src/MPI.jl

@@ -77,14 +77,14 @@ function __init__()
         # though that would probably trigger a race condition
         error("MPI library has changed, please restart Julia")
     end
-    
-    # disable UCX memory hooks since it can mess up dlopen
-    # https://github.com/openucx/ucx/issues/4001
-    ENV["UCX_MEM_MMAP_RELOC"] = "no"
-    ENV["UCX_MEM_MALLOC_HOOKS"] = "no"
-    ENV["UCX_MEM_MALLOC_RELOC"] = "no"
-    ENV["UCX_MEM_EVENTS"] = "no"
 
+
+    # disable UCX memory cache, since it doesn't work correctly
+    # https://github.com/openucx/ucx/issues/5061
+    if !haskey(ENV, "UCX_MEMTYPE_CACHE")
+        ENV["UCX_MEMTYPE_CACHE"] = "no"
+    end
+    
     # Julia multithreading uses SIGSEGV to sync thread
     # https://docs.julialang.org/en/v1/devdocs/debuggingtips/#Dealing-with-signals-1
     # By default, UCX will error if this occurs (issue #337)