Add workaround for slow intra-node communication with Cray MPICH

Author: msimberg

docs/software/communication/cray-mpich.md

@@ -94,6 +94,31 @@ Note that this has a performance impact for small message sizes, so it should on
 export FI_CXI_SAFE_DEVMEM_COPY_THRESHOLD=0
 ```
 
+[](){#ref-communication-cray-mpich-slow-intranode}
+#### Slow intra-node host communication with Cray MPICH
+
+Cray MPICH can perform badly when doing intra-node CPU-CPU memory communication.
+
+!!! info "Workaround"
+    In some situations Cray MPICH can perform better when communication is done over the NICs, even within a node.
+    To force Cray MPICH to use NICs for all communication, set:
+
+    ```bash
+    export MPIR_CVAR_NO_LOCAL=1
+    ```
+
+    Whenever possible, prefer using GPU-GPU communication instead of CPU-CPU communication.
+    It can even be beneficial to transfer data to the GPU only for the communication even if the buffer originally is in CPU memory.
+
+#### `"cxil_map: write error"` when doing inter-node GPU-aware MPI communication
+
+This error message is sometimes triggered by applications that use GPU Direct MPI calls when they trigger a bug in gdrcopy (a low-level library used to copy buffers between GPUs).
+Setting the following option will completely disable gdrcopy.
+Note that this has a performance impact for small message sizes, so it should only be enabled on a case-by-case basis.
+```bash
+export FI_CXI_SAFE_DEVMEM_COPY_THRESHOLD=0
+```
+
 ### Resolved issues
 
 #### `"cxil_map: write error"` when doing inter-node GPU-aware MPI communication