diff --git a/docs/software/nodelevel_parallelism/index.md b/docs/software/nodelevel_parallelism/index.md new file mode 100644 index 00000000..f1af3a65 --- /dev/null +++ b/docs/software/nodelevel_parallelism/index.md @@ -0,0 +1,6 @@ +[](){#ref-software-nodelevel-parallelism} +# Node-Level Parallelism Libraries + +See the individual pages for each library for information on how to use and best configure the libraries. + +* [OpenMP][openmp] diff --git a/docs/software/nodelevel_parallelism/openmp.md b/docs/software/nodelevel_parallelism/openmp.md new file mode 100644 index 00000000..8b3d1ffe --- /dev/null +++ b/docs/software/nodelevel_parallelism/openmp.md @@ -0,0 +1,23 @@ +[](){#openmpi} +# OpenMP + +[OpenMP](https://www.openmp.org/specifications/) OpenMP is a shared-memory parallel programming API that provides compiler directives, runtime routines, and environment variables to simplify multithreading in C, C++, and Fortran. +It is a common library that HPC applications use to achieve node-level parallelism. + + +## Known issues + +!!! warning + As of 26-Mar-2025 the [Santis][ref-cluster-santis] cluster is configured in "Low Noise Mode" (LNM). This setting confines system processes and operations + to the first core of each of the four NUMA regions in a node (i.e., cores 0, 72, 144, 216). + + The consequence of this setting is that only 71 cores per socket can be requested by an application (for a total of 284 cores instead of 288 cores per node). + + Furthermore, LNM requires that OpenMP-based applications instruct OpenMP to properly place threads on the remaining cores. + +On [Santis][ref-cluster-santis] the recommended OpenMP affinity settings are + +```bash +export OPENMP_PLACES=cores +export OPENMP_PROC_BIND=close +``` diff --git a/mkdocs.yml b/mkdocs.yml index 792c0c65..590bcd4b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -64,6 +64,9 @@ nav: - 'NCCL': software/communication/nccl.md - 'RCCL': software/communication/rccl.md - 'libfabric': software/communication/libfabric.md + - 'Node-level Parallelism Libraries': + - software/nodelevel_parallelism/index.md + - 'OpenMP': software/nodelevel_parallelism/openmp.md - 'Building and Installing software': - build-install/index.md - 'uenv': build-install/uenv.md