Add more infrastructure for timing

Adds a new module for timing under infra/. This module builds upon the Pacer library, but extends it
with support for specifying timing levels and automatically adding Kokkos fences. It also includes more advanced
timer options that are useful in special circumstances. Finally, it adds integration with NVTX ranges on NVIDIA GPUs for profiling with vendor-specific tools.

Author: philipwjones

components/omega/OmegaBuild.cmake

@@ -29,6 +29,7 @@ macro(common)
   option(OMEGA_DEBUG "Turn on error message throwing (default OFF)." OFF)
   option(OMEGA_LOG_FLUSH "Turn on unbuffered logging (default OFF)." OFF)
   option(OMEGA_TEST_CDASH "Turn on CDash support (default ON)." ON)
+  option(OMEGA_EXTERNAL_PROF "Integration of Omega timers with external profiling tools (default OFF)." OFF)
 
   if("${OMEGA_BUILD_TYPE}" STREQUAL "Debug" OR "${OMEGA_BUILD_TYPE}" STREQUAL "DEBUG")
     set(OMEGA_DEBUG ON)

components/omega/configs/Default.yml

@@ -1,4 +1,9 @@
 Omega:
+  Timing:
+    Level: 2
+    AutoFence: true
+    TimingBarriers: false
+    PrintAllRanks: false
   TimeIntegration:
     CalendarType: No Leap
     TimeStepper: Forward-Backward

components/omega/doc/devGuide/Timing.md

@@ -0,0 +1,96 @@
+(omega-dev-timing)=
+
+# Timing
+
+The timing infrastructure builds upon the E3SM Pacer library for wall clock timing.
+Pacer integrates, whenever possible, platform-specific marker APIs.
+
+# Initialization
+
+To initialize the Pacer library call
+```c++
+Pacer::initialize(Comm);
+```
+where `Comm` is an MPI communicator.
+
+# Writing-out timing data
+
+To write out timing data call
+```c++
+Pacer::print(FilePrefix, PrintAllRanks);
+```
+where `FilePrefix` is the prefix for all output files and an optional argument `PrintAllRanks` determines
+if all MPI ranks should print their data.
+
+# Finalization
+
+To finalize the Pacer library call
+```c++
+Pacer::finalize();
+```
+
+# Basic timer use
+
+To time a region of code enclose it with calls to `Pacer::start` and `Pacer::stop` functions, like so
+```c++
+Pacer::start(Name, Level);
+// region of code to be timed
+Pacer::stop(Name, Level);
+```
+These functions take a string `Name` and a non-negative integer `Level`.
+The added timer will be active only if the timing level set in the config file is greater or equal to `Level`.
+
+# Advanced timing functions
+
+## Conditional MPI barriers
+
+Properly timing MPI communication might require inserting MPI barriers. It might be desirable
+to remove those barriers in production runs. Pacer provides a function
+```c++
+Pacer::timingBarrier(TimerName, Level, Comm)
+```
+which adds an MPI barrier and puts a timer around it using the communicator `Comm`.
+Whether barriers added by this function are actually called can be controlled by the following functions
+```c++
+  Pacer::enableTimingBarriers();
+  Pacer::disableTimingBarriers();
+```
+
+## Adding parent prefixes
+
+It might be desirable to add a prefix to a group of timers based on their parent timer.
+To enable this Pacer provides the `addParentPrefix()` and `removeParentPrefix()` functions.
+For example, the following call sequence
+```c++
+Pacer::start("Parent", 0);
+Pacer::addParentPrefix();
+
+Pacer::start("Child", 0);
+Pacer::stop("Child", 0);
+
+Pacer::removeParentPrefix();
+Pacer::start("Parent", 0);
+```
+results in output where the "Child" timer shows up as "Parent:Child" in the output files.
+This is useful when timers are added inside general purpose routines, that are called
+from many places in the code, such as halo exchange.
+
+## Disabling timers
+
+It might be desirable programmatically disable or enable timing. To allow that, Pacer provides
+the `disableTimers()` and `enableTimers()` functions. In the following call sequence
+```c++
+Pacer::disableTiming();
+
+Pacer::start("Timer1", 0);
+Pacer::stop("Timer1", 0);
+
+Pacer::enableTiming();
+
+Pacer::start("Timer2", 0);
+Pacer::stop("Timer2", 0);
+```
+`Timer1` is not timed while `Timer2` is. This is useful mainly when done conditionally.
+For example, the first call to some function takes much longer than subsequent calls,
+and having a detailed timing breakdown of the first call is not important. In that case,
+it might be desirable to have a separate timer for the first call with its child timers disabled.

components/omega/doc/index.md

@@ -49,6 +49,7 @@ userGuide/Reductions
 userGuide/Tracers
 userGuide/TridiagonalSolvers
 userGuide/VertCoord
+userGuide/Timing
 ```
 
 ```{toctree}
@@ -88,6 +89,7 @@ devGuide/Reductions
 devGuide/Tracers
 devGuide/TridiagonalSolvers
 devGuide/VertCoord
+devGuide/Timing
 ```
 
 ```{toctree}

components/omega/doc/userGuide/Timing.md

@@ -0,0 +1,51 @@
+(omega-user-timing)=
+
+# Timing
+
+Omega uses the Pacer library for timing the code and incorporates timers around
+various parts of the code.
+
+By default, the timing output is written to two files: `omega.summary` and `omega.timing0`.
+The `omega.summary` file presents accumulated timing statistics across all MPI ranks.
+The `omega.timing.0` show timing result only from the first rank.
+
+There are four parameters that are set by the user in the input configuration
+file that control the timing behavior. These are:
+```yaml
+Timing:
+   Level: 2
+   AutoFence: true
+   TimingBarriers: false
+   PrintAllRanks: false
+```
+The `Level` parameter is a non-negative integer that determines the granularity of timers.
+Increasing it will turn on more timers.
+Having more timers provides more detailed information, but it also comes with increased overhead,
+and may be counter-productive if a high-level look at model performance is sufficient.
+
+The `Autofence` Boolean option determines if Kokkos fences are automatically added before every timer call.
+This option **needs** to be true for accurate timing using Omega timers on GPU-based systems.
+However, there are circumstances when turning off automatic fences is useful.
+The main use case is using external profiling tools.
+Another one is measuring the overhead of automatic synchronization for very high timing levels.
+
+The `TimingLevel` Boolean option determines if MPI barriers are added before or after certain timers.
+Adding barriers may be necessary to properly measure communication time, but it can add top much overhead in
+production runs.
+
+The `PrintAllRanks` Boolean option determines if all ranks should print their timing information. If this
+option is set to `true` the output will include additional files `omega.timing.i` with the
+timing data from rank `i`.
+
+## Integration with external profiling tools
+
+External profilers often include APIs to mark regions of code for detailed profiling.
+On some platforms, Omega timers automatically add these annotations.
+Currently, this is only implemented on systems with NVIDIA GPUs using NVTX.
+
+This allows, for example, to use the Nsight Compute kernel profiler to obtain
+detailed kernel information for all kernels enclosed in the `Tend:computeVelocityTendencies`
+Omega timer.
+```bash
+mpirun -np 1 ncu --nvtx --nvtx-include "Tend:computeVelocityTendencies/" omega.exe
+```

components/omega/external/CMakeLists.txt

@@ -94,9 +94,23 @@ if (NOT TARGET pacer)
 
     target_link_libraries(
         pacer
-        PUBLIC
+        PRIVATE
         gptl
+        Kokkos::kokkos
     )
+
+    target_compile_definitions(
+        pacer
+        PRIVATE
+        PACER_HAVE_KOKKOS=1
+     )
+    if (OMEGA_EXTERNAL_PROF)
+      target_compile_definitions(
+          pacer
+          PRIVATE
+          PACER_ADD_RANGES=1
+      )
+    endif()
 endif()
 
 # Add the parmetis and related libraries