Add timing docs

Author: mwarusz

components/omega/doc/devGuide/Timing.md

@@ -0,0 +1,96 @@
+(omega-dev-timing)=
+
+# Timing
+
+The timing infrastructure builds upon the E3SM Pacer library for wall clock timing.
+Pacer integrates, whenever possible, platform-specific marker APIs.
+
+# Initialization
+
+To initialize the Pacer library call
+```c++
+Pacer::initialize(Comm);
+```
+where `Comm` is an MPI communicator.
+
+# Writing-out timing data
+
+To write out timing data call
+```c++
+Pacer::print(FilePrefix, PrintAllRanks);
+```
+where `FilePrefix` is the prefix for all output files and an optional argument `PrintAllRanks` determines
+if all MPI ranks should print their data.
+
+# Finalization
+
+To finalize the Pacer library call
+```c++
+Pacer::finalize();
+```
+
+# Basic timer use
+
+To time a region of code enclose it with calls to `Pacer::start` and `Pacer::stop` functions, like so
+```c++
+Pacer::start(Name, Level);
+// region of code to be timed
+Pacer::stop(Name, Level);
+```
+These functions take a string `Name` and a non-negative integer `Level`.
+The added timer will be active only if the timing level set in the config file is greater or equal to `Level`.
+
+# Advanced timing functions
+
+## Conditional MPI barriers
+
+Properly timing MPI communication might require inserting MPI barriers. It might be desirable
+to remove those barriers in production runs. Pacer provides a function
+```c++
+Pacer::timingBarrier(TimerName, Level, Comm)
+```
+which adds an MPI barrier and puts a timer around it using the communicator `Comm`.
+Whether barriers added by this function are actually called can be controlled by the following functions
+```c++
+  Pacer::enableTimingBarriers();
+  Pacer::disableTimingBarriers();
+```
+
+## Adding parent prefixes
+
+It might be desirable to add a prefix to a group of timers based on their parent timer.
+To enable this Pacer provides the `addParentPrefix()` and `removeParentPrefix()` functions.
+For example, the following call sequence
+```c++
+Pacer::start("Parent", 0);
+Pacer::addParentPrefix();
+
+Pacer::start("Child", 0);
+Pacer::stop("Child", 0);
+
+Pacer::removeParentPrefix();
+Pacer::start("Parent", 0);
+```
+results in output where the "Child" timer shows up as "Parent:Child" in the output files.
+This is useful when timers are added inside general purpose routines, that are called
+from many places in the code, such as halo exchange.
+
+## Disabling timers
+
+It might be desirable programmatically disable or enable timing. To allow that, Pacer provides
+the `disableTimers()` and `enableTimers()` functions. In the following call sequence
+```c++
+Pacer::disableTiming();
+
+Pacer::start("Timer1", 0);
+Pacer::stop("Timer1", 0);
+
+Pacer::enableTiming();
+
+Pacer::start("Timer2", 0);
+Pacer::stop("Timer2", 0);
+```
+`Timer1` is not timed while `Timer2` is. This is useful mainly when done conditionally.
+For example, the first call to some function takes much longer than subsequent calls,
+and having a detailed timing breakdown of the first call is not important. In that case,
+it might be desirable to have a separate timer for the first call with its child timers disabled.

components/omega/doc/index.md

@@ -49,6 +49,7 @@ userGuide/Reductions
 userGuide/Tracers
 userGuide/TridiagonalSolvers
 userGuide/VertCoord
+userGuide/Timing
 ```
 
 ```{toctree}
@@ -88,6 +89,7 @@ devGuide/Reductions
 devGuide/Tracers
 devGuide/TridiagonalSolvers
 devGuide/VertCoord
+devGuide/Timing
 ```
 
 ```{toctree}

components/omega/doc/userGuide/Timing.md

@@ -0,0 +1,51 @@
+(omega-user-timing)=
+
+# Timing
+
+Omega uses the Pacer library for timing the code and incorporates timers around
+various parts of the code.
+
+By default, the timing output is written to two files: `omega.summary` and `omega.timing0`.
+The `omega.summary` file presents accumulated timing statistics across all MPI ranks.
+The `omega.timing.0` show timing result only from the first rank.
+
+There are four parameters that are set by the user in the input configuration
+file that control the timing behavior. These are:
+```yaml
+Timing:
+   Level: 2
+   AutoFence: true
+   TimingBarriers: false
+   PrintAllRanks: false
+```
+The `Level` parameter is a non-negative integer that determines the granularity of timers.
+Increasing it will turn on more timers.
+Having more timers provides more detailed information, but it also comes with increased overhead,
+and may be counter-productive if a high-level look at model performance is sufficient.
+
+The `Autofence` Boolean option determines if Kokkos fences are automatically added before every timer call.
+This option **needs** to be true for accurate timing using Omega timers on GPU-based systems.
+However, there are circumstances when turning off automatic fences is useful.
+The main use case is using external profiling tools.
+Another one is measuring the overhead of automatic synchronization for very high timing levels.
+
+The `TimingLevel` Boolean option determines if MPI barriers are added before or after certain timers.
+Adding barriers may be necessary to properly measure communication time, but it can add top much overhead in
+production runs.
+
+The `PrintAllRanks` Boolean option determines if all ranks should print their timing information. If this
+option is set to `true` the output will include additional files `omega.timing.i` with the
+timing data from rank `i`.
+
+## Integration with external profiling tools
+
+External profilers often include APIs to mark regions of code for detailed profiling.
+On some platforms, Omega timers automatically add these annotations.
+Currently, this is only implemented on systems with NVIDIA GPUs using NVTX.
+
+This allows, for example, to use the Nsight Compute kernel profiler to obtain
+detailed kernel information for all kernels enclosed in the `Tend:computeVelocityTendencies`
+Omega timer.
+```bash
+mpirun -np 1 ncu --nvtx --nvtx-include "Tend:computeVelocityTendencies/" omega.exe
+```