Kokkos Thermodynamics

[Thanduriel]

Kokkos Thermodynamics

Work in progress on the Kokkos(GPU) thermodynamics. (issue #672)

To track changes in buffers and to sync data between host and device a more data oriented approach is needed. To this end, the management of fields was made more explicit through the ModelArrayAccessor. The whole model and the tests have been updated to use this new interface which requires:

• explicitly listing all fields that have to be accessed for an operation
• moving the update code into a lambda to capture the fields
• updating member functions called in a kernel to get all field data explicitly through their arguments.

See for example physics/src/modules/LateralIceSpreadModule/HiblerSpread.cpp The USE_KOKKOS block should be ignored since is just a copy that has not been updated yet.

To actually enable device execution, the code needs to be ported to Kokkos which involves further changes:

• request the device buffers instead of the host ModelArray
• all data used in the kernel has to be captured explicitly by copy (especially static member variables)
• math functions (cos,min,sqrt) in the kernel have to be replaced with versions from Kokkos

See for example physics/src/modules/IceThermodynamicsModule/ThermoWinton.cpp.

The question is, how should these two different code paths be separated? In the dynamics, the Kokkos code is kept in separate files that build on top of the existing reference implementation, replacing all the relevant parts. This is necessary to achieve a tight integration for the actual computations. In long-term, the idea was to focus on maintaining the Kokkos version since it runs on both CPU and GPU and perhaps remove the old implementation at some point.

Thermodynamics could be ported in the same way, but since the individual components are much smaller it is also feasible to insert the switch inside the update routines and reuse more of the code. In fact, I build a light abstraction layer that hides all the specifics so there is just one version of the code, seeThermoWinton::update. Of course, the combined kernel for overElements still needs to adhere to the stricter rules of a Kokkos kernel, so using these abstractions would not automatically make all the code compatible. This abstraction is also somewhat redundant because Kokkos does effectively the same thing with its execution space concept. It just brings the added benefit of working directly with ModelArray and not needing Kokkos to build NeXtSIM-DG.