Updated AMS models

[marchdf]

Still left to do:

• unit tests
• regression tests
• ensure docs are up to date
• test GPU

Co-authored-by: Jeremy Melvin [email protected]

[jamelvin]

Looks good, some clean up needed and some consistency across different AMS files needs to be added

[marchdf]

On NVIDIA GPU:

   Nalu-Wind GIT Commit SHA: 1899277c211909679749120d701d7720cc1fbd1b
   Trilinos Version: 13.5

[==========] Running 534 tests from 126 test suites.
[----------] Global test environment set-up.
[----------] 1 test from Basic
[ RUN      ] Basic.CheckCoords1Elem
[       OK ] Basic.CheckCoords1Elem (1 ms)
[----------] 1 test from Basic (1 ms total)

[----------] 5 tests from BasicKokkos
[ RUN      ] BasicKokkos.discover_execution_space

Kokkos::Cuda is available.
Default execution space info: Device Execution Space:
  KOKKOS_ENABLE_CUDA: yes
Cuda Atomics:
  KOKKOS_ENABLE_CUDA_ATOMICS: no
Cuda Options:
  KOKKOS_ENABLE_CUDA_LAMBDA: yes
  KOKKOS_ENABLE_CUDA_LDG_INTRINSIC: yes
  KOKKOS_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE: yes
  KOKKOS_ENABLE_CUDA_UVM: yes
  KOKKOS_ENABLE_CXX11_DISPATCH_LAMBDA: yes

Cuda Runtime Configuration:
macro  KOKKOS_ENABLE_CUDA      : defined
macro  CUDA_VERSION          = 11020 = version 11.2
Kokkos::Cuda[ 0 ] Tesla V100-PCIE-16GB capability 7.0, Total Global Memory: 15.78 G, Shared Memory per Block: 48 K : Selected
Kokkos::Cuda[ 1 ] Tesla V100-PCIE-16GB capability 7.0, Total Global Memory: 15.78 G, Shared Memory per Block: 48 K

[...]

Note: Google Test filter = MijTensor*
[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from MijTensor
[ RUN      ] MijTensor.tri3
[       OK ] MijTensor.tri3 (0 ms)
[ RUN      ] MijTensor.quad4
[       OK ] MijTensor.quad4 (1 ms)
[ RUN      ] MijTensor.tet4
[       OK ] MijTensor.tet4 (0 ms)
[ RUN      ] MijTensor.wedge6
[       OK ] MijTensor.wedge6 (1 ms)
[ RUN      ] MijTensor.hex8
[       OK ] MijTensor.hex8 (0 ms)
[----------] 5 tests from MijTensor (2 ms total)

[----------] Global test environment tear-down
[==========] 5 tests from 1 test suite ran. (2 ms total)
[  PASSED  ] 5 tests.

[...]

Note: Google Test filter = AMSKernelHex8Mesh*
[==========] Running 16 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 16 tests from AMSKernelHex8Mesh
[ RUN      ] AMSKernelHex8Mesh.ngp_sst_ams_diff
[       OK ] AMSKernelHex8Mesh.ngp_sst_ams_diff (15 ms)
[ RUN      ] AMSKernelHex8Mesh.ngp_sstlr_ams_diff
[       OK ] AMSKernelHex8Mesh.ngp_sstlr_ams_diff (8 ms)
[ RUN      ] AMSKernelHex8Mesh.ngp_ko_ams_diff
[       OK ] AMSKernelHex8Mesh.ngp_ko_ams_diff (8 ms)
[ RUN      ] AMSKernelHex8Mesh.ngp_ke_ams_diff
[       OK ] AMSKernelHex8Mesh.ngp_ke_ams_diff (7 ms)
[ RUN      ] AMSKernelHex8Mesh.NGP_tke_sstams_node
[       OK ] AMSKernelHex8Mesh.NGP_tke_sstams_node (6 ms)
[ RUN      ] AMSKernelHex8Mesh.NGP_tke_sstlrams_node
[       OK ] AMSKernelHex8Mesh.NGP_tke_sstlrams_node (7 ms)
[ RUN      ] AMSKernelHex8Mesh.NGP_tke_keams_node
[       OK ] AMSKernelHex8Mesh.NGP_tke_keams_node (6 ms)
[ RUN      ] AMSKernelHex8Mesh.NGP_tke_koams_node
[       OK ] AMSKernelHex8Mesh.NGP_tke_koams_node (6 ms)
[ RUN      ] AMSKernelHex8Mesh.NGP_sdr_sstams_node
[       OK ] AMSKernelHex8Mesh.NGP_sdr_sstams_node (7 ms)
[ RUN      ] AMSKernelHex8Mesh.NGP_sdr_sstlrams_node
[       OK ] AMSKernelHex8Mesh.NGP_sdr_sstlrams_node (7 ms)
[ RUN      ] AMSKernelHex8Mesh.NGP_tdr_keams_node
[       OK ] AMSKernelHex8Mesh.NGP_tdr_keams_node (6 ms)
[ RUN      ] AMSKernelHex8Mesh.NGP_sdr_koams_node
[       OK ] AMSKernelHex8Mesh.NGP_sdr_koams_node (7 ms)
[ RUN      ] AMSKernelHex8Mesh.NGP_sstams_forcing
[       OK ] AMSKernelHex8Mesh.NGP_sstams_forcing (7 ms)
[ RUN      ] AMSKernelHex8Mesh.NGP_keams_forcing
[       OK ] AMSKernelHex8Mesh.NGP_keams_forcing (7 ms)
[ RUN      ] AMSKernelHex8Mesh.NGP_koams_forcing
[       OK ] AMSKernelHex8Mesh.NGP_koams_forcing (8 ms)
[ RUN      ] AMSKernelHex8Mesh.NGP_turb_visc_sstams_alg
[       OK ] AMSKernelHex8Mesh.NGP_turb_visc_sstams_alg (6 ms)
[----------] 16 tests from AMSKernelHex8Mesh (118 ms total)

[----------] Global test environment tear-down
[==========] 16 tests from 1 test suite ran. (118 ms total)
[  PASSED  ] 16 tests.

[marchdf]

Does anyone know why the CI fails here?

[psakievich]

@marchdf the docker container we use to drive CI is busted. I put in a fix for this to spack-manager that should fix the container. I will retrigger the CI to see if the container has been regenerated yet.

[marchdf]

@psakievich can we merge this? I am worried that a bunch of nalu-wind modernization is going to pass us by...

[psakievich]

@marchdf I would have merged it but the CI has been failing. I think it might be good to do a mini-hack-a-thon to update some of these PR's. I don't want this one to stay in purgatory any longer, but I also have quite a bit of changes to how fields are registered coming down the pipeline. I think it will make the code simpler. I am not planning to start that work until I get tooling updated a bit though. So I might suggest waiting until I get that started...