Stokhos: Compilation errors with `Stokhos_ENABLE_Ensemble_Reduction=ON`

[maartenarnst]

@trilinos/stokhos
@etphipp
@crtrott

We see compilation errors when we compile Trilinos with Stokhos with Stokhos_ENABLE_Ensemble_Reduction=ON.

The compilation errors are related to the file packages/stokhos/src/sacado/kokkkos/vector/linalg/Kokkos_Blas1_MP_Vector.hpp. There are a number of lines in this file that look like

typename Kokkos::FlatArrayType<XVector>::type x_flat = x;

and those lines now fail after the update to Kokkos 5.0.

The PR:

• Refactor Stokhos to support Kokkos 5.0 #14700

fixed many such occurrences in other files. A hypothesis is that the occurrences in this blas1 file were missed because that PR was compiled and tested with Stokhos_ENABLE_Ensemble_Reduction=OFF.

It appears that certain occurrences of this "flattening" in this blas1 file can be fixed along the lines of what was done in #14700. However, this blas1 file contains functions that perform blas1 operations column-wise on multi vectors. It seems that these functions cannot be handled with the helper function reinterpret_as_unmanaged_scalar_flat_view from #14700. It seems that we would need a new helper function to flatten columnwise.

As an alternative to making changes, a solution could also consist in deprecating the Stokhos_ENABLE_Ensemble_Reduction=ON code path.

[etphipp]

Thanks for reporting @maartenarnst and sorry for the trouble! You are right that this was missed because there is no automated testing with ensemble-reduction on. I guess the question is whether it is important to preserve that code path. Do you use or have any need for ensemble-reduction being on?

[maartenarnst]

Hey @etphipp. We have been using Trilinos with Stokhos with ensemble reduction on in our code, but I suspect it will be fairly straightforward for us to switch to ensemble reduction off. I'll try to make the switch in the next few days and let you know.

Keeping the current code paths and perhaps exploring whether test coverage could be increased could be a path forward. Perhaps there is a way to configure some of the Trilinos testers to use different configurations? Perhaps the nightlies? Dropping certain code paths could be an option too. I'm not sure where the right trade-off is or which code path to drop. Both ensemble reduction on and off have quite significant specific code paths and helper functions.

[maartenarnst]

Looking in more detail at switching our code to stokhos with ensemble reduction off, it turned out that it is actually quite a significant task. One of the reasons is that we use ensembles with NOX, which hardcodes the scalar double.

And so I've switched gears to extending the changes for ensemble reduction off to also cover the case with ensemble reduction on.

I've made the PR:

• Stokhos: fix ensemble propagation with reduction for new view #14901

@etphipp, @crtrott Would you have a moment to take a look?