Devise a method for BFB testing in (standalone) mam4xx

[jeff-cohere]

Right now, integrating code changes into E3SM takes significant effort, because our Fortran -> C++ validation does not use bit-for-bit identical testing for merging PRs into mam4xx's main branch. Consequently, we may not find out about answer-changing code until we attempt to update the mam4xx submodule in E3SM, at which point it can be difficult to isolate and identify specific issues. We would like to establish some sort of bit-for-bit testing within mam4xx so we can identify these kinds of answer-changing modifications before integrating them into E3SM/EAMxx.

Proposed Approach

Because tracking bit-for-bit changes to C++ is not related to Fortran code, we can modify and augment our validation suite with machinery to generate "baselines" against which we can do bit-for-bit comparisons with new code changes. Instead of generating binary files for each validation test, we can make use of Andrew Bradley's BFB hashing tool. This tool is located in the E3SM repo, but it's so little code (2 files) that we can either duplicate it within this repo or ask for it to be moved into a common submodule such as EKAT. In particular, the code for creating and printing the hash for an atmosphere process is here.

It will take a little thinking to find the right granularity for a hash in order to help us identify specific changes to answers, but these hashes are tiny compared to binary files, so I think this approach gives us enough flexibility to improve our visibility of answer-changing code modifications.

[jeff-cohere]

If we want to use our validation tests as our BFB protection, we could add a function to Skywalker to have it generate BFB hashes for each of its output fields, which can be written to its Python module. These hashes could then be retrieved either via grep or a Python script that compares hashes. Maybe this can be done in a simple way that doesn't require us to add the hashing code to Skywalker, but take advantage of the context it provides with all the output fields of interest.

[jeff-cohere]

Actually, no modification to Skywalker is necessary. This can be done with a post-processing function added to the validation namespace that accepts a Skywalker ensemble. This function can iterate over the ensemble inputs and outputs, accumulating a hash for output. Then each validation driver (of which there's only a handful) can be modified to call this hashing function.

[jeff-cohere]

Okay, I lied. I had to add some features to Skywalker to allow one to traverse the inputs and outputs in an ensemble so they can be hashed (PR here). I'm using these new features in this branch to do the BFB hashing.