Add workflow for weekly kokkos develop testing

[tcclevenger]

This is in an effort to continually update Kokkos with new releases, instead of large subversion jumps each time a component needs a new version.

[BFB]

[tcclevenger]

@bartgol I added comments for every place this differs from eamxx-sa. Is there an easy way for me to manually test a .github workflow?

[bartgol]

I think that, unless the workflow is in master, it must run at least once to appear on the actions page. If you put back the PR trigger, it may run, and at that point, you could manually test it form the actions tab via workflow dispatch...

[tcclevenger]

@bartgol Looks like it doesn't pick it up even with the PR tab.

[tcclevenger]

Previously, C++20 was not being used (EAMxx overwrote user defined CMAKE_CXX_STANDARD var) and tests were passing. Now that we are using C++20, the builds are failing, so I changed to draft while I address the errors.

[bartgol]

One thought I just had: why don't we make the new workflow do the following:

• create a temporary branch like kokkos-develop-testing/<DATE>
• update kokkos submod of ekat in this branch, and push to the remote
• call the eamxx-v1 and eamxx-sa (or just one) workflow

the AI bot suggests this step for triggering another workflow (not sure if it works)

          - name: Trigger target workflow
            uses: actions/github-script@v6
            with:
              github-token: ${{ secrets.PAT_TOKEN }} # Use a PAT with 'repo' scope
              script: |
                await github.rest.actions.createWorkflowDispatch({
                  owner: context.repo.owner,
                  repo: context.repo.repo,
                  workflow_id: 'target-workflow.yml', // The file name of the target workflow
                  ref: 'main', // The branch on which the workflow should be triggered
                  inputs: {
                    # Optional inputs for the dispatched workflow
                    my_input_key: 'my_input_value',
                    another_input: 'another_value'
                  }
                });

But whatever the syntax, I'm thinking that calling the existing eamxx-sa/eamxx-v1 workflows (to run on the newly created "kokkos-update" branch) is more robust, as we only have to maintain the testing framework in one workflow.

[tcclevenger]

One thought I just had: why don't we make the new workflow do the following:

* create a temporary branch like `kokkos-develop-testing/<DATE>`

* update kokkos submod of ekat in this branch, and push to the remote

* call the eamxx-v1 and eamxx-sa (or just one) workflow

the AI bot suggests this step for triggering another workflow (not sure if it works)

          - name: Trigger target workflow
            uses: actions/github-script@v6
            with:
              github-token: ${{ secrets.PAT_TOKEN }} # Use a PAT with 'repo' scope
              script: |
                await github.rest.actions.createWorkflowDispatch({
                  owner: context.repo.owner,
                  repo: context.repo.repo,
                  workflow_id: 'target-workflow.yml', // The file name of the target workflow
                  ref: 'main', // The branch on which the workflow should be triggered
                  inputs: {
                    # Optional inputs for the dispatched workflow
                    my_input_key: 'my_input_value',
                    another_input: 'another_value'
                  }
                });

But whatever the syntax, I'm thinking that calling the existing eamxx-sa/eamxx-v1 workflows (to run on the newly created "kokkos-update" branch) is more robust, as we only have to maintain the testing framework in one workflow.

I can try, the only difference is that new cmake options must be added in, so maybe if I can hack those in this will delete a lot of duplicated code.

[tcclevenger]

@bartgol This is ready to take a look. I opted to make changes inside the test-all-eamxx action instead of having a new update-kokkos-to-dev action which checks out the branch and edits the .cmake files with 3 needed options. I did this because I needed to modify test-all-eamxx anyways to alter the name of the cdash report to distinguish from the normal nightlies.

I manually ran and submitted to CDASH. Here's one of the tests: https://my.cdash.org/build/3099376

Note, CUDA fails because we need to update version 12.1 -> 12.2+ to satisfy Kokkos' version requirements.

[bartgol]

This looks good. I left just one comment on the test-all arg.

Also, our ghci container for CUDA will be soon updated to be ubi9 based, with cuda 12.4. It used to have 12.1 to mimic pm-gpu, but our pm-gpu defaults have since been updated, so the container should be updated as well. I already submitted the image to the AT2 folks, so it's ready to change any moment (prob this week).

[tcclevenger]

Manually submitted to CDASH, and went as expected. Merging.