GOCART v2.5.0: Fails start-stop and Possible UFS issues

[mathomp4]

Summary

GOCART v2.5.0 does not pass start-stop regression. GOCART v2.4.0 does. As does v2.4.3. As does v2.5.0-rc.3

So the current changes to focus on are:

v2.5.0-rc.3...v2.5.0

How to Reproduce

1. Clone GEOSgcm v11 main
2. Get GOCART v2.5.0 with fix with mepo checkout bugfix/v2.5-fix-ni-yaml GOCART
3. Build
4. Set up an experiment (I did c24 AMIP)
5. Go the experiment, set it up as usual with restarts, etc. (I do makeoneday.bash link allhis)
6. On an interactive node, go into the regress/ folder and run ./gcm_regress.j --startstop

---

Longwinded Explanation

This is a "high-level" issue.

@tclune reported that our UFS collaborators were seeing non-reproducibility when running with "recent" GOCART (v2.5.0-ish)

So, to test I built GEOSgcm v11 with GOCART v2.5.0. My first run failed because of a missing ExtData entry for AMIP emissions. This is fixed in #381 (and #382 for v2.6.0). Once I fixed this, I found that GOCART v2.5.0 passes layout regression but is failing start-stop regression. Because of this, I've drafted the v2.5.1 and v2.6.1 PRs since we need other fixes.

I am now building GEOSgcm v11 with GOCART v2.4.0 to see if that has the same regression failure. I will update that when I see.

---

cc @vbuchard