RTW configuration issues

[ehogan]

Following my investigation related to ESMValGroup/ESMValCore#2738, I discovered that some of the compare tasks in the RTW were failing.

If what I wrote at ESMValGroup/ESMValCore#2736 (comment) is correct, the reason for the failing compare tasks is that the ~/.esmvaltool/user-config.yml configuration file is used in favour of the one we provide in the RTW via --config_dir. This means the recipe outputs were written to an unexpected output_dir (the one specified in the ~/.esmvaltool/user-config.yml configuration file). When the compare tasks ran, the logs stated:

Comparing recipe run(s) in:
~/cylc-run/recipe_test_workflow/run13/share/cycle/20250530T1426Z/recipe_ocean_amoc_????????_??????
to reference in /data/users/esmval/KGO/recipe_ocean_amoc_20240620_101604

Summary
=======

All 0 recipe runs were identical.

and did not raise an error, meaning the compare task passed.

If I remove the ~/.esmvaltool/user-config.yml configuration file, the recipe outputs are written to the expected location, and the compare script shows differences.

I'm not sure how long this has been happening for 😭 (Edited to add that I just noticed the KGOs on DKRZ are pointing to outputs generated from the v2.11.0 release, so possibly almost a year?)

I believe there are two issues to resolve:

• The compare script must raise an error when one (or more!) of the directories are missing (this should be resolved following Add compare command, via new develop command group #4054 and Find compare.py more robustly within container #3437)
• The KGOs will need verifying (I don't know how long this issue has been happening for, so it will take effort to determine the cause of the change, and it might just be quicker to verify the KGOs.)

Perhaps now would be a good time to prototype the Zenodo KGO approach? 🤔