Technical Lead Team rolling issues, shoutouts, and meeting deciders - Part 3, to be discussed at TLT Meeting 16 Apr 2023

[valeriupredoi]

This is the third installment of our popular series in which we gather around the Zoom camp fire, and yap about technical matters. We had the eponymous and imaginatively named Part 2 and we had a few TLTs as well, with minutes from the latest but @remi-kazeroni alerted me, and for good measure too, that it's about time we discuss techny stuff again in such a Discussion point, and also, I will schedule us another TLT Call very soon - am waiting for the v2.8 to be out the way (cheers, Remi 🍺 ) so we can talk on the back of it, but by all means, please start adding items here, @ESMValGroup/technical-lead-development-team

[remi-kazeroni]

Generation of conda-lock files

• Do we need to generate them more frequently and do we need to assist @valeriupredoi in this task (cheers for taking care of that 🍺)?
• Do these files need special attention during release processes when we switch conda channel and esmvalcore versions quite frequently? See Recreate conda lock file to harpoon esmvalcore=2.8.0rc1 ESMValTool#3108

[remi-kazeroni]

@valeriupredoi - do we still need to discuss something for this topic or are we good for now?

[valeriupredoi]

no, we should be good for now, if things hit the fan, I can always recreate these files manually, and if they're a lil bit out of date it's not the end of the world as we know it 😃

[remi-kazeroni]

Changelog (for the Core)

• We may need to revise our approach so the changelog generation is not left to the RM as a last-minute work. See Changelog for v2.8.0rc2 ESMValCore#1959 as an illustration of this
• Shall we make it the responsibility of the PR author to provide some text for the changelog? This is particularly important to document backward incompatibility and deprecations.
• Handling of the label: shall we relax the 1 label / PR policy (see Change to github handling: clarify label use ESMValCore#1365)?
  • This makes it more complicated to write information about deprecations and backward incompatibilities
  • Several labels are often used for PRs, choice of section is left to the RM with too many PRs to categorize at the last moment
• Common practices for usage of labels when deprecated features
• Shall we split the "Bugs" section into "Released bugs" and "Development bugs" (with better names)

See also ESMValGroup/ESMValCore#1951

[remi-kazeroni]

Change esmvalcore pin in environment and setup.py files

(from last meeting)

Can we also have a look at ESMValGroup/ESMValTool#2998 in the meeting?

Originally posted by @bouweandela in #79 (comment)

[remi-kazeroni]

Is this discussion topic still relevant here? Or did we address everything when doing release v2.8 and adding documentation about pinning the Core in https://docs.esmvaltool.org/en/latest/community/release_strategy/release_strategy.html#detailed-timeline-steps?

[bouweandela]

We can talk about it at the workshop.

[remi-kazeroni]

Administration of the VM

(from last meeting)

This topic was already on the agenda of the past few meetings. We've never had a chance to discuss that. Given that the release involves the usage of the VM, this meeting could be a good time for that. See #62 (comment)

Main items to be discussed:

• Could we delete intermediate recipe results not produced with the last release candidate?
• Could we only keep the results of the final run (success or failure) for each recipe? Do we need to write a script to help the Release Manager cleaning that up?
• Could we avoid uploading results from large work directories to the VM? Either by adding an option to the few problematic recipe not to generate large/many nc-files or with a script that we clean that up?
• We would need to improve the documentation on how to use the VM for a release. Readthedocs may not be the right place for that. Probably better to have a Readme in the VM.

[remi-kazeroni]

I think this is pretty much addressed and was documented in https://docs.esmvaltool.org/en/latest/community/release_strategy/detailed_release_procedure.html#share-the-results-with-the-community for release v2.8.0. I would suggest not to discuss about this now and leave it for the next release should something not be documented well enough.

[bouweandela]

Only this point seems not addressed

Could we avoid uploading results from large work directories to the VM? Either by adding an option to the few problematic recipe not to generate large/many nc-files or with a script that we clean that up?

Would it be an idea to open issues in ESMValTool about the individual recipes that cause problems?

[remi-kazeroni]

Done in ESMValGroup/ESMValTool#3183

[remi-kazeroni]

Follow-up from previous meetings

• a Gmail-based solution that can be used by some Core developers and/or the TLT (see Technical Lead Team rolling issues, shoutouts, and meeting deciders - Part 2, to be discussed at TLT Meeting 13 Dec 2022 #62 (comment)). Attn: @valeriupredoi

[valeriupredoi]

we shall find out tomorrow, it's a surprise 🤣

[valeriupredoi]

ah but why wait for a surprise (truth is, I forgot again 😆 ) - it looks like it's possible to set up a shared Gmail box https://www.helpscout.com/blog/shared-inbox-gmail/

[valeriupredoi]

Collaborative Inbox looks to be the right way - am not 100% sure if that's free though, will check

[valeriupredoi]

needs to set up a Google Group, see how that's done https://support.google.com/groups/answer/2464926?hl=en

[valeriupredoi]

OK too much research, need 🍕

[remi-kazeroni]

Backward incompatible changes in OBS directories

I'd be curious to hear opinions about how to handle changes made to our OBS directories. This could be the case if we decide to move some datasets from Tier3 to Tier2 or if we decide to read facets from directories, instead of filenames (see ESMValGroup/ESMValCore#1944), rename datasets, change metadata, ...
Such changes may help us improve the handling of OBS in ESMValTool but may require adaptation for our users / maintainers of data pools. This is manageable on DKRZ and Jasmin but I'm less sure about data pools elsewhere. Would there be ways to make such changes without requiring many changes on the user side?

[remi-kazeroni]

Let's forget about this for now. It would make more sense to discuss this during the session on OBS data that Ranjini and myself would like to have at the next ESMValTool workshop.

[remi-kazeroni]

Add a site option to the get_config_user command

I would like to discuss ESMValGroup/ESMValCore#1706.
It seems we all have different opinions on how to improve the configuration. My experience after teaching quite a few tutorials is that commenting out the wrong line or making a space typo in the config-file leads to all kinds of weird errors not so easy to debug live during a training. A lot of time can be wasted doing that during trainings and this does not necessarily gives a good first impression of the Tool.

We have 1 line commands to install the tool (module load x, conda create x), to run recipes and even download data on-the-fly (esmvaltool run x) but no such thing for the config file. I think we should really try to better support the automatic generation of config files for all machines listed in the config-user.yml.

[remi-kazeroni]

Next ESMValTool release (v2.9.0)

Last (?) from me, but not least (!): we will need a release manager for the next release. The next Core feature freeze is already scheduled in 2 months from now, see schedule. Since the release of v2.8 was delayed by about one month to allow the integration and testing of major changes (some of them needed before the closure of IS-ENES3), I really wonder if it makes sense to keep the next release so close to v2.8. But on the positive side, we have clarified the release process and documentation, and we have nice scripts to run all recipes and parse all output (errors) in one go. That should help next RMs to save time with the testing 😅

[valeriupredoi]

whoa - @remi-kazeroni decided that we are a bit short on topics so he'd take care of that 😆 Good stuff! We have plenty to talk about then - shall we aim at a call next week before General Jesus H. Christ decides it's time to go back to Headquarters ie Easter strikes? @ESMValGroup/technical-lead-development-team react with 👍 if you want to meet next week (3-6 April) or with 👎 if you want a call after Easter (10-13 April)

[remi-kazeroni]

Yes, sorry about the number of posts 😅 But not all are mine as I also reported stuff from issues / PRs that we said we would take to the next TLT.
I think it would be helpful if the members of the @ESMValGroup/technical-lead-development-team upvote (👍) the topics that they want to discuss in priority. This would help making an agenda.

[bouweandela]

Thanks for keeping the agenda @remi-kazeroni, this is very useful.

[remi-kazeroni]

Recording known good output

During recipe testing for v2.8, it was not clear if all recipe output should be inspected by a maintainer or developer. The comparison tool reported that 120 recipes should be checked manually. This high number is possibly due to some changes and bugfixes in the Core. See ESMValGroup/ESMValTool#3127. It may not be feasible to have all recipe output checked after each release. But what happens if some recipe output are affected but unknown Core/diag issues which silently pass from release to release until someone does check the plots. That is how an issue with masking of 0s was eventually noticed.

• Could we define a robust procedure to record known good output and fully rely on the comparison tool if it returns "nothing has changed since the past release for recipe_abc"?
• How often should we look at recipe output? Once per year?

[bouweandela]

It would be great to create a list of recipes that were checked for the last release and say those are 'known good output'. Then we know that there is no need to check those again for the upcoming releases unless the comparison tool says they changed.

[zklaus]

Readthedocs kinda broke, noticed by @sloosvel on 2023-04-05. The issue was that the project part of the URL turned case sensitive, which broke the links in the side-bar. I fixed it by putting a redirect into our readthedocs config.

I have no idea what changed; if anyone can share some light on this that would be great, but in any case let's talk about this and decide on a permanent solution.

[bouweandela]

Next release

The next release is currently scheduled for June, but we still need a release manager for it. I would also be interested in seeing if we can potentially postpone the release a bit in the light of SciTools/iris#5191 (comment).

[bouweandela]

Postponing our release may not be necessary anymore since the iris 3.6 release was moved to an earlier date so is now well on time for our release.

[schlunma]

Make DRS path templates configurable per rootpath

I really would like to discuss ESMValGroup/ESMValCore#1894 on Tuesday. Being able to define DRS path templates per rootpath would be super helpful!

Example:

download_dir: /work/bd0854/DATA/ESMValTool2/download
rootpath:
  CMIP6:
    /work/bd0854/DATA/ESMValTool2/CMIP6_DKRZ: DKRZ
    /work/bd0854/DATA/ESMValTool2/download: ESGF

[remi-kazeroni]

Bugfix release (v2.8.1) to unlock the use of Python 3.11 in ESMValCore

Support for Python 3.11 was added in ESMValGroup/ESMValCore#1832 but a new release of ESMValCore is necessary to enable its use. Options are:

1. A bugfix release (v2.8.1) with minimal changes just for Python 3.11
2. A bugfix release (v2.8.1) with all PRs that have been merged since v2.8.0 release
3. Wait until the release of v2.9.0 (see also this post).

Additionally, if a bugfix release option is chosen, shall we test all recipes and to which extent?

[bouweandela]

The advantage of doing a bugfix release with Python 3.11 support is that we can already start working with Python 3.11 for doing development work on ESMValTool v2.9, so there are likely fewer last-minute surprises when it is time to make the release.

The advantage of including the other bug fixes is that users can already benefit from them now instead of having to wait another two months.

Additionally, if a bugfix release option is chosen, shall we test all recipes and to which extent?

My recommendation would be only to run those recipes that actually use the small amount code that has changed in ESMValCore since the v2.8 release.

[remi-kazeroni]

Agenda - May 16, 2023

1. Short round of updates
2. Status of the next release (timeline, manager, Iris, ...)
3. Bugfix release for Python3.11?
4. New policy to generate Core Changelog?
5. Site option for the configuration
6. Make DRS path templates configurable per rootpath
7. Recording good known output
8. Follow-up from previous meetings
9. Change esmvalcore pin in environment and setup.py files
10. Readthedocs issues

I have ordered the items by number of upvotes. It could be that some topics are no longer relevant for discussion and should be removed. Let me know. Also, some discussion items may only be worth discussing if we have the resources to tackle the problems...

[remi-kazeroni]

Topics deferred to the next workshop

• Configuration of the Tool:
  5. Site option for the configuration
  6. Make DRS path templates configurable per rootpath
• Environment issues:
  9. Change esmvalcore pin in environment and setup.py files

Attn: @zklaus, organizer of the next workshop 🍻

[remi-kazeroni]

Topics to be discussed at the new TLT call

4. New policy to generate Core Changelog?
5. Recording good known output
6. Gmail-based solution
7. Readthedocs issues

Attn: @valeriupredoi 🍻

[remi-kazeroni]

@ESMValGroup/technical-lead-development-team, here is our agenda from last time. Just reviving this as a starting point to make an agenda for our meeting tomorrow. I guess we should start with the upcoming v2.9 release.

[valeriupredoi]

Minutes

ESMValTool Technical Lead Team Call 16 May 2023

1. Quick round of updates

• Remi: applied to get an RSE at DLR: they'll start in June for 6 months (for Daskification)
  • Bouwe, Saskia, Klaus and V are interested indiscussing with them
• Bouwe: availability about 1 day/week for the rest of 2023, no funding for 2024; funding ran out, sadly;
  • also about a retrospective meeting (maybe at SMHI? if everyone's there: Emma will be remote)
  • another thing he forgot, Bouwe's old 😀
• Saskia: been working on the lazy regridder (not yet lazy, in the process - help from Iris folk)
• Klaus: ISENES3 reporting; new RSE who's now involved in ESMValTool
  • netCDF/ESMF work on conda-forge (nompi will start working with ESMF soon)
• Emma: help with support; work on MO-internal CMEW (Climate Model Evaluation Workflow)
• V: tech stuff (Python 3.11 etc)
• Manu: ICON developments (issues detected in grids reading) - continuing with this

2. Topics:

v2.9 Release

• Remi: delayed 2.8 for valid reasons, iris=3.6 is coming out this week

• Bouwe: configuration of iris with DASK (at scheduler-level), feature out in iris=3.6 so it's good to have it in ESMValTool new release; possible good: discuss how to extend the configuration

• Klaus: supports a timely release even if it's a slim one; raising the fixed deps issue for a release, to preserve the environment at release point

• Saskia: autumn release is already very slim (ppl don't develop over summer), possibly no point for 2 slim releases

• Manu: most pressing issue atm is the RM: do we have anyone? Bouwe's a legend and he might do the June release; Klaus is a legend too - possible RM for the October one

• We do have the next v2.9 RM: Bouwe (Remi helping out)

• We do have the next v2.10 (v3.0 ?) RM: Klaus (Bouwe helping out)

• possibly discuss if "3.0" at the workshop

• Emma: willing to shadow a RM to learn, and pass the expertise to MO

Python 3.11

• Bouwe: a good idea to act now, to spot possible issues
• Manu: do we expect problems with python=3.11? - Bouwe: not so much with 3.11, but rather, with pins that
  restrict other packages (eg esmpy, cartopy etc); ideally the RTW would detect such issues;
• Remi: better a bugfix release that doesn't include all developments of Core (since 2.8 release); good: a barebones bugfix (just the bits that allow support for 3.11)
• Klaus: bugfix release in the truest sense - that just adds (bug) fixes, and we've done these before - that will fall in the right category
• Saskia: users' perspective: users don't really follow bugfix releases, a lot use older versions
• Klaus: possiblity do a "dev" Core release (the dev channel in conda) - detached from Tool using latest Core version
• Emma: severe Core bug at MO (location extractor not working due to peculiar SSL settings at MO - now fixed in Core main)
• Klaus: have in mind the central installations too: we need a reliable set of bugfix releases
• Bouwe: distinguish between bugfixes (bug+fixes) and dependencies pins/upgrades - the latter may cause a lot of problems, the former not
• Klaus: how about a 2.8.1rc0 that we use for testing recipes
• Emerging decision: a 2.8.1rc0 on Friday and Remi tests recipes next week

AOB

Update the v2.9 milestone table: https://github.com/ESMValGroup/ESMValCore/milestone/15

• Manu: give an indication of availability (for reviews): ask on Slack, remind ppl
• Klaus: a few PRs that are still hanging about that need to be finished before 2.9 release
  • PR in question: Make DRS path templates configurable per rootpath ESMValCore#1894
  • configuration issue: have a first look/agreement then have us a good discussion about it at the workshop
• Manu needs a review and merge of Introduce Fix.session to access Session in fixes ESMValCore#1988
• Klaus+Pep+Saskia: set up a meeting about CORDEX shtuff
• Remi: help get rid of outdated topics

[valeriupredoi]

agenda_TLT_16May.md

[remi-kazeroni]

Thanks a lot @valeriupredoi! That was super helpful to draft the minutes, see #102