Post-code freeze for v2.8 Technical Lead Team (TLT) meeting

[valeriupredoi]

attn: @ESMValGroup/technical-lead-development-team

@remi-kazeroni and myself will organize a TLT post-freeze discuss the upcoming release, and for that Remi has already put together a summary discussion point here ESMValGroup/ESMValCore#1924

We would ideally meet in the following intervals (once, not twice):

• February 23-24
• February 27-28

I will set up a Doodley-doodle but before I do that does any of you fellas have any conflicts for those days (vacation, visit from mother-in-law etc)? Respond with 👍 to this item if you can make it in principle around those dates, and with 👎 if you can't. Cheers!

[valeriupredoi]

4/7 👍 's - good start - @sloosvel @zklaus @bsolino what say you guys? ⬆️

[valeriupredoi]

arite peeps @ESMValGroup/technical-lead-development-team cheers for your rections, I have now created a poll for this meeting, see https://doodle.com/meeting/participate/id/b28llZKe - and you should have received that as an email as well, but double posting just in case - please fill that out the sooner the better, just in case the best date is this Thu or Fri 🍺

[bouweandela]

Can we also have a look at ESMValGroup/ESMValTool#2998 in the meeting?

[valeriupredoi]

in the words of Barak H Obama - yes we can! 🇺🇸

[bouweandela]

And ESMValGroup/ESMValTool#3032

[valeriupredoi]

yay that's a stuffy one! Let's do that - am totally agreeing with @remi-kazeroni to get that shipped when it's all shipshape, gonna need a bit of discussion

[remi-kazeroni]

I went through #3032 again and would suggest to discuss the following items:

1. Data directories: which ones to use? Only the "official" paths or also the previously downloaded data. Note that the DKRZ CMIP5 pool is no longer maintain and lots of datasets used in our recipes need to be downloaded from ESGF.
2. Submission scripts: shall we wait for the recipe test workflow, port the Cylc suite v8 to Levante (see Port cylc suite to cylc version 8 ESMValTool#2722) or rely on submission scripts from Add DKRZ/Levante batch scripts for release recipe running ESMValTool#2883?
3. Who should check the recipe output when the comparison tool returns "Human inspection needed"? Should it be the RM or the maintainers of recipes?
4. Publishing the lists of successful and failed recipes on GitHub: via issues or discussions? Let's try to harmonize that a bit, if possible.

[remi-kazeroni]

Administration of the VM

This topic was already on the agenda of the past few meetings. We've never had a chance to discuss that. Given that the release involves the usage of the VM, this meeting could be a good time for that. See #62 (comment)

Main items to be discussed:

• Could we delete intermediate recipe results not produced with the last release candidate?
• Could we only keep the results of the final run (success or failure) for each recipe? Do we need to write a script to help the Release Manager cleaning that up?
• Could we avoid uploading results from large work directories to the VM? Either by adding an option to the few problematic recipe not to generate large/many nc-files or with a script that we clean that up?
• We would need to improve the documentation on how to use the VM for a release. Readthedocs may not be the right place for that. Probably better to have a Readme in the VM.

[remi-kazeroni]

Follow-up from previous meetings

(I'm not sure that the right moment for that. The meeting will be mostly about v2.8 release. But let's not leave things from previous meetings behind...)

• a Gmail-based solution that can be used by some Core developers and/or the TLT (see Technical Lead Team rolling issues, shoutouts, and meeting deciders - Part 2, to be discussed at TLT Meeting 13 Dec 2022 #62 (comment)). Attn: @valeriupredoi
• Opening our Slack worksapce (see Technical Lead Team rolling issues, shoutouts, and meeting deciders - Part 1, to be discussed at TLT Meeting 12 Jan 2023 #68 (comment)): feedback from the User Engagement Team

[valeriupredoi]

we found a decent time for this show: TLT February 28, 2023 • 2-3 PM Europe (CEST) time - prepare yourselves it'll go overboard 🚢 (ie @sloosvel bring a hefty lunch 😁 )

[valeriupredoi]

We meet today at 2pm your times (am only 1pm GMT), here is the Zoom link
https://ncas.zoom.us/j/91735893028?pwd=K0hTcUprVCsyQ0hDRTEzcGxEME5hUT09
If the thing asks you for a password that is 491994

[remi-kazeroni]

Update on the release process

1. Core milestone v2.8 about to be completed. If needed let's briefly talk about:

• Combined offline and always_search_esgf into a single option search_esgf ESMValCore#1935

2. Feature freeze for the Core, creation of the Changelog: tomorrow
3. First release candidate and first round of recipe: before the end of the week.
   The idea with this first round is to check how many Core/diagnostics/data missing issues we have.

[remi-kazeroni]

Agenda - Feb 27, 2023

1. Action items from previous meeting
2. Update on the release process
3. Detailed instructions for the release procedure, including:
   • Data directories to be used for the recipe testing
   • Submission scripts/tools
   • Who should check the recipe output after the comparison tool was run?
   • Report on testing results in issues or discussions?
4. Change esmvalcore pin in environment and setup.py files
5. Administration of the VM
6. AOB

This list contains links to all suggested topics for this meeting. We can reorder the list at the beginning of the meeting.

[remi-kazeroni]

Points 4 and 5 to be discussed at a next meeting

[valeriupredoi]

• Bouwe: we should talk about what feature freeze etc - pre-release instructions
• Remi: talked about opening up Slack with the User Eng. Team: they recommended not opening it as a Helpdesk
  Bouwe thinks they got it wrong - Slack should be open for communication not as a Helpdesk
  Saskia: thinks it will eventually become a Helpdesk
  Remi: postpone discussion to workshop, a f2f
• Remi: Release schedule:
  Manu: change default for esgf download option (replace offline) - only problem is the Deprecation warning is printed twice
  V: issue with a few recipes prob bc of CanESM CCCMA crappe
• Remi: Core feature freeze tomorrow(ish) (March 1st 2023)
  Tool (targeted) feature freeze March 8
  gonna rename the Discussion item to Release something something (Bouwe suggested following iris procedures)

Release documentation procedure

• Remi summarizing the procedure:
  • (Core) feature freeze -> no more changes apart from bugfixes
  • merge this in main -> RC -> test by running recipes -> if issues, bugfixes in core -> another RC -> ... -> final Core RC
  • (Tool) feature freeze -> final test all recipes ->
  • final Core release from last Core RC, final release for Tool at the same time
  • Important to keep the RM in the loop avoid major changes that diverge branches

Release testing procedure (Tool)

• (ad-hoc) Data directories: issues with missing CMIP5 data at DKRZ -> centralize data dirs at DKRZ distill to two at most dirs
• Remi: handling of broken recipes: we should keep track of them and don't gloss over them every release
• Bouwe: add a table to the docs listing the broken recipes -> Remi: add criteria by which we list the broken ones (Core, data, diagnostic etc)
• how to run the recipes: ad-hoc scripts written by Saskia, used for 2.6 and 2.7 -> issues from greatly differing resources needed
• ideally we should have a single end-to-end workflow -> Bouwe suggesting dedicated workflow writers -> good idea, but we need discuss, maybe with the Met Office, not to duplicate MO work (which is currently suspended, anyway)
• maybe less use of Levante - although JASMIN is unreliable, and data is most plentiful on Levante
• possible come back to Cylc (8.0) but need to get in touch with Met Office
• Manu: add resource info in the recipe's header - rough numbers to give an idea about what it needs - Bouwe: maybe add that to the Debug Table - plop it in the documentation every latest/stable release

Final temp solution: for now, use current ad-hoc solution - Bouwe suggests we merge those PRs that contain these scripts.

Examining recipe runs results:

• who checks the recipe run output: ask support from the Science Team and Devs, allow for a few days for this to happen
• how to check: RM checks via a quick look - much more important for the recipe devs to look at output

esmvalcore pin in Tool is not stringent