Integrating Input4MIPs

[julienboussard]

We are a team working on causal ML and climate, and are willing to use ESMValTools for downloading and preprocessing climate datasets. The only limitation is that we need the data from Input4MIPs for our analysis.

Since ESMValTools already has the infrastructure to work with ESGF data, we were hoping that Input4MIPs data could be added to ESMValTools, and this would allow to study the impact of various forcings on the climate.

Would you be able to help out?

We have been in contact with @schlunma who suggested that I open an issue. I have some time and am willing to help out but I have never worked with ESMValTools and don't necessary have the skills to do this.

Thank you

[schlunma]

Thanks so much for reaching out @julienboussard! Sounds like a very exciting collaboration!

Here are some pointers that might help to implement support for the automatic download of input4MIPs data:

• To get started, you will need to set up a development installation of ESMValCore, see here for details.
• To be able to map "our" facets (e.g., dataset, exp), to ESGF facets (e.g., source_id, experiment_id), the FACETS dictionary needs to be extended.
• The input4MIPs CMOR tables will need to be copied to our table directory.
• An input4MIPs entry needs to be created in the config-developer file. This will probably be very similar to the obs4MIPs entry.

That might already be sufficient to download, read, and process input4MIPs data. @bouweandela is there any other major change that's necessary? Since I've never added support for a new project myself, I don't have much experience in this.

I will try to set up a running prototype before my leave which could be used as a starting point for this!

[bouweandela]

To get started with ESMValTool and ESMValCore, I would recommend taking a look at the tutorial: https://tutorial.esmvaltool.org

@bouweandela is there any other major change that's necessary?

I would also recommend adding an example recipe, perhaps copying examples/recipe_python.yml, that you can use to test and share with people to show how it works.

[julienboussard]

Thanks a lot for the pointers. I will look at all this to get started asap.
Keep me posted about the running prototype, I'll be more than happy to test it whenever you need me to.

[valeriupredoi]

@bouweandela and I can help while @schlunma is on 👶 leave 🍻

[schlunma]

I just opened a draft PR here: #2774

As expected, the implementation was rather straightforwards and already works for many datasets.

However, as mentioned in #2774, it seems that the input4MIPs CMOR tables are not complete and many (maybe most) variables and datasets are not listed in the tables. Currently, there's no way to read these missing data with ESMValCore.

Let's continue the discussion in the PR.