Add an interface for adding new data sources and add support for intake-esgf as a first example

[bouweandela]

Description

Add an interface for adding new data sources. Documentation of the new interface is available here: esmvalcore.io.

The existing esmvalcore.local and esmvalcore.esgf modules have been modified to make use of the new interface and as an example use case, support for using intake-esgf to find input data has been added.

Several commands have been added:

• esmvaltool config show: print the current configuration
• esmvaltool config list: list available example configuration files
• esmvaltool config copy: copy an example configuration file to your configuration directory, i.e. ~/.config/esmvaltool or the path defined by the ESMVALTOOL_CONFIG_DIR environment variable.

To try the new intake-esgf data source, configure esmvaltool to use it by running the command esmvalcore config copy intake-esgf-data.yml.

Related to #2584

Contains changes to esmvalcore.local.DataSource that are not backwards compatible.

Link to documentation:

Follow up ideas:

• Add descriptions to the example configuration files for displaying in the command esmvaltool config list
• Improve validation of the data source configuration
• Move the modules esmvalcore.esgf and esmvalcore.local into esmvalcore.io. To avoid introducing even more changes in the pull request, I will do this in a follow up pull request.
• Make the fixes module configurable per data source
• Add a site configuration setting that selects defaults appropriate to that site, e.g. site: levante would select data sources and dask settings appropriate to Levante, site: jasmin for Jasmin, to simplify configuration of the tool Add a site option to the get_config_user command #1706

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[valeriupredoi]

I'll work with you on this one @bouweandela 🍺

[valeriupredoi]

having a dive in this, bud - let me know how I can help!

[valeriupredoi]

I wonder if we want to see if we can first convert it to a DataSource before we toss it out the window

[valeriupredoi]

this is an excellent addition - we are finally abstracting a data object that gets ingested by esmvalcore, and we generalize it: let's be careful how we implement this so it can be reused with little fuss for the future: I'd argue that "data that can be loaded" can be anything ie the most generic file object (not needing to be on disk, nor it needing it to be downloaded), so we can operate with object stores too

[valeriupredoi]

this one here ties in very well with this PR, bud #2785 - enjoy your time off 🏖️

[valeriupredoi]

hey @bouweandela hope you're enjoying your holiday time! I kept myself busy and we now have Zarr support (in _io.load) and have done other improvements, hence the conflicts with main, let me fix those for you now. Alas, you can now pass an Intake catalog via this PR, and if that has Zarr files in S3 buckets, then we can load them and test this one 😃

[codecov]

Codecov Report

❌ Patch coverage is 93.78428% with 34 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.37%. Comparing base (459587d) to head (5fbe25c).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
esmvalcore/_main.py	20.00%	28 Missing ⚠️
esmvalcore/local.py	96.34%	3 Missing ⚠️
esmvalcore/config/_data_sources.py	92.30%	2 Missing ⚠️
esmvalcore/io/intake_esgf.py	98.95%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2765      +/-   ##
==========================================
- Coverage   95.46%   95.37%   -0.09%     
==========================================
  Files         260      264       +4     
  Lines       15519    15862     +343     
==========================================
+ Hits        14815    15129     +314     
- Misses        704      733      +29     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.