Allow grouping input cubes by date (instead of filename) for fix_metadata

[schlunma]

Description

This PR allows grouping the input cubes for our fix_metadata functions by date, i.e., all files with the same data range are passed to the fix simultaneously). This can be enabled by setting the class variable GROUP_CUBES_BY_DATE = True in the corresponding fix class. This allows implementing fixes where variables from multiple input files are necessary (for example, to derive rsut for ERA5).

This solution only works for projects where the input files are located in the same directory, and the input file pattern is flexible enough to find all files. This is fine for the native ERA5 data in netCDF format (that we need to manually download and put into the corresponding directories). For other projects where files are stored in different directories, further changes are necessary (potentially in local.py). However, this PR is a prerequisite to make these other cases work.

By default, input cubes are grouped by filename for fix_metadata (i.e., each fix_metadata call operates only on a single file):

ESMValCore/esmvalcore/cmor/fix.py

Lines 197 to 204 in fd82b43

	by_file = defaultdict(list)
	for cube in cubes:
	by_file[cube.attributes.get("source_file", "")].append(cube)
	for cube_list in by_file.values():
	cube_list = CubeList(cube_list)
	for fix in fixes:
	cube_list = fix.fix_metadata(cube_list)

Note that this is fully backwards-compatible since the new functionality needs to be explicitly enabled.

Closes #1806

Link to documentation: TBA

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[schlunma]

Here's a small recipe to test this for rsut (I also included rsdt to ensure that this still works fine):

# ESMValTool
---
documentation:
  title: test
  description: test
  authors:
    - schlund_manuel

datasets:
  - {project: native6, dataset: ERA5, type: reanaly, version: v1, tier: 3, timerange: 2000/2001}

diagnostics:

  test:
    variables:
      rsut:
        mip: Amon
      rsdt:
        mip: Amon
    scripts:
      null

Input files need to be arranged like this:

.
└── Tier3
    └── ERA5
        └── v1
            └── mon
                ├── rsdt
                │   ├── era5_toa_incident_solar_radiation_2000_monthly.nc
                │   └── era5_toa_incident_solar_radiation_2001_monthly.nc
                └── rsut
                    ├── era5_mean_top_net_short_wave_radiation_flux_2000_monthly.nc
                    ├── era5_mean_top_net_short_wave_radiation_flux_2001_monthly.nc
                    ├── era5_toa_incident_solar_radiation_2000_monthly.nc
                    └── era5_toa_incident_solar_radiation_2001_monthly.nc

@bouweandela do you think this approach is a reasonable solution to this problem? As mentioned in the description, it doesn't solve the problem for all cases, but a different grouping will be necessary for all of them. And it is fully sufficient for the ERA5 netCDF case.

[codecov]

Codecov Report

Attention: Patch coverage is 57.14286% with 12 lines in your changes missing coverage. Please review.

Project coverage is 95.12%. Comparing base (0dc146c) to head (cabc32c).

Files with missing lines	Patch %	Lines
esmvalcore/cmor/_fixes/native6/era5.py	36.36%	7 Missing ⚠️
esmvalcore/cmor/fix.py	66.66%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2551      +/-   ##
==========================================
- Coverage   95.20%   95.12%   -0.08%     
==========================================
  Files         259      259              
  Lines       15211    15232      +21     
==========================================
+ Hits        14481    14489       +8     
- Misses        730      743      +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bouweandela]

It may be nicer to define the grouping operation on the Fix object, so this code would look like:

    fixed_cubes = CubeList(cubes)
    for fix in fixes:
        fixed_cubes = CubeList(
            cube
            for group in fix.group_input_for_fix_metadata(fixed_cubes)
            for cube in fix.fix_metadata(group)
        )

[bouweandela]

do you think this approach is a reasonable solution to this problem?

If it works for you, it should be fine