CMIP7 support in v2.14.0

[k-a-webb]

Is v2.14.0 meant to support CMIP7 CMORized datasets?

I am currently testing ESMValTool with preliminary CMIP7-CMORized CCCma model output, as a means to validate our CMORization procedure.

I have installed v2.14.0 by cloning the github repositories for both ESMValCore and ESMValTool (installing from source via mamba, per the instructions), using branches v2.14.0 for both.

Describe the bug

config_developer.yml

I think the config_developer.yml file has not been updated for the CMIP7 project?
https://github.com/ESMValGroup/ESMValCore/blob/v2.14.0/esmvalcore/config-developer.yml#L29

CMIP7:
  cmor_path: "cmip7"
  cmor_strict: true
  cmor_type: "CMIP6"

With the above, I get the following error:

...
    )
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/local.py", line 294, in _get_output_file
    preprocessor_filename_template: "{cfg["output_file"]}"
                                      ~~~^^^^^^^^^^^^^^^
KeyError: 'output_file'
2026-03-16 15:55:04,919 UTC [3494845] INFO    esmvalcore._main:805 
If you have a question or need help, please start a new discussion on https://github.com/ESMValGroup/ESMValTool/discussions
If you suspect this is a bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues
To make it easier to find out what the problem is, please consider attaching the files run/recipe_*.yml and run/main_log_debug.txt from the output directory.

If I modify this file, the recipe runs successfully.

CMIP7:
  cmor_path: "cmip7"
  cmor_strict: false
  cmor_type: "CMIP7"
  output_file: '{project}_{short_name}_{branding_suffix}_{frequency}_{region}_{grid}_{dataset}_{exp}_{ensemble}'

CMIP7 facets

What is the meaning of mip in the context of CMIP7 datasets? I had to include it in my recipe (partial example below) to avoid the errors listed below.

...
diagnostics:
  diagnostic:
    description: Time mean map - Ice Water Path (CMIP7.3hr.clivi.tavg-u-hxy-u)
    themes: []
    realms: []
    variables:
      clivi:
        preprocessor: spatial_pattern_clivi
        additional_datasets:
        - runid: cmip7-v51-ref
          project: CanESM_dev_CMIP7 # I've added this project to config-developer.yml the similarly as CMIP7
          subdir: rrd001/canesm_runs/cmip7-v51-ref/data/nc_output
          activity: CMIP
          institute: CCCma
          dataset: CanESM5-1
          exp: piControl
          ensemble: r1i1p1f1
          cmor_type: CMIP7
          version: v20190429
          dataset_ID: CanESM5-1 (cmip7-v51-ref)
          # timerange: P30Y/* # Do wildcard timeranges no longer work ???
          timerange: 6000/6000
          frequency: 3hr
          short_name: clivi
          branding_suffix: tavg-u-hxy-u
          mip: 3hr # Mandatory????
          region: glb
          grid: gn
      
    scripts:
      quickplot_diag:

Error message:

  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_recipe/recipe.py", line 92, in read_recipe_file
    return Recipe(raw_recipe, session, recipe_file=filename)
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_recipe/recipe.py", line 919, in __init__
    self.datasets = Dataset.from_recipe(recipe_file, session)
                    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/dataset.py", line 167, in from_recipe
    return datasets_from_recipe(recipe, session)
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_recipe/to_datasets.py", line 424, in datasets_from_recipe
    variable_datasets = _get_datasets_for_variable(
        recipe,
    ...<2 lines>...
        session=session,
    )
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_recipe/to_datasets.py", line 388, in _get_datasets_for_variable
    for facets, supplementaries in _get_facets_from_recipe(
                                   ~~~~~~~~~~~~~~~~~~~~~~~^
        recipe,
        ^^^^^^^
    ...<2 lines>...
        session=session,
        ^^^^^^^^^^^^^^^^
    ):
    ^
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_recipe/to_datasets.py", line 362, in _get_facets_from_recipe
    yield _get_dataset_facets_from_recipe(
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        variable_group=variable_group,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
        session=session,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_recipe/to_datasets.py", line 313, in _get_dataset_facets_from_recipe
    check.variable(
    ~~~~~~~~~~~~~~^
        facets,
        ^^^^^^^
    ...<7 lines>...
        variable_group=variable_group,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_recipe/check.py", line 184, in variable
    raise RecipeError(msg)
esmvalcore.exceptions.RecipeError: Missing keys {'mip'} in

i.e., it failed here:
https://github.com/ESMValGroup/ESMValCore/blob/v2.14.0/esmvalcore/_recipe/check.py#L170
because mip is specified as a mandatory facet here:
https://github.com/ESMValGroup/ESMValCore/blob/v2.14.0/esmvalcore/_recipe/to_datasets.py#L312

[k-a-webb]

Relevant to my question about mip, #2970

Edit: Which is to say, for CMIP7 tables realm is equivalent to the CMIP6 mip. Including mip as a facet for my dataset, with value of the realm attribute, seemed to work.

[k-a-webb]

While I've since found the changelog notes about the deprecation of output_file in config-developer.yml, in favour of preprocessor_filename_template being defined in config/configurations/defaults/preprocessor_filename_template.yml, this perhaps does not work as intended as the esmvalcore/local.py script remains to look for output_file explicitly?

I found the note about the deprecation of esmvalcore.local, https://docs.esmvaltool.org/projects/ESMValCore/en/latest/changelog.html#deprecations, but it appears to still be in use. Here's a full Traceback of the error included in the first message in this thread,

Traceback (most recent call last):
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_main.py", line 786, in run
    fire.Fire(ESMValTool())
    ~~~~~~~~~^^^^^^^^^^^^^^
  File "/fs/site8/eccc/crd/cccma/users/scrd115/envs/a4d_env_esmvalv214/lib/python3.13/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/fs/site8/eccc/crd/cccma/users/scrd115/envs/a4d_env_esmvalv214/lib/python3.13/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ~~~~~~~~~~~~~~~~~~~^
        component,
        ^^^^^^^^^^
    ...<2 lines>...
        treatment='class' if is_class else 'routine',
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        target=component.__name__)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fs/site8/eccc/crd/cccma/users/scrd115/envs/a4d_env_esmvalv214/lib/python3.13/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_main.py", line 620, in run
    self._run(recipe, session, cli_config_dir)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_main.py", line 675, in _run
    process_recipe(recipe_file=recipe, session=session)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_main.py", line 153, in process_recipe
    recipe = read_recipe_file(recipe_file, session)
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_recipe/recipe.py", line 92, in read_recipe_file
    return Recipe(raw_recipe, session, recipe_file=filename)
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_recipe/recipe.py", line 927, in __init__
    self.tasks = self.initialize_tasks()
                 ~~~~~~~~~~~~~~~~~~~~~^^
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_recipe/recipe.py", line 1302, in initialize_tasks
    tasks = self._create_tasks()
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_recipe/recipe.py", line 1274, in _create_tasks
    new_tasks, failed = self._create_preprocessor_tasks(
                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        diagnostic_name,
        ^^^^^^^^^^^^^^^^
    ...<2 lines>...
        any_diag_script_is_run,
        ^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_recipe/recipe.py", line 1236, in _create_preprocessor_tasks
    task = _get_preprocessor_task(
        datasets=list(datasets),
        profiles=self._preprocessors,
        task_name=task_name,
    )
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_recipe/recipe.py", line 850, in _get_preprocessor_task
    products = _get_preprocessor_products(
        datasets=datasets,
    ...<2 lines>...
        name=task_name,
    )
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/_recipe/recipe.py", line 686, in _get_preprocessor_products
    filename = _get_preprocessor_filename(dataset)
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/preprocessor/__init__.py", line 302, in _get_preprocessor_filename
    return _get_output_file(
        dataset.facets,
        dataset.session.preproc_dir,
    )
  File "/fs/site7/eccc/crd/cccma/users/scrd115/A4D/A4D_validation/A4D_standard_diagnostics__ESMVal214/esmvalcore/local.py", line 294, in _get_output_file
    preprocessor_filename_template: "{cfg["output_file"]}"
                                      ~~~^^^^^^^^^^^^^^^
KeyError: 'output_file'

[bouweandela]

Is v2.14.0 meant to support CMIP7 CMORized datasets?

Yes, great that you're giving it a try! 🥇

What is the meaning of mip in the context of CMIP7 datasets?

As you already found out, mip is the same as realm for CMIP7. In general, mip in ESMValCore is table_id in CMOR/on ESGF. People have complained in the past that this is confusing #834, but we never got around to changing it. Maybe we should put some effort into that. Please reopen the issue if you have an opinion. You can find the complete mapping from the names used in ESMValCore to the names used on ESGF here, but it looks like I forgot to add mip for CMIP7. I will open a pull request to update it.

Regarding config-developer.yml

Indeed, config-developer.yml and the related functionality (including that in esmvalcore.local) have been deprecated in v2.14 and will be removed in v2.16. I would recommend immediately switching over to the new way of configuring the tool, as it has the following advantages: 1) it is more user-friendly, 2) you won't have to change your configuration again when v2.16 comes around, and 3) we will fix bugs if you find them. You can do this by making sure there is no config_developer_file entry in your configuration.

How to configure a new project

First off, there are now three useful commands available that make it easier to work with configuration files:

• esmvaltool config show: this shows your current configuration
• esmvaltool config list: this shows a list of built-in configuration files that you can copy and tailor to your needs
• esmvaltool config copy: this command can be used to copy one of the built-in configuration files to your configuration directory (~/.config/esmvaltool by default or can be specified using the --config-dir command line argument or the ESMVALTOOL_CONFIG_DIR environment variable)

To set up a (new) project, there are four things that you can configure:

1. The minimum you need to do, is set up a data source for your project so that ESMValCore knows how to find and load your data. Documentation on how to configure data sources is available here. For your specific case, I would recommend running the command esmvaltool config copy data-local.yml, copying the CMIP7 entry to a new entry called CanESM_dev_CMIP7 and adjusting the rootpath, dirname_template, and filename_template to the way you have organized your data.
2. If you want CMOR checks and fixes, and automatically adding facets such as long_name from the CMOR table to work, you will need to configure a CMOR table for your project. Documentation is available here. For your specific case, I think you would need to create an entry like:

   projects:
     CanESM_dev_CMIP7:
       cmor_table:
         paths:
         - cmip7/tables
         - cmip6-custom
         type: esmvalcore.cmor.table.CMIP6Info

   this will use the tables from the directory esmvalcore/cmor/tables of your installed ESMValCore. If you want to use other tables, you can replace the values under paths or add an extra path if you want to extend the built-in tables.
3. Optionally, if you want control over what the preprocessor output files are called, you can configure that as described here.
4. Optionally, you can automatically augment the facets from the recipe / esmvalcore.dataset.Dataset with additional facets as described here.

[k-a-webb]

Thank you so much for taking the time to get me up an running with the new release.
And apologies that I did not more carefully review the latest version of the documentation, nor do more testing prior to posting an issue.

Using esmvaltool config show I was able to resolve the issue with my configuration files (I am using the configuration file organization suggested for v2.14+) where now it does not look for config-developer.yml. I did not pin down the exact reason why it had defaulted to that file originally, however. This is such a handy tool!

I'll admit that the confusion of mip/realm tripped me up, so I appreciate that you're adding it to the documentation.

Overall the transition from v2.11 has been smooth, so thank you (again!) for all your efforts with this release!

[k-a-webb]

Just wanted to make a note here, in case anyone else encountered this issue when adding new projects.

I created a project "CanESM_dev_CMIP7" which is similar to the "CMIP7" project except for dirname_template and filename_template (because the dataset facet is not consistently generated on different development branches). I added project sections data, extra_facets, and preprocessor_filename_template in my config.

I was running into an issue where the automatic-attachment of supplementary files was not working, it seemed to drop the frequency field.
I eventually tracked this down to the fact that the FACETS dictionary in esmvalcore.io.esgf.facets.py are keyed off of the project name (and hence did not include my custom project). Adding new entries to the dictionary solved the issue with automatic-finding of supplementary variables.

Perhaps a better solution would be to include a field, cmor_type or mip_era in each project, which indicates the CMOR standard, which is instead picked up by FACETS?

[bouweandela]

Thanks for reporting that, indeed the facets used for automatically finding supplementary variables are not configurable yet. I've opened #3031 to describe the issue and a potential solution.

In your case, the simplest solution may be to add a LocalDataSource that can find your CMIP7-like data to the CMIP7 project.

[k-a-webb]

In your case, the simplest solution may be to add a LocalDataSource that can find your CMIP7-like data to the CMIP7 project.

This is a great suggestion, but in my case, because the facets in dirname_template are different for the different LocalDataSource objects, it gets awkward. I found that I couldn't project just the requested facets for "LocalDataSourceA", but had to provide all the facets (even if null) for all LocalDataSource's. It had seemed more straight-forward to create a separate project.

[bouweandela]

That should be solved by #3021