Move the esmvalcore.local and esmvalcore.esgf modules into the esmvalcore.io module

doc/api/esmvalcore.esgf.rst

@@ -5,6 +5,7 @@ esmvalcore.esgf
 ---------------
 .. automodule:: esmvalcore.esgf
    :no-inherited-members:
+   :no-index:
 
 esmvalcore.esgf.facets
 ----------------------

doc/api/esmvalcore.io.esgf.rst

@@ -0,0 +1,8 @@
+esmvalcore.io.esgf
+==================
+.. automodule:: esmvalcore.io.esgf
+   :no-inherited-members:
+
+esmvalcore.io.esgf.facets
+-------------------------
+.. automodule:: esmvalcore.io.esgf.facets

doc/api/esmvalcore.io.local.rst

@@ -0,0 +1,5 @@
+esmvalcore.io.local
+===================
+
+.. automodule:: esmvalcore.io.local
+    :no-inherited-members:

doc/api/esmvalcore.io.rst

@@ -7,12 +7,17 @@ In the future, this module may be extended with support for writing output data.
 The interface is defined in the :mod:`esmvalcore.io.protocol` module and
 the other modules here provide an implementation for a particular data source.
 
+esmvalcore.io
+-------------
+.. automodule:: esmvalcore.io
+
+Submodules
+``````````
+
 .. toctree::
    :maxdepth: 1
 
-   esmvalcore.io.protocol
+   esmvalcore.io.esgf
    esmvalcore.io.intake_esgf
-
-esmvalcore.io
--------------
-.. automodule:: esmvalcore.io
+   esmvalcore.io.local
+   esmvalcore.io.protocol

doc/quickstart/configure.rst

@@ -732,8 +732,8 @@ There are three modules available as part of ESMValCore that provide data source
 - :mod:`esmvalcore.io.intake_esgf`: Use the
   `intake-esgf <https://intake-esgf.readthedocs.io>`_ library to load data that
   is available from ESGF.
-- :mod:`esmvalcore.local`: Use :mod:`glob` patterns to find files on a filesystem.
-- :mod:`esmvalcore.esgf`: Use the legacy `esgf-pyclient
+- :mod:`esmvalcore.io.local`: Use :mod:`glob` patterns to find files on a filesystem.
+- :mod:`esmvalcore.io.esgf`: Use the legacy `esgf-pyclient
   <https://esgf-pyclient.readthedocs.io>`_ library to find and download data
   from ESGF.
 
@@ -755,7 +755,7 @@ commands:
     esmvaltool config copy data-local-esmvaltool.yml
 
 This will use the :mod:`esmvalcore.io.intake_esgf` module to access data
-that is available through ESGF and use :mod:`esmvalcore.local` to find
+that is available through ESGF and use :mod:`esmvalcore.io.local` to find
 observational and reanalysis datasets that have been
 :ref:`CMORized with ESMValTool <esmvaltool:inputdata_observations>`
 (``OBS6`` and ``OBS`` projects for CMIP6- and CMIP5-style CMORization
@@ -805,7 +805,7 @@ and tailor it for your system.
 .. note::
 
     Deduplicating data found via :mod:`esmvalcore.io.intake_esgf` data sources
-    and the :mod:`esmvalcore.local` data sources has not yet been implemented.
+    and the :mod:`esmvalcore.io.local` data sources has not yet been implemented.
     Therefore it is recommended not to use the configuration option
     ``search_data: complete`` when using both data sources for the same project.
     The ``search_data: quick`` option can be safely used.
@@ -831,7 +831,7 @@ This is particularly useful for native datasets which do not follow the CMOR
 standard by default and consequently produce a lot of warnings when handled by
 Iris.
 This can be configured using the ``ignore_warnings`` argument to
-:class:`esmvalcore.local.LocalDataSource`.
+:class:`esmvalcore.io.local.LocalDataSource`.
 
 Here is an example on how to ignore specific warnings when loading data from
 the ``EMAC`` model in its native format:
@@ -964,7 +964,7 @@ The ``esmvaltool run`` command can automatically download the files required
 to run a recipe from ESGF for the projects CMIP3, CMIP5, CMIP6, CORDEX, and obs4MIPs.
 
 Refer to :ref:`config-data-sources` for instructions on how to set this up. This
-section describes additional configuration options for the :mod:`esmvalcore.esgf`
+section describes additional configuration options for the :mod:`esmvalcore.io.esgf`
 module, which is based on the legacy esgf-pyclient_ library. Most users
 will not need this.
 
@@ -987,7 +987,7 @@ will not need this.
 Configuration file
 ------------------
 An optional configuration file can be created for configuring how the
-:class:`esmvalcore.esgf.ESGFDataSource` uses esgf-pyclient_
+:class:`esmvalcore.io.esgf.ESGFDataSource` uses esgf-pyclient_
 to find and download data.
 The name of this file is ``~/.esmvaltool/esgf-pyclient.yml``.
 
@@ -1076,7 +1076,7 @@ but it may be useful to understand its content.
 The settings from this file are being moved to the
 :ref:`new configuration system <config_overview>`. In particular, the
 ``input_dir``, ``input_file``, and ``ignore_warnings`` settings have already
-been replaced by the :class:`esmvalcore.local.LocalDataSource` that can be
+been replaced by the :class:`esmvalcore.io.local.LocalDataSource` that can be
 configured via :ref:`data sources <config-data-sources>`.
 The developer configuration file will be installed along with ESMValCore and can
 also be viewed on GitHub:
@@ -1121,7 +1121,7 @@ Preprocessor output files
 -------------------------
 
 The filename to use for preprocessed data is configured using ``output_file``,
-similar to the filename template in :class:`esmvalcore.local.LocalDataSource`.
+similar to the filename template in :class:`esmvalcore.io.local.LocalDataSource`.
 Note that the extension ``.nc`` (and if applicable, a start and end time) will
 automatically be appended to the filename.
 

doc/quickstart/find_data.rst

@@ -810,7 +810,7 @@ file name than for the netCDF4 variable name.
 
 To apply the extra facets for this purpose, simply use the corresponding tag in
 the applicable ``filename_template`` or ``dirname_template`` in
-:class:`esmvalcore.local.LocalDataSource`.
+:class:`esmvalcore.io.local.LocalDataSource`.
 
 For example, given the extra facets
 
@@ -834,5 +834,5 @@ a corresponding entry in the configuration file could look like:
 
 The same replacement mechanism can be employed everywhere where tags can be
 used, particularly in ``dirname_template`` and ``filename_template`` in
-:class:`esmvalcore.local.LocalDataSource`, and in ``output_file`` in
+:class:`esmvalcore.io.local.LocalDataSource`, and in ``output_file`` in
 :ref:`config-developer.yml <config-developer>`.

esmvalcore/_recipe/check.py

@@ -16,7 +16,7 @@
 
 import esmvalcore.preprocessor
 from esmvalcore.exceptions import InputFilesNotFound, RecipeError
-from esmvalcore.local import _parse_period
+from esmvalcore.io.local import _parse_period
 from esmvalcore.preprocessor import TIME_PREPROCESSORS, PreprocessingTask
 from esmvalcore.preprocessor._multimodel import _get_operator_and_kwargs
 from esmvalcore.preprocessor._other import _get_var_info

esmvalcore/_recipe/recipe.py

@@ -14,15 +14,16 @@
 
 import yaml
 
-from esmvalcore import __version__, esgf
+import esmvalcore.io.esgf
+from esmvalcore import __version__
 from esmvalcore._provenance import get_recipe_provenance
 from esmvalcore._task import DiagnosticTask, ResumeTask, TaskSet
 from esmvalcore.config._config import TASKSEP
 from esmvalcore.config._dask import validate_dask_config
 from esmvalcore.config._diagnostics import TAGS
 from esmvalcore.dataset import Dataset
 from esmvalcore.exceptions import InputFilesNotFound, RecipeError
-from esmvalcore.local import (
+from esmvalcore.io.local import (
     GRIB_FORMATS,
     _dates_to_timerange,
     _get_multiproduct_filename,
@@ -1327,7 +1328,7 @@ def run(self) -> None:
 
         # Download required data
         # Add a special case for ESGF files to enable parallel downloads
-        esgf.download(self._download_files)
+        esmvalcore.io.esgf.download(self._download_files)
         for file in self._download_files:
             file.prepare()
 

esmvalcore/_recipe/to_datasets.py

@@ -10,9 +10,9 @@
 from esmvalcore._recipe.check import get_no_data_message
 from esmvalcore.cmor.table import _CMOR_KEYS, _update_cmor_facets
 from esmvalcore.dataset import INHERITED_FACETS, Dataset, _isglob
-from esmvalcore.esgf.facets import FACETS
 from esmvalcore.exceptions import RecipeError
-from esmvalcore.local import _replace_years_with_timerange
+from esmvalcore.io.esgf.facets import FACETS
+from esmvalcore.io.local import _replace_years_with_timerange
 from esmvalcore.preprocessor._derive import get_required
 from esmvalcore.preprocessor._io import DATASET_KEYS
 from esmvalcore.preprocessor._supplementary_vars import (

esmvalcore/cmor/_fixes/icon/_base_fixes.py

@@ -23,7 +23,7 @@
 from iris.cube import CubeList
 from iris.mesh import Connectivity, MeshXY
 
-import esmvalcore.local
+import esmvalcore.io.local
 from esmvalcore.cmor._fixes.native_datasets import NativeDatasetFix
 from esmvalcore.config._data_sources import _get_data_sources
 from esmvalcore.iris_helpers import add_leading_dim_to_cube, date2num
@@ -328,7 +328,7 @@ def _get_grid_from_rootpath(self, grid_name: str) -> CubeList | None:
         """Try to get grid from the ICON rootpath."""
         glob_patterns: list[Path] = []
         for data_source in _get_data_sources(self.session, "ICON"):  # type: ignore[arg-type]
-            if isinstance(data_source, esmvalcore.local.LocalDataSource):
+            if isinstance(data_source, esmvalcore.io.local.LocalDataSource):
                 glob_patterns.extend(
                     data_source._get_glob_patterns(**self.extra_facets),  # noqa: SLF001
                 )

esmvalcore/config/_data_sources.py

@@ -7,8 +7,8 @@
 
 import yaml
 
-import esmvalcore.esgf
-import esmvalcore.esgf.facets
+import esmvalcore.io.esgf
+import esmvalcore.io.esgf.facets
 import esmvalcore.local
 from esmvalcore.exceptions import InvalidConfigParameter, RecipeError
 from esmvalcore.io import load_data_sources
@@ -52,16 +52,18 @@ def _get_data_sources(
     # Use legacy data sources from config-user.yml and config-developer.yml.
     data_sources: list[DataSource] = []
     try:
-        legacy_local_data_sources = esmvalcore.local._get_data_sources(project)  # noqa: SLF001
+        legacy_local_data_sources = esmvalcore.local._get_data_sources(  # noqa: SLF001
+            project,
+        )
     except (RecipeError, KeyError):
         # The project is not configured in config-developer.yml
         legacy_local_data_sources = []
     else:
         if (
             session.get("search_esgf", "") != "never"
-            and project in esmvalcore.esgf.facets.FACETS
+            and project in esmvalcore.io.esgf.facets.FACETS
         ):
-            data_source = esmvalcore.esgf.ESGFDataSource(
+            data_source = esmvalcore.io.esgf.ESGFDataSource(
                 name="legacy-esgf",
                 project=project,
                 priority=2,

esmvalcore/config/configurations/data-esmvalcore-esgf.yml

@@ -1,12 +1,12 @@
-# Download CMIP, CORDEX, and obs4MIPs data from ESGF using the `esmvalcore.esgf`
+# Download CMIP, CORDEX, and obs4MIPs data from ESGF using the `esmvalcore.io.esgf`
 # module, which uses the legacy ESGF search interface.
 projects:
   CMIP6: &esgf-pyclient-data
     data:
       esgf-pyclient:
-        type: "esmvalcore.esgf.ESGFDataSource"
+        type: "esmvalcore.io.esgf.ESGFDataSource"
         download_dir: ~/climate_data
-        # Use a lower priority than for esmvalcore.local.LocalDataSource
+        # Use a lower priority than for esmvalcore.io.local.LocalDataSource
         # to avoid searching ESGF with the setting `search_esgf: when_missing`.
         priority: 10
   CMIP5:

esmvalcore/config/configurations/data-hpc-badc.yml

@@ -3,49 +3,49 @@ projects:
   CMIP6:
     data:
       badc:
-        type: "esmvalcore.local.LocalDataSource"
+        type: "esmvalcore.io.local.LocalDataSource"
         rootpath: /badc/cmip6/data
         dirname_template: "{project}/{activity}/{institute}/{dataset}/{exp}/{ensemble}/{mip}/{short_name}/{grid}/{version}"
         filename_template: "{short_name}_{mip}_{dataset}_{exp}_{ensemble}_{grid}*.nc"
   CMIP5:
     data:
       badc:
-        type: "esmvalcore.local.LocalDataSource"
+        type: "esmvalcore.io.local.LocalDataSource"
         rootpath: /badc/cmip5/data
         dirname_template: "{project.lower}/{product}/{institute}/{dataset}/{exp}/{frequency}/{modeling_realm}/{mip}/{ensemble}/{version}"
         filename_template: "{short_name}_{mip}_{dataset}_{exp}_{ensemble}*.nc"
   CMIP3:
     data:
       badc:
-        type: "esmvalcore.local.LocalDataSource"
+        type: "esmvalcore.io.local.LocalDataSource"
         rootpath: /badc/cmip3_drs/data
         dirname_template: "{project.lower}/output/{institute}/{dataset}/{exp}/{frequency}/{modeling_realm}/{short_name}/{ensemble}/{version}"
         filename_template: "{short_name}_*.nc"
   CORDEX:
     data:
       badc:
-        type: "esmvalcore.local.LocalDataSource"
+        type: "esmvalcore.io.local.LocalDataSource"
         rootpath: /badc/cordex/data
         dirname_template: "{project}/output/{domain}/{institute}/{driver}/{exp}/{ensemble}/{institute}-{dataset}/{rcm_version}/{mip}/{short_name}/{version}"
         filename_template: "{short_name}_{domain}_{driver}_{exp}_{ensemble}_{institute}-{dataset}_{rcm_version}_{mip}*.nc"
   obs4MIPs:
     data:
       badc:
-        type: "esmvalcore.local.LocalDataSource"
+        type: "esmvalcore.io.local.LocalDataSource"
         rootpath: /gws/nopw/j04/esmeval/obsdata-v2
         dirname_template: "Tier{tier}/{dataset}"
         filename_template: "{short_name}_*.nc"
   OBS6:
     data:
       badc:
-        type: "esmvalcore.local.LocalDataSource"
+        type: "esmvalcore.io.local.LocalDataSource"
         rootpath: /gws/nopw/j04/esmeval/obsdata-v2
         dirname_template: "Tier{tier}/{dataset}"
         filename_template: "{project}_{dataset}_{type}_{version}_{mip}_{short_name}[_.]*nc"
   OBS:
     data:
       badc:
-        type: "esmvalcore.local.LocalDataSource"
+        type: "esmvalcore.io.local.LocalDataSource"
         rootpath: /gws/nopw/j04/esmeval/obsdata-v2
         dirname_template: "Tier{tier}/{dataset}"
         filename_template: "{project}_{dataset}_{type}_{version}_{mip}_{short_name}[_.]*nc"