Error Out of bounds nanosecond timestamp: 2300-12-01 00:00:00 & Add cftime support

[senclimate]

** Out of bounds nanosecond timestamp Error

import intake_esgf
from intake_esgf import ESGFCatalog
cat = ESGFCatalog()
sel_cat = cat.search(experiment_id='ssp585', variable_id='uas', table_id="Amon", source_id='ACCESS-ESM1-5', grid_label='gn', member_id='r2i1p1f1', latest=True)
sel_cat.df

dsd = sel_cat.to_dataset_dict(prefer_streaming=True, add_measures=False)

Describe the solution you'd like
add xarray_kwargs={'cftime': True} or similar args in to_dataset_dict to support cftime in xarray

[senclimate]

Ideas for the solution:

1. Replace the following get_time_extent in base.py

import cftime
from typing import Union, Tuple

def get_time_extent(filename: str) -> Tuple[Union[pd.Timestamp, cftime.DatetimeNoLeap, None], Union[pd.Timestamp, cftime.DatetimeNoLeap, None]]:
    """
    Parse the timespan out of the filename, if possible.
    Returns a tuple of (start_time, end_time), using pd.Timestamp when valid,
    or cftime.DatetimeNoLeap if out of bounds for pandas.
    """

    def _to_datetime(s: str) -> Union[pd.Timestamp, cftime.DatetimeNoLeap]:
        if len(s) == 6:
            s += "01"
        try:
            return pd.Timestamp(s)
        except (pd.errors.OutOfBoundsDatetime, ValueError):
            year = int(s[:4])
            month = int(s[4:6])
            day = int(s[6:8]) if len(s) == 8 else 1
            return cftime.DatetimeNoLeap(year, month, day)

    match = re.search(r"_(\d+)-(\d+)", filename)
    if not match:
        return None, None

    return _to_datetime(match.group(1)), _to_datetime(match.group(2))

2. add cftime = True in

   intake-esgf/intake_esgf/catalog.py

   Line 740 in b680447

   ds[key] = xr.open_mfdataset(sorted(files))

   ds[key] = xr.open_mfdataset(sorted(files), use_cftime=True)

[nocollier]

Thank you for this report. I will give this some thought, I want to see what intake-esm does. Perhaps an additional argument to to_dataset_dict() of the sort xr_kwargs=dict(use_cftime=True). I am afraid that others will have other needs, especially as we see more exotic streaming options.

Please note in the near term you can call to_path_dict() instead with the same arguments and then call your xarray constructor as you wish.