Conversion `to_xarray()` fails if data inconsistent

[j-haacker]

What happened?

I received a grib2 dataset with missing time steps for one observable. This is not convertable to xarray using the corresponding method. I couldn't figure out how to fill no-data values in the grib data.

What are the steps to reproduce the bug?

I don't know how to produce a grib dataset like the one making trouble. I will not be able to offer a minimal working example. Moreover, you will need an upgraded destine.eu account. I apologize for the inconvenience.

Define the request:

    request = {
        'activity': 'CMIP6',
        'class': 'd1',
        'dataset': 'climate-dt',
        'experiment': 'HIST',
        'expver': '0001',
        'generation': '1',
        'levtype': 'sfc',
        'model': 'IFS-NEMO',
        'param': '167/168',  # 2t, 2d
        'realization': '1',
        'resolution': 'high',
        'stream': 'clte',
        'type': 'fc',
        'date': '1993-08-23/1993-08-24',
        'time': '0000',
    }

Download and convert:

data = earthkit.data.from_source("polytope", "destination-earth", request, address="polytope.lumi.apps.dte.destination-earth.eu", stream=False)#
ds = data.to_xarray()

Version

0.12.1

Platform (OS and architecture)

Linux 6.8.0-52-generic Ubuntu 24.04 x86_64

Relevant log output

/lib/python3.12/site-packages/earthkit/data/utils/xarray/builder.py", line 425, in check_tensor_coords
    check_coords(var_name, coord_name, tensor_coords, self.tensor_coords)
  File "/lib/python3.12/site-packages/earthkit/data/utils/xarray/check.py", line 37, in check_coords
    raise ValueError(
ValueError: Variable "2d" has inconsistent dimension "forecast_reference_time" compared to other variables. Expected values: (5) ['1993-08-23T00:00:00', '1993-08-23T06:00:00', '1993-08-23T12:00:00', '1993-08-23T18:00:00', '1993-08-24T00:00:00'], got: (4) ['1993-08-23T00:00:00', '1993-08-23T06:00:00', '1993-08-23T12:00:00', '1993-08-23T18:00:00']. Length mismatch: 5 != 4

Accompanying data

Not allowed.

Organisation

BOKU University

[j-haacker]

Reproducing the error does not work like described above, or at least I just tried and did not reproduce it.

For my personal demands I tried to download a variable that is not available on the "sfc" level, namely "157". If I request this variable 157, usually I receive a dataset of all other requested variables without 157. However, here it may be responsible for the error because with it the conversion to_xarray fails without it the conversion worked in one try another (but exactly the same script) threw:

Traceback (most recent call last):
  File "/download_script.py", line 88, in <module>
    main()
  File "/snippets/monitoring.py", line 15, in applicator
    return f(*args,**kwargs)
           ^^^^^^^^^^^^^^^^^
  File "/download_script.py", line 71, in main
    ds = data.to_xarray()
         ^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/readers/grib/xarray.py", line 376, in to_xarray
    return engines[engine](user_xarray_open_dataset_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/readers/grib/xarray.py", line 417, in to_xarray_earthkit
    return from_earthkit(self, backend_kwargs=backend_kwargs, other_kwargs=other_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/utils/xarray/builder.py", line 694, in from_earthkit
    return xarray.open_dataset(ds, backend_kwargs=backend_kwargs, **other_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/xarray/backends/api.py", line 679, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/utils/xarray/engine.py", line 294, in open_dataset
    return SingleDatasetBuilder(fieldlist, profile, from_xr=True, backend_kwargs=_kwargs).build()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/utils/xarray/builder.py", line 586, in build
    ds_sorted, _ = self.parse(self.ds, self.profile)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/utils/xarray/builder.py", line 540, in parse
    ds_xr = XArrayInputFieldList(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/core/__init__.py", line 22, in __call__
    obj.__init__(*args, **kwargs)
  File "/.venv/lib/python3.12/site-packages/earthkit/data/utils/xarray/fieldlist.py", line 118, in __init__
    self.db = IndexDB(*self.unique_values(keys, component=component))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/utils/xarray/fieldlist.py", line 199, in unique_values
    r = f._attributes(keys, remapping=self.remapping, joiner=joiner)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/core/fieldlist.py", line 725, in _attributes
    result[name] = metadata(name, default=default)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/core/metadata.py", line 119, in wrapped
    v = func(self, key, default=default, astype=astype, raise_on_missing=raise_on_missing)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/core/metadata.py", line 81, in wrapped
    return func(cls, key, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/readers/grib/metadata.py", line 383, in get
    v = self._handle.get(key, ktype=astype, **_kwargs)
        ^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/readers/grib/metadata.py", line 667, in _handle
    return self._field.handle
           ^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/readers/grib/codes.py", line 260, in handle
    handle = self._handle_manager.handle(self, self._create_handle)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/readers/grib/index/__init__.py", line 319, in handle
    handle = create()
             ^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/readers/grib/codes.py", line 272, in _create_handle
    return GribCodesReader.from_cache(self.path).at_offset(self.offset)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/earthkit/data/utils/message.py", line 366, in at_offset
    handle = eccodes.codes_new_from_file(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/gribapi/gribapi.py", line 313, in codes_new_from_file
    return grib_new_from_file(fileobj, headers_only)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/gribapi/gribapi.py", line 415, in grib_new_from_file
    GRIB_CHECK(err)
  File "/.venv/lib/python3.12/site-packages/gribapi/gribapi.py", line 232, in GRIB_CHECK
    errors.raise_grib_error(errid)
  File "/.venv/lib/python3.12/site-packages/gribapi/errors.py", line 381, in raise_grib_error
    raise ERROR_MAP[errid](errid)
gribapi.errors.PrematureEndOfFileError: End of resource reached when reading message

[j-haacker]

It turns out the data was fixed in the background which is why the error is not reproducible anymore.

However, if the maintainers agree, I would leave the issue open until a decision is made on whether inconsistent datasets should raise an error or whether they should be filled with no-data values.

[frazane]

I am encountering situations where I would prefer to have data with missing values instead of an error. It would be nice to be able to specify an option for this!

[sandorkertesz]

Thanks for the proposal, we will look into it. I agree an option to control this behaviour would be useful.

[sandorkertesz]

FYI, there is now a separate issue dedicated to this problem #651, so I close this issue and suggest to move any related discussion there.