Combining polytope source with saving and regridding causes crash

[vegardb]

What happened?

A specific sequence of actions causes a crash:

• Read data with polytope
• Save to a grib file
• regrid the originally read data, using earthkit.regrid
• Try to save regrid results to another grib file

This does not crash if the data source is a grib file instead of polytope

What are the steps to reproduce the bug?

import earthkit.data as ekd
import earthkit.regrid as ekr

request = {
    'date': '2025-06-15', 
    'time': 6, 
    'stream': 'scda', 
    'type': 'fc', 
    'step': [0], 
    'grid': 'N1280', 
    'param': ['2t', '2d', 'skt', 'tp', '10u', '10v', 'z', 'lsm', 'tcw', 'sp', 'msl', 'lcc', 'mcc', 'hcc', 'tcc', '3020', '10si', 'cbh', 'ssrd', 'strd'], 
    'levtype': 'sfc', 
    'class': 'od' # Not needed before
}
ds = ekd.from_source("polytope", "ecmwf-mars", request)
ds.to_target('file', 'test.grib')
# ds = ekd.from_source('file', 'test.grib')
ds = ekr.interpolate(ds, out_grid={'grid': [0.25,0.25]}, method='nearest-neighbour')
ds.to_target('file', 'test2.grib')
print(ds)

Running this gives a crash. See below for output.

Version

0.14.4

Platform (OS and architecture)

Mac

Relevant log output

2025-06-23 14:17:55 - INFO - Key read from /Users/vegardb/.polytopeapirc
2025-06-23 14:17:55 - INFO - Sending request...
{'request': 'class: od\n'
            "date: '2025-06-15'\n"
            'grid: N1280\n'
            'levtype: sfc\n'
            'param:\n'
            '- 2t\n'
            '- 2d\n'
            '- skt\n'
            '- tp\n'
            '- 10u\n'
            '- 10v\n'
            '- z\n'
            '- lsm\n'
            '- tcw\n'
            '- sp\n'
            '- msl\n'
            '- lcc\n'
            '- mcc\n'
            '- hcc\n'
            '- tcc\n'
            "- '3020'\n"
            '- 10si\n'
            '- cbh\n'
            '- ssrd\n'
            '- strd\n'
            'step:\n'
            '- 0\n'
            'stream: scda\n'
            'time: 6\n'
            'type: fc\n',
 'verb': 'retrieve'}
2025-06-23 14:17:55 - INFO - Polytope user key found in session cache for user vegardb
2025-06-23 14:17:57 - INFO - Request accepted. Please poll ./1b99da4f-a978-4149-84f2-d4e2a3cfbfc1 for status
2025-06-23 14:17:57 - INFO - Polytope user key found in session cache for user vegardb
2025-06-23 14:17:57 - INFO - Checking request status (1b99da4f-a978-4149-84f2-d4e2a3cfbfc1)...
2025-06-23 14:17:58 - INFO - The current status of the request is 'processing'
2025-06-23 14:18:19 - INFO - The current status of the request is 'processed'
Traceback (most recent call last):
  File "/Users/vegardb/src/gitlab.met.no/metno/proj/brismalawi/bris-initialcondition/test.py", line 19, in <module>
    ds.to_target('file', 'test2.grib')
  File "/Users/vegardb/src/gitlab.met.no/metno/proj/brismalawi/bris-initialcondition/.venv/lib/python3.11/site-packages/earthkit/data/sources/__init__.py", line 105, in to_target
    to_target(target, *args, data=self, **kwargs)
  File "/Users/vegardb/src/gitlab.met.no/metno/proj/brismalawi/bris-initialcondition/.venv/lib/python3.11/site-packages/earthkit/data/targets/__init__.py", line 271, in to_target
    t.write(**kwargs)
  File "/Users/vegardb/src/gitlab.met.no/metno/proj/brismalawi/bris-initialcondition/.venv/lib/python3.11/site-packages/earthkit/data/targets/__init__.py", line 146, in write
    self._write(data, **kwargs)
  File "/Users/vegardb/src/gitlab.met.no/metno/proj/brismalawi/bris-initialcondition/.venv/lib/python3.11/site-packages/earthkit/data/targets/file.py", line 129, in _write
    for d in r:
  File "/Users/vegardb/src/gitlab.met.no/metno/proj/brismalawi/bris-initialcondition/.venv/lib/python3.11/site-packages/earthkit/data/encoders/grib.py", line 418, in _encode_fieldlist
    for f in fs:
  File "/Users/vegardb/src/gitlab.met.no/metno/proj/brismalawi/bris-initialcondition/.venv/lib/python3.11/site-packages/earthkit/data/core/index.py", line 528, in __getitem__
    return self._getitem(n)
           ^^^^^^^^^^^^^
AttributeError: 'FieldList' object has no attribute '_getitem'. Did you mean: '__getitem__'?

Accompanying data

No response

Organisation

Met Norway

[sandorkertesz]

@vegardb, thank you for reporting this issue. We are looking into it and try to find a solution.

[sandorkertesz]

@vegardb The problem is that the polytope source by default returns a stream fieldlist (stream=True). You consume this stream when calling ds.to_target('file', 'test.grib'). So the input to interpolate() is an empty object and the result is an empty object too. Unfortunately to_target() does not handle it correctly resulting in an AttributeError. However, the real problem is that you pass an empty object to interpolate().

The possible solution is to use the read_all=True option in the polytope retrieval. This would result in an in-memory fieldlist so everything would work. Please see the documentation about the streams in earthkit-data: https://earthkit-data.readthedocs.io/en/latest/guide/streams.html

[vegardb]

Ok, thanks, I had overlooked that part of the docs.

For another time: Is it possible to give a more explicit error message, such as Exception('stream already consumed')?