feature: memory source retrieve more than one field

[HCookie]

What happened?

When reading from a memory buffer, only the first field is retrieved and recorded in the ds.

import earthkit.data as ekd
from io import BytesIO

sample_ds = ekd.from_source("sample", "storm_ophelia_wind_850.grib")
print(sample_ds.ls())
#   centre shortName    typeOfLevel  level  dataDate  dataTime stepRange  
# 0   ecmf         u  isobaricInhPa    850  20171016         0         0   
# 1   ecmf         v  isobaricInhPa    850  20171016         0         0   

bufr = BytesIO()
sample_ds.to_target('file', bufr)

mem_ds = ekd.from_source("memory", bufr.getvalue())
print(mem_ds.ls())
#   centre shortName    typeOfLevel  level  dataDate  dataTime stepRange  
# 0   ecmf         u  isobaricInhPa    850  20171016         0         0   

Request

Can this be enhanced to retrieve more?

Version

0.15.0

Platform (OS and architecture)

ATOS

Organisation

ECMWF

[HCookie]

Discussing with @oiffrig,
The issue is here:

earthkit-data/src/earthkit/data/readers/grib/memory.py

Lines 87 to 100 in 6e77019

	class GribMessageMemoryReader(GribMemoryReader):
	def __init__(self, buf, **kwargs):
	super().__init__(**kwargs)
	self.buf = buf
	def __del__(self):
	self.buf = None
	def _next_handle(self):
	if self.buf is None:
	return None
	handle = eccodes.codes_new_from_message(self.buf)
	self.buf = None
	return handle

_next_handle can only ever return one handle, as it does in eccodes-python.
As we don't have a pointer in the memory buffer it's unclear where the reader is up to.

Possible fixes

• Find a way to tell the size of the handle and iterate over it
• Convert to stream and return GribStreamReader

class GribMessageMemoryReader(GribStreamReader):
   def __init__(self, buf, **kwargs):
       super().__init__(BytesIO(buf), **kwargs)

[sandorkertesz]

@HCookie, this is not a bug but a feature. See the docs:

So what you want is an enhancement.