H5Dread_chunk with partial reads

[hmaarrfk]

I would love to be able to do a "partial read" of my raw chunk.
Some compression algorithm allow multi-level decompression, so doing a partial read, would give us information like:

1. what part of the chunk do we actually need to read to decode at a certain level.

I understand that this breaks some of the models in HDF5, but for certain very large dataset use cases, this kind of "multi-level reading" really does help.

Describe the solution you'd like

Being able to provide a:

• offset within the chunk
• size of read

to read at the raw level.

Describe alternatives you've considered

Saving the data in multiple resolutions.
That isn't fun now is it ;).

It just becomes hard to manage

Additional context

I thought of doing this with the direct file pointer, but in certain cases, such as virtual dataset, it doesn't always exist.

[ajelenak]

Is this related to some JPEG2000 compressed image data? How large in bytes those HDF5 datasets are? Where are you reading from? local file system, object store, something else?

[hmaarrfk]

Is this related to some JPEG2000 compressed image data?

yes.

How large in bytes those HDF5 datasets are?

Generally, we work with TB (uncompressed) datasets.

Where are you reading from? local file system, object store, something else?

Typically this is a concern for network storage, where data transfer can be severely limited.

[hmaarrfk]

I guess JPEG2000 (and likely other compression algorithms) have their own "chunking" mechanisms.

So like "1 HDF5 chunk = JPEG2000 codestream". But you would like some way to navigate around that codestream without reading it (since it can be quite large)

[ajelenak]

But you would like some way to navigate around that codestream without reading it (since it can be quite large)

Can the JPEG2000 library provide "waypoints" through one codestream?

[hmaarrfk]

I don't understand the word "waypoints".

Instead of me interpreting poorly, can you elaborate before I answer?

[ajelenak]

Since those chunks are compressed with JPEG2000, can its library provide information where in that chunk to perform partial reads and for how many bytes?

[hmaarrfk]

yes, generally this information is stored at the beginning of a stream, and JPEG2000 can be optimized for different usages that make it easier to access data by resolution or by color for example, depending on the application need.

[hmaarrfk]

These are sometimes called "TLM", which is "optional" but is designed for this usecase
https://discourse.pangeo.io/t/cloud-optimized-access-to-sentinel-2-jpeg2000/5072

[ajelenak]

It seems to me the Pangeo link is essentially about partial chunk reading bypassing the library which created those GeoTIFFs. Have you considered that option?

[hmaarrfk]

i have considered bypassing HDF5 totally. One can do that.

I'm happy to continue down that path, but certain features of HDF5 are nice. Virtual Datasets, different backends.

I guess not being able to overcome this partial read would be one more reason for us to move away from HDF5.

but i also see it as an opportunity to extend the form and keep it modern.

[gheber]

Just thinking out loud, for a quick hack w/o API change, couldn't the offset and read size be channeled through the dxpl (second argument)? The implementation would need to check for those properties and act accordingly.

[hmaarrfk]

The implementation would need to check for those properties and act accordingly.

This was my original thinking. However, my understanding is that the current implementation 'ignores dxpl'. I'm not sure how "breaking" it would be to start to look at it!

[gheber]

Someone with great foresight put it there, and we might as well use it. The only API change would be another H5P[get,set]_sub_chunk function pair.