Support HDF4?

[TomNicholas]

Could we support generating chunk manifests pointing to HDF4 files too? I know nothing about this format, but in #85 (comment) @jgallagher59701 mentioned that DMR++ can (or soon will) support it.

I should add to the above that many of the newer features in DMR++ are there to support HDF4 - yes, '4' - and that requires some hackery in the interpreter. Look at how can now contain elements. In HDF4, a 'chunk' is not necessarily atomic. Also complicating the development of an interpreter is the use of fill values in both HDF4 and HDF5, even for scalar variables. That said, we have a full interpreter in C++, which i realize is not exactly enticing for many ;-), but that means there is code for this and this 'documentation' is 'verifiable' since it's running code.

If DMR++ can index HDF4, and DMR++ can be translated to zarr chunk manifests (see #85), then presumably a reader for HDF4 directly to chunk manifests would also be possible?

cc @ayushnag @betolink

[jgallagher59701]

On Aug 7, 2024, at 13:39, Tom Nicholas ***@***.***> wrote: Could we support generating chunk manifests pointing to HDF4 files too? I know nothing about this format, but in #85 (comment) <#85 (comment)> @jgallagher59701 <https://github.com/jgallagher59701> mentioned that DMR++ can (or soon will) support it.

We use the same code to interpret the DMR++ for HDF5 and HDF4.

I should add to the above that many of the newer features in DMR++ are there to support HDF4 - yes, '4' - and that requires some hackery in the interpreter. Look at how can now contain elements. In HDF4, a 'chunk' is not necessarily atomic. Also complicating the development of an interpreter is the use of fill values in both HDF4 and HDF5, even for scalar variables. That said, we have a full interpreter in C++, which i realize is not exactly enticing for many ;-), but that means there is code for this and this 'documentation' is 'verifiable' since it's running code. If DMR++ can index HDF4, and DMR++ can be translated to zarr chunk manifests, then presumably a reader for HDF4 directly to chunk manifests would also be possible?

Yes. There’s quite a bit to HDF4, however, because it is a more complex format than HDF5. And, NASA’s HDF4 is not vanilla HDF4, so it has its own complexities on top of that. Bottom line, you will probably have to extend the interpreter you have, but it’s certainly possible and there is lots of data in HDF4. HTH, James

…

cc @ayushnag <https://github.com/ayushnag> @betolink <https://github.com/betolink> — Reply to this email directly, view it on GitHub <#216>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7Q4KVNGFPHGNACJJUXNKTZQJZVZAVCNFSM6AAAAABMFBYFYOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ2TIMRQGM4TQOA>. You are receiving this because you were mentioned.

-- James Gallagher ***@***.***

[TomNicholas]

@martindurant has an in-progress PR to kerchunk to add support for reading HDF4 directly. If that makes it in we can just call it from vz.open_virtual_dataset, which would fully close this issue.

[martindurant]

I should warn you, that I am working to match only specific NASA data (provided by @maxrjones ), not HDF4 in general, and I suspect that the chunks in general may be tiny.

[jgallagher59701]

Older data in HDF4/5 almost always has small chunks (spinning disks, low-latency, small block sizes). But that is not a big problem. Group the contiguous chunks and transfer them in a single I/O operation and then decompress them in parallel. We call these grouped chunks 'Super Chunks.' It is an optimization that Patrick Quinn first implemented and we stumbled on later. This is far more efficient than transferring the small chunks in parallel (in general, exceptions exist).

[martindurant]

Yes, kerchunk also joins near-contiguous chunks; the problem I actually see

• the large number of references means relatively big reference stores
• relatively small gains for reading only select chunks compared to grabbing the whole file every time.

[TomNicholas]

Group the contiguous chunks and transfer them in a single I/O operation and then decompress them in parallel.

Yes, kerchunk also joins near-contiguous chunks

This is something interesting that I've not heard about before. By "grouping" or "joining" do you mean literally concatenating the byte ranges together? Or something else?

[jgallagher59701]

Group the contiguous chunks and transfer them in a single I/O operation and then decompress them in parallel.
Yes, kerchunk also joins near-contiguous chunks

This is something interesting that I've not heard about before. By "grouping" or "joining" do you mean literally concatenating the byte ranges together? Or something else?

I mean concatenating the byte ranges. Often in these files the chunks lie right next to each other (for a given array).

[jgallagher59701]

Yes, kerchunk also joins near-contiguous chunks; the problem I actually see
...

• relatively small gains for reading only select chunks compared to grabbing the whole file every time.

That's true for files with a small number of variables. Get the whole file. If there are O(10^2) variables and only 2-3 are needed, it's faster to get just those 2-3. Again, there are exceptions.

[martindurant]

In ReferenceFS, if you cat() with a number of references, those within a single file may be merged depending on the arguments

        max_gap=64_000,
        max_block=256_000_000,

For example, for references [remote://file, 10, 10] , [remote://file, 30, 10], the actual request will be bytes 10->40, if the gap is smaller than max_gap. The result is sliced into two outputs.
Naturally, if max_gap=0, only truly contiguous parts are merged, and <0 for no merge at all. The requests would still be concurrent, however.

[mdsumner]

That said, we have a full interpreter in C++, which i realize is not exactly enticing for many

Why aren't we using DMR++? Is it not in good enough shape to bind to Python/R? Is there other challenges, there's plenty of C++ used seamlessly in Python and calling out to h5 libs is doing that anyway.

That sounds like the crosslang solution already ?? I only have a few HDF4 stores of interest outside of NASA, and maybe only one.

[mdsumner]

There's something I'm missing given #113 🙏 I'll keep exploring I keep finding new aspects 👌.

[martindurant]

I'm sorry if I have done some duplication of work. I think it may be worthwhile to have a pure-python solution too, though, for the case that no dmr++ index files exist for some HDF4. Also, it has been (so far) nerdy fun, definitely work a blog post.