HDF5 support

[klamike]

This PR adds support for tabular HDF5 file(s) by converting each row to an Arrow table. It supports columns with the usual dtypes including up to 5-dimensional arrays as well as support for complex/compound types by using Features(dict). All datasets within the HDF5 file should have rows on the first dimension (groups/subgroups are still allowed). Closes #3113.

Replaces #7625 which only supports a relatively small subset of HDF5.

[klamike]

A few to-dos which I think can be left for future PRs (which I am happy to do/help with -- just this one is already huge 😄 ):

• Enum types
• HDF5 io
• dataset-viewer support (not sure if changes are needed with the way it is written now)

[klamike]

@lhoestq any interest in merging this? Let me know if I can do anything to make reviewing it easier!

[lhoestq]

Sorry for the delay, I'll review your PR soon :)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lhoestq]

This is great ! I left a few comments :)

Btw feel free to run make style to fix code formatting

[lhoestq]

maybe it should create nested features like this instead ?

Features({
    "position": {
        "x": List(Value("int64")),
        "y": List(Value("int64")),
    },
    "veolocity": {
        "vx": List(Value("int64")),
        "vy": List(Value("int64")),
    },
})

[klamike]

does db2e76a look good? I'm not sure it will be as fast as the separate columns but it does clean up the collision checks.

[lhoestq]

Looks good ! btw with ds = ds.flatten() you can get a flat structure with columns named "position.x", "position.y" etc.

[klamike]

Thanks for the review @lhoestq! Rebased on main and incorporated most of your suggestions.

I believe the only one left is the zero-dim handling with table_cast...

[klamike]

@lhoestq is 2c4bfba what you meant?

[lhoestq]

Yay ! LGTM :)

Let's document this now, this is big !
Would you like to to open a PR for the docs ?

also cc @georgiachanning for viz

[klamike]

Awesome! Yes, I'm happy to help with the docs. Would appreciate any pointers, we can discuss in #7740.

It does look like there was a CI test failure, though it seems unrelated?

FAILED tests/test_dataset_dict.py::test_dummy_datasetdict_serialize_fs - ValueError: Protocol not known: mock
FAILED tests/test_arrow_dataset.py::test_dummy_dataset_serialize_fs - ValueError: Protocol not known: mock

Also, what do you think of the todos in #7690 (comment) ? In particular I think support in dataset-viewer would be nice.

[lhoestq]

Cool ! Yeah the failure is unrelated

Regarding the Viewer, it should work out of the box when it's updated with the next version of datasets :)