feat: primitive parquet reader with page pruning

[kszucs]

Prototype implementation for an arrow-rs based page pruning parquet reader for low latency limit/offset queries.

It is a standalone library for now, haven't been integrated to the viewer yet.

Install

cd libs/libviewer
pip install maturin
maturin develop -r  

Index Dataset

dv --use-cache nvidia/OpenCodeReasoning index

This uses huggingface_hub to download and cache the dataset files.
Then creates a metadata file for each parque file in the dataset with
offset index included.

Remove --use-cache to directly download the files from the hub.

Execute a limit/offset query

dv --use-cache nvidia/OpenCodeReasoning query --limit 10 --offset 0

This will query the dataset using the local metadata index files.
The scanner only reads the necessary parquet pages to minimize the
network traffic.

Remove --use-cache to directly query data from the hub.

Integration and testing

Before covering it with tests, it would be nice to see the necessary API for integration.

Supersedes #3199

[kszucs]

I temporarily turned it off because it wasn't installing the optional libviewer dependency properly.

[kszucs]

Building the rust based libviewer as a wheel to not include the compiler toolchains in the final docker images.

[kszucs]

Originally I added it as a mandatory dependency but I wasn't able to convince poetry to skip installing it from source in the docker image, but rather use a prebuilt wheel, see https://github.com/huggingface/dataset-viewer/pull/3244/files#r2432505025.

Apparently the path dependencies doesn't work well with compiled extension modules. Ideally we should build wheels for all the internal libs (libviewer, libcommon, libapi) but the dependency versions pinned in the pyproject files are more loose than what we have in the poetry lockfiles and some of the builds/tests are sensitive to those dependencies.

So I chose to define libviewer as an optional dependency which we install only to the relevant services using prebuilt wheels in the containers and using --with libviewer during local development.

[lhoestq]

Looks pretty good ! The main point to address is raising TooBigRows when possible to avoid OOMing the /rows worker

[lhoestq]

We can see later to truncate binary data (sometimes users have a column with very long binary data and we truncate them when reading them to not OOM)

Though what we will need right away is to raise TooBigRows if the resulting record batches are likely to cause a OOM (if they can use >300MB of ram). We can use a simple heuristic based on average row size in the row group to know if it's safe to run the query or not

[kszucs]

Here we can either enable libviewer globally or provide a set of dataset names to selectively enable libviewer for them.

[kszucs]

hf_token is needed for libviewer but not for the old index since httpfs already contains it. I'm not unifying it because of the session handling for httpfs is a little hard to follow, also would complicate the testing. If libviewer turns out to be stable we can just remove httpfs entirely with the old indexer.

[kszucs]

Query the dataset information from mongo here, note that we should switch to an asynchronous approach here to avoid blocking the event loop. Libviewer supports async queries so RowsIndex can be async first in the future.

[lhoestq]

Looks good to me overall ! 🚀

Any idea why we get this in the e2e CI ?

File "/src/libs/libcommon/src/libcommon/parquet_utils.py", line 561, in query_libviewer_index
    batches = self.viewer_index.sync_scan(offset=offset, limit=length, scan_size_limit=self.max_scan_size)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
libviewer.PyDatasetError: External: Generic LocalFileSystem error: Requested range was invalid

and for the worker CI, you can fix this with this change I believe

- assert len(mock_http_file.call_args_list) == 1
+ assert len(mock_http_file.call_args_list) == 0  #  it uses libviewer, not pyarrow + HTTPFile

To fix the problem, an explicit permissions block should be added to the workflow file. In this case, since no job appears to need write access (all jobs do quality checks and run unit tests, based on their names), it is safest to set contents: read at the workflow level, unless any reusable workflow requires something broader in its own documentation (not shown here).

Steps to fix:

• Add a permissions: section at the root of .github/workflows/l-libcommon.yml (e.g., after name: and before on:).
• Use contents: read as the default minimal permissions required.
• If later you discover a job or called reusable workflow requires more (such as pull-requests: write), you can add those specifically.

Regions to change:

• Insert the following after line 4 (after name: libs/libcommon):

  permissions:
    contents: read
  

This does not require any new methods, imports, or definitions, as it is a YAML configuration change.

---