Introduce read_parquet_uniform node

[rjzamora]

Adds a new streaming node for reading Parquet data with uniform chunk distribution. Here, "uniform" does not mean that all chunks will be a uniform size. Rather, it means that every chunk will correspond to the same file count of file fraction. This approach assumes the files and row-groups in the dataset have a relatively uniform distribution.

Motivation:

• This partitioning approach is similar to the approach already used in cudf-polars. Therefore, it is easy to integrate this API with cudf-polars.
• This approach produces file- or row-group-aligned reads in most cases.
  • The only exception is when a large file is partitioned into a larger number of chunks than the number of row-groups (which is rare).
• This approach supports distributed chunking across a single large file. The read_parquet node doesn't support this yet (though, it certainly can in the future. See: Support large-file splitting between ranks in read_parquet #736).

Other considerations:

• The main read_parquet node may be safer for datasets with non-uniform file and/or row-group sizes.
• The estimate_target_num_chunks utility cannot account for the effects of filters. Therefore, the size of each chunk may be significantly smaller than the corresponding num_rows_per_chunk argument when a filter is applied at IO time (even if the dataset is "uniform"). This may also be the case for row-count-based chunks (?)
• The effect of aligning reads with row-group/file boundaries was almost negligible in my local testing (<5%). Therefore, using read_parquet_uniform is unlikely to provide a measurable performance improvement (more testing is needed to say for sure).

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[rjzamora]

/ok to test