perf(parquet): column parallelism + S3 byte range prefetching for arrow-rs reader

[desmondcheongzx]

The arrow-rs parquet reader was 15-25x slower than parquet2 on S3 reads due to three root causes: no cross-column I/O coalescing, pseudo-sequential async decode, and insufficient task parallelism. This PR closes the gap to parity.

S3 Benchmarks (TPC-H lineitem ~728MB, c7i.4xlarge)

Test	RGs	parquet2	arrow-rs (before)	arrow-rs (after)
all_cols	1RG	2191ms	4834ms	1982ms
all_cols	8RG	1355ms	5638ms	1355ms
all_cols	64RG	1406ms	5592ms	1434ms
3_contig	1RG	778ms	1326ms	869ms
3_contig	8RG	562ms	1219ms	638ms
3_contig	64RG	630ms	1341ms	754ms

Root Causes and Fixes

1. No cross-column I/O coalescing (15-25x slowdown)

Each per-column DaftAsyncFileReader ran its own ReadPlanner, so coalescing only saw one column's byte ranges at a time. Parquet2 computed ALL column byte ranges upfront and coalesced them into fewer large HTTP requests.

To fix this, prefetch_column_ranges() computes all (RG, column) byte ranges and feeds them through a single ReadPlanner. The coalesced data is cached in an Arc<RangesContainer>, and each per-column reader gets a PrefetchedAsyncFileReader that serves get_byte_ranges() from cache with zero HTTP requests.

2. Pseudo-sequential async decode

try_join_all polls all sub-futures from a single tokio task - CPU-bound column decode ran on one worker thread despite having 16 column tasks. Replaced with get_compute_runtime().spawn() per task, creating independent tasks that tokio's work-stealing scheduler distributes across all DAFTCPU threads (NUM_CPUS).

3. Insufficient task parallelism for many-RG files

Per-column streams (16-way parallelism) forced each task to process all 64 RGs sequentially. Switched to per-(RG, col) tasks on the compute runtime, giving N_RGs * N_cols parallelism (1024 tasks for 64 RGs and 16 cols). Results are grouped by column, concat'd per-column in parallel, then hconcat'd once via spawn_column_decode().

Changes

src/daft-parquet/src/async_reader.rs

• PrefetchedAsyncFileReader: AsyncFileReader impl backed by pre-fetched RangesContainer
• build_read_planner_and_collect(): shared ReadPlanner setup used by both DaftAsyncFileReader and the prefetch path

src/daft-parquet/src/arrowrs_reader.rs

• prefetch_column_ranges(): bulk byte range pre-fetching with cross-column/RG coalescing
• root_to_leaf_columns(): maps root column indices to parquet leaf indices
• decode_rg_predicate_phase_async_prefetched() / decode_rg_column_async_prefetched(): prefetched decode variants
• spawn_column_decode(): shared helper for per-(RG, col) task spawn, collect, group-by-column, parallel concat
• Modified read_parquet_single_arrowrs (Path 3) and stream_parquet_single_arrowrs (Path 4) to use prefetching + compute runtime dispatch
• Removed old DaftAsyncFileReader-based async decode helpers

Intra-RG Column Parallelism (pre-existing on branch)

Adds column-level parallelism within each row group across all four read paths (sync bulk, sync stream, async bulk, async stream). For wide tables with few row groups, this parallelizes across columns rather than just across RGs.

[codspeed-hq]

Merging this PR will not alter performance

✅ 36 untouched benchmarks

---

_{Comparing desmond/intra-rg-col-parallelism (f7fc593) with main (24e4c1d)}

[codecov]

Codecov Report

❌ Patch coverage is 35.96639% with 762 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.44%. Comparing base (2e68c2b) to head (f7fc593).
⚠️ Report is 22 commits behind head on main.

Files with missing lines	Patch %	Lines
src/daft-parquet/src/arrowrs_reader.rs	33.86%	742 Missing ⚠️
src/daft-parquet/src/async_reader.rs	70.58%	20 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6353      +/-   ##
==========================================
- Coverage   74.82%   74.44%   -0.39%     
==========================================
  Files        1022     1021       -1     
  Lines      136386   137603    +1217     
==========================================
+ Hits       102051   102435     +384     
- Misses      34335    35168     +833     

Files with missing lines	Coverage Δ
src/daft-parquet/src/async_reader.rs	60.55% <70.58%> (-15.05%)	⬇️
src/daft-parquet/src/arrowrs_reader.rs	58.22% <33.86%> (-25.96%)	⬇️

... and 115 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.