Description

Currently, scanning a cloud file for CSV and NDJSON perform a full download of the data onto the local disk before scanning. Ideally we can instead download and process the file in chunks.

• scan_ndjson / scan_lines perf: Streaming cloud download for scan_ndjson / scan_lines #26563
• scan_csv perf: Streaming cloud download for scan_csv #26637
• pl.len() for CSV (currently using count_rows)
• restrict number of files (issue Restrict CSV schema inference to 10 files by default and add configuration API #26064)
• update public APIs for file cache deprecation