datasets downloads and generates all splits, even though a single split is requested (for dataset with loading script)

[sei-eschwartz]

Describe the bug

datasets downloads and generates all splits, even though a single split is requested. This is the dataset in question. It uses a loading script. I am not 100% sure that this is a bug, because maybe with loading scripts datasets must actually process all the splits? But I thought loading scripts were designed to avoid this.

Steps to reproduce the bug

See this notebook

Or:

from datasets import load_dataset
dataset = load_dataset('jordiae/exebench', split='test_synth', trust_remote_code=True)

Expected behavior

I expected only the test_synth split to be downloaded and processed.

Environment info

• datasets version: 3.5.1
• Platform: Linux-6.1.123+-x86_64-with-glibc2.35
• Python version: 3.11.12
• huggingface_hub version: 0.30.2
• PyArrow version: 18.1.0
• Pandas version: 2.2.2
• fsspec version: 2025.3.0

[lhoestq]

Hi ! there has been some effort on allowing to download only a subset of splits in #6832 but no one has been continuing this work so far. This would be a welcomed contribution though

Also note that loading script are often unoptimized, and we recommend using datasets in standard formats like Parquet instead.

Btw there is a CLI tool to convert a loading script to parquet:

datasets-cli convert_to_parquet <dataset-name> --trust_remote_code

[sei-eschwartz]

Closing in favor of #6832