HuggingFace dataset streaming support

[abhinavpola]

To run an eval on a very large dataset that can't fit in memory or cached on disk, we have to use HF dataset loader's streaming option. Currently Inspect AI's HF dataset wrapper by default will call dataset.save_to_disk which is unsupported on the dataset iterator. Additionally, MemoryDataset assumes a fully loaded dataset for slicing, shuffling, etc.

I would love to be able to run evals on large datasets so just wanted to put in this as a feature request.

Much <3 to the maintainers of this repo.

[jjallaire]

If we can do the slicing, shuffling etc. on a streaming dataset then we could create another subclass of MemoryDataset that works w/ streaming. That said, there is other downstream code that depends on having all of the samples in memory so this might in require a much bigger architectural change.

[abhinavpola]

Slicing, shuffling, etc. don't really make sense to me on a stream. I would be okay with not having any of those features and requiring buffering the whole stream via something like StreamingDataset.to_memory_dataset() or something to make the conversion to a MemoryDataset explicit. We can still get limit to work but probably not anything else that requires length/indexing. As for multiple epochs, I think there isn't anything to cache so we'd have to rebuild the dataset (whatever HF decides to store locally for a streaming dataset, unsure of the details here). Something like StreamingDataset.reload() to reload the dataset if each epoch needs a fresh iterator.

I understand this would add a fair bit of branching. I need this anyway so I'm happy to hack around it for my use case.