Cont. Failed to Resume Training w/ CombinedStreamingDataset

[karinazad]

🐛 Bug

I'm encountering the same issue as #363. I have the latest litdata version 0.2.52

ValueError: The provided `num_samples_yielded` state is greater than the dataset length. Found `10566743` instead of `9017936`.

Expected behavior

Resume training from a checkpoint

Additional context

Environment detail

• PyTorch Version (e.g., 1.0): 2.8
• OS (e.g., Linux): Linux
• How you installed PyTorch (conda, pip, source): uv
• Python version: 12.9
• GPU models and configuration: 40 nodes, 8 B200s per node (observed also on H100s)
• CUDA/cuDNN version: 13

[karinazad]

Oh, when I started training, my litdata version was lower, 0.2.50 - is there any way to resume that run?

[bhimrazy]

Hi @karinazad, thanks for reporting this!

The versions should be fine, as there haven’t been any major changes to streaming or combined datasets between 0.2.50 and 0.2.52.

To better understand the issue, could you share a bit more details about your setup and configuration (or a minimal reproducible code example, if possible)?

In the meantime, you might try overwriting the state manually. The related logic was introduced in PR #497

[karinazad]

Thanks for your quick reply. I'm traning with 5 combined datasets and can resume without any issues if training for less than one epoch but after that, it gives me the error.

I'm already using the force override state option.

[bhimrazy]

Thanks for the info! I’ll try to see if I can reproduce this. We also have a test case for resuming a CombinedStreamingDataset with two datasets here.

btw to confirm — could this be related to weighting in the CombinedStreamingDataset by any chance?

[karinazad]

Thanks a lot for looking into it. I'm not using weights for this run.

I might close this issue for now until I have a better example that can be reproduced, since the test cases cover my case