chunkable datasets and dataloaders #20576
Description
Description & Motivation
Current large model training requires a huge number of traning samples, so that the traditional Mapped dataloaders are failed to load the training data because of limited memory. so I think there can be a chunkable datasets and dataloaders available. so that we can use the Mapped logic to load the first subset of training data in the dataset and dataloader 1 for training and the loading the second subset of training data for prepare.
Pitch
I had seen the CombinedLoader in lighting, and I did not find too much documents and examples about it, It seems that it can not resolve the requirement about "loading large traning data in memory chunk by chunk". If there had been some resolution, please give me some help, thanks!
Alternatives
No response
Additional context
No response