fetch data in a non-epoch way

[YingdiZhang]

Thanks for the great job.
I wonder when trainnig a network，if I could fetch batch data without considering the epoch parameter. It means in each iteration，a batch of data is extracted from dataset and dont need to set the traning epoch.

[Nic-Ma]

Hi @ericspod ,

Could you please help share some training skills about this question?
I feel it can be achieved by some Iterable dataset?

Thanks in advance.

[ericspod]

What I'm guessing you want is to have a single epoch for training which extracts a set number of batches from you dataset with duplication so that the total up to more data than what's in your dataset. What I've done in the past for this is use Pytorch's samplers with the data loader to define this randomised duplication sampling, allowing me to define epochs of however many iterations I wanted:

train_ds = # training dataset
batch_size = 100  # number of items in batches
num_iters = 400  # number of iterations in an epoch

rs=torch.utils.data.RandomSampler(train_ds, replacement=True, num_samples=num_iters * batch_size)

train_loader = monai.data.DataLoader(train_ds, batch_size=batch_size, num_workers=8, sampler=rs)

train_loader will now produce 400 iterations per epoch totally 40,000 samples, if the dataset is smaller than this then duplication will occur which is fine with random augmentations applying different transformations to each duplicate.

[YingdiZhang]

Thanks for your answer, I wonder if I can use a Distributed sampler to achieve this, which I reference the MONAI tutorial from https://github.com/Project-MONAI/tutorials/blob/34d44807bbde2418633fa07f51c54c8c75603a1c/acceleration/distributed_training/unet_training_ddp.py#L116

[ericspod]

I don't think so, the DistributedSampler only partitions data amongst processes in a fixed deterministic way assuming the dataset is not changed.