Dataset changes Length: How to use the Trainer

[hg6185]

Hello,

I am using a custom dataset class that changes its length during the training. It gets longer with each getitem-call.
However, when I instantiate a DataLoader and set persistent_workers=True, indeed, the dataset objects remains (and doesn't get reseted after each epoch), but also the initial length of the DataLoader:

Example:
Initial dataset length: 4 -----> iter(dataloader)=4
After 1 Epoch: dataset length: 8 -------> iter(dataloader)=4

Question:
Is there a way to use the trainer such that it keeps the dataset-instance, but rebuilds dataloaders at every epoch, such that they have the new dataset length?

Beside just using persistent dataloaders, I tried:

class CustomDataModule(LightningDataModule):
    def __init__(self, dataset, val_loader) -> None:
        super().__init__()
        self.dataset = dataset
        self.val_loader = val_loader

    def train_dataloader(self):
        return DataLoader(self.dataset, batch_size=32, num_workers=2, persistent_workers=True)

This does not work, because the dataset remains in the initial condition, probably untouched through the design patterns of the fabric?

Also: I used the CustomTrainer available here: https://github.com/Lightning-AI/lightning/blob/master/examples/fabric/build_your_own_trainer/trainer.py

and add self.dataset and instatiated the DataLoader at the beginning of each epoch. However the reference to the dataset is lost too.

What can I do? The DataLoader needs to keep the reference to the dataset object?