Training without epochs

[miccio-dk]

Hi, everyone!
I have a use case where I'd like to train a model on a continuous flow of randomly generated data and perform validation every e.g. 1000 steps instead of every epoch. Thus, everything would run within one epoch, and the length of the training would be specified in steps.

Is this doable with pytorch-lightning?
One way I can think of would be to set the dataset __length__() to the number of steps between validation cycles and then pick the data randomly (disregarding the index argument in the dataset __getitem().

[rohitgr7]

hey @miccio-dk

I think you can use Trainer(max_steps=...) for this and configure a PT Iterable dataset that can generate data on the fly and configure validation with Trainer(val_check_interval=1000). Maybe give it a try.

[miccio-dk]

that's very helpful, I totally missed these two arguments. Thank you!!

I'll try to implement a toy example but have a couple of doubts.
Right now I'm using a regular pytorch Dataset class, so __length__() is called during training (by e.g. _reset_eval_dataloader). What would happen when it's no longer possible to access the dataset length?
Also, how does the dataloader handle iterable datasets (e.g. shuffling, multi-threading)?

[rohitgr7]

in your use case, since the dataset is not fixed and there's no length, I'd recommend using IterableDataset. I don't think shuffling is possible with an iterable dataset because there is no fixed length.

[tchaton]

Yes, when the size isn't fixed, here are. the consequences. The progress tracking doesn't know when the end of the epoch would happen, but when your dataloader exhausts entirely the iterable, then epoch end would trigger.

If your dataloader never finishes, then the epoch end hook would never be triggered, which might be an issue for you to check the metrics. From there, you have 2 choices: create fake epochs or manipulate metrics to compute metrics over a period of steps.

class Model(LightningModule):

    def __init__(self):
        ...

        self.acc = Accuracy()


    def training_step(self, batch, batch_idx): # batch_idx range in [0, infinity]
        loss = ...

        # compute metric over 1k batches and log the value
        self.acc(...)
        if batch_idx % 1000 == 0:
            self.log(self.acc.compute())
            self.acc.reset()
        return loss