does each worker (num_workers>1) has its own instance of the Dataset object?

[malfonsoarquimea]

I am currently working on a code where the dataset has one property that

1. has a custom Dataset which has an attribute at self.items_served that counts how many items have been served. It is initialized at 0 in the init() method and uptadted every time get_item() is called
2. on my LightningModule's init(), I initialize one instance of this custom Dataset and assign it to an attribute of the class: d= Dataset(args) and then self.train_dataset=d
3. on my LightningModule's train_dataloader(), I return a dataloader which has self.train_dataset as dataset and numworkers = 10
4. on my LightningModule's training_step(), I try to check the attribute at self.train_dataset.items_served and it is always 0, it does not update.

However, everything works fine if I use only one worker (numworkers=0), so I guess I a doing something wrong and/or I don't understand how dataloader works with multiple workers

Any suggestion on how to tackle this problem?

[rohitgr7]

It's because when you use num_workers> 0, the data is loaded, and the collate_fn is called inside the worker and that fetched data is pushed to the main process. So all the updates are happening inside the sub-worker process, but training_step runs on the main process, so no effect is reflected there.

potential sol you can try: https://stackoverflow.com/questions/63460992/how-do-i-fix-the-dataset-to-return-desired-output-pytorch

[malfonsoarquimea]

thanks very much, that is what I was looking for!