Dynamic adjusting dataloader sampling weight

[whc688]

Hi,
Thanks for your wonderful work! I'm wondering is there a way to dynamically adjust the sampling weight of each data in the dataset during training? Looking forward to your reply, thanks again.

[lhoestq]

You mean based on a condition that has to be checked on-the-fly during training ? Otherwise if you know in advance after how many samples you need to change the sampling you can simply concatenate the two mixes

[whc688]

Yes, like during training, if one data sample's prediction is consistently wrong, its sampling weight gets higher and higher, and if one data sample's prediction is already correct, then we rarely sample it

[lhoestq]

it's not possible to use interleave_datasets() and modify the probabilities while iterating on the dataset at the moment, so you'd have to implement your own torch Sampler or your ownIterableDataset to implement this logic