this is great! any bias in the sampling considerations?

[aldopareja]

just wondering if packing in this way could represent a bias in the aggregated batches (across all GPUs) when compared to simple random sampling. Ideally you want the probability of any sample being in a batch being the same as random sampling, but I can't really figure the math behind it.

[imoneoi]

Multipack should not have any bias compared to random sampling. The algorithm is equivalent to randomly shuffling the data, then retrieving one by one from the shuffled data and pack into the batch until full.

As long as the order inside a batch doesn't matter, Multipack is equivalent to random sampling, with one notable exception that the batch size is dynamic.