numclass must be divided by num_gpu?

[cavalleria]

torchshard/utils.py", line 2, in check_divisibility assert numerator % denominator == 0, \ AssertionError: 85742 is not divisible by 8

[kaiyuyue]

Sorry for my late response.

I force this division manner because I think evenly-divided number of classes can promise every GPU has the same memory occupation and cost. This can ease users to (1) debug additional and unexpected memory occupation caused by some kind of poor codes and abnormal coding issues; (2) benchmark out accurate statistics, such as memory cost, running speed and so on. It’s just like using PyTorch debugging way without considering the uneven memory occupation caused by TorchShard if we force the division manner.

Additionally, setting a class number that can be divided by the GPU number is easy, quick and friendly for development. It’s like setting channel numbers, for example 64 for stage1, 128 for stage2, and 1024 for stage3 of ResNet. All these numbers can be divided by 8, which is a common GPU number. If asking why not using 60 instead of 64 for stage1? Well, it can be but setting 64 is a kind of common rule in our community because we cannot tell the difference between setting those two numbers. So changing the class number 85742 into 85744 does make sense because it has no effects on the softmax function, cross-entropy loss and final inference. This revision in class number was mentioned in the tutorial of arcface shard training.

But yeah, it absolutely can let each shard layer hosts the different number of partial classes. To get rid of this restriction, you just need to comment the corresponding assert code and make sure the right input number of each shard ts.nn.Linear when building your networks.

[kaiyuyue]

We can keep this discussion being OPEN for newcomers because this is a valuable issue for reference.