Batch size 1 gradient accumulation

[mfarazi1991]

Hello,

I have an issue with variable-size graphs/meshes. So I need to include some matrices with my PyG data loader but obviously, I cannot as the matrices have different sizes like N by N or M by M. I try to use batch size one with gradient accumulation but the models I use overfit very fast (learning rate is handled) and it should not. Any idea for the root cause? for images and pytorch the gradient accumulation works rather fine but it seems for graphs not maybe?

[rusty1s]

Is the question more related to how to use mini-batching here? I think you have two ways for that:

1. Pad the dense matrices to equal shape
2. Represent them as nested tensors (fully supported as part of our DataLoader)

Besides that, it is hard to give a good answer here why gradient accumulation wouldn't work here. I don't think this is necessarily an issue with graph-based data.