OOM error when executing the bash scripts/train_backward_Myx.sh

[shamanez]

I am using : 8 * A100 40GB

[Spico197]

Hi there. Sorry for the late response. I run the experiment with 4*A100 40GB and didn't encounter the OOM problem. Did you apply the flash-attention?

[lss11005]

I also encountered this problem,I test on 4*A100 40GB, set
num_nodes=1
num_gpu_per_node=4
bsz=16
is there any problems？