DPA-2 does not support multi-card invocation.

[wangyi01]

I was trying to utilize multiple cards when using DPA-2, but in practice, only one card was being invoked, leading to insufficient memory. I have four GPUs with 16GB each. However, it indicates that the memory of the first GPU is insufficient, suggesting that only the first GPU is being utilized. So why doesn't DPA-2 support multi-card operation?

[wanghan-iapcm]

#3657

[wangyi01]

I used torchrun to implement multi-GPU invocation, but there was a conflict between the -m parameter inside and the -m parameter in the dp --pt parameter. Specifically, I used: torchrun --no_python --nproc_per_node=1 --nnode=4 dp --pt train input.json --finetune ./pretrained_model.pt -m Domains_OC2M --skip-neighbor-stat. I encountered the following error message: error: argument -m/--mpi-log: invalid choice: 'Domains_OC2M' (choose from 'master', 'collect', 'workers').

[iProzd]

@wangyi01 You should use --model-branch instead of -m (which is short for --mpi-log) during finetuning.