[Reimplementation] Cannot use GPU combinations that do not start from gpu0, such as gpu2,3 when using dist_train.sh

[galliotxy]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• I have read the FAQ documentation but cannot get the expected help.
• The bug has not been fixed in the latest version (master) or latest version (3.x).

💬 Describe the reimplementation questions

When I use dist_train.sh for multi-gpu training on one computer using commands:
bash mmdetection/tools/dist_train.sh configs/cfg.py 2
the gpu_ids = [2,3] in cfg.py does not work，
it seems that in train.py, the cfg.gpu_ids is forced to be re-set to [0,1] according to code:

if args.launcher == 'none':
    distributed = False
else:
    distributed = True
    init_dist(args.launcher, **cfg.dist_params)
    # re-set gpu_ids with distributed training mode
    _, world_size = get_dist_info()
    cfg.gpu_ids = range(world_size)

Is this forced assignment necessary? How can I use my GPU2 and GPU3 for the 2-gpu training？

Environment

mmdet==2.25.0 mmcv==1.6.0 python==3.7.10 pytorch==1.8.1
it seems that the re-set operation still exists in the latest version

Expected results

No response

Additional information

No response

[ZwwWayne]

You should set CUDA_VISIBLE_DEVICES=2,3,4 when running the dist_train.sh

[ZwwWayne]

See examples in https://mmdetection.readthedocs.io/en/latest/1_exist_data_model.html#training-on-multiple-gpus

[galliotxy]

    您好，您的邮件已经收到。我会尽快查收的~