DynUNet Parallel training

[bip5]

Hi guys, I've been trying to use the multi_gpu option since yesterday. So far I've had a few errors in the process of trying to fix it- this is one of them

Traceback (most recent call last):
File "/scratch/a.bip5/BraTS 2021/nnUNet/./nnu.py", line 1187, in
train(args)
File "/scratch/a.bip5/BraTS 2021/nnUNet/./nnu.py", line 896, in train
dist.init_process_group(backend="nccl",rank=dist.get_rank(), init_method="env://",world_size=dist.get_world_size())
File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 725, in get_rank
default_pg = _get_default_group()
File "/home/a.bip5/.conda/envs/pix2pix/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 358, in _get_default_group
raise RuntimeError("Default process group has not been initialized, "
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

Is the DynUNet pipeline in the tutorial working correctly for distributed training ? Is there anything that I need to add to the code?

Thank you

[Nic-Ma]

Hi @yiheng-wang-nv ,

Could you please help share some best practices for this question? I remember there is some PyTorch parameters we need to set for mulit-gpu?

Thanks in advance.

[yiheng-wang-nv]

What is your command for ddp training?
you can refer to https://github.com/Project-MONAI/tutorials/blob/b429c883dccde3bfaa16c7c42fc2a35a8157fc3f/modules/dynunet_pipeline/commands/task04/train_multi_gpu.sh

[bip5]

The transforms are from the tutorial.

def get_task_transforms(mode, task_id, pos_sample_num, neg_sample_num, num_samples):
    if mode != "test":
        keys = ["image", "label"]
    else:
        keys = ["image"]

    load_transforms = [
        LoadImaged(keys=keys),
        EnsureChannelFirstd(keys=keys),
    ]
    # 2. sampling
    sample_transforms = [
        PreprocessAnisotropic(
            keys=keys,
            clip_values=clip_values[task_id],
            pixdim=spacing[task_id],
            normalize_values=normalize_values[task_id],
            model_mode=mode,
        ),
    ]
    # 3. spatial transforms
    if mode == "train":
        other_transforms = [
            SpatialPadd(keys=["image", "label"], spatial_size=patch_size[task_id]),
            RandCropByPosNegLabeld(
                keys=["image", "label"],
                label_key="label",
                spatial_size=patch_size[task_id],
                pos=pos_sample_num,
                neg=neg_sample_num,
                num_samples=num_samples,
                image_key="image",
                image_threshold=0,
            ),
            RandZoomd(
                keys=["image", "label"],
                min_zoom=0.9,
                max_zoom=1.2,
                mode=("trilinear", "nearest"),
                align_corners=(True, None),
                prob=0.15,
            ),
            RandGaussianNoised(keys=["image"], std=0.01, prob=0.15),
            RandGaussianSmoothd(
                keys=["image"],
                sigma_x=(0.5, 1.15),
                sigma_y=(0.5, 1.15),
                sigma_z=(0.5, 1.15),
                prob=0.15,
            ),
            RandScaleIntensityd(keys=["image"], factors=0.3, prob=0.15),
            RandFlipd(["image", "label"], spatial_axis=[0], prob=0.5),
            RandFlipd(["image", "label"], spatial_axis=[1], prob=0.5),
            RandFlipd(["image", "label"], spatial_axis=[2], prob=0.5),
            CastToTyped(keys=["image", "label"], dtype=(np.float32, np.uint8)),
            EnsureTyped(keys=["image", "label"]),
        ]
    elif mode == "validation":
        other_transforms = [
            CastToTyped(keys=["image", "label"], dtype=(np.float32, np.uint8)),
            EnsureTyped(keys=["image", "label"]),
        ]
    else:
        other_transforms = [
            CastToTyped(keys=["image"], dtype=(np.float32)),
            EnsureTyped(keys=["image"]),
        ]

    all_transforms = load_transforms + sample_transforms + other_transforms
    return Compose(all_transforms)

[bip5]

Any ideas on why the parallel version has the size issue and the original doesn't?

[yiheng-wang-nv]

Hi @bip5 , sorry for mis-read your last response. Could you please double check if the resize transform PreprocessAnisotropic works? Since you are using a new dataset, the corresponding parameters should be correct. Especially for the spacing, does all images have the same spacings?

For the parallel issue, I'm not sure since for all 10 decathlon datasets, I did not meet the same issue. If possible, you can also provide an example data, thus I'm able to reproduce your issue first. Thanks!

[bip5]

In that case, I'll try with the brain tumour task from the decathlon dataset to see if the parallel version works for that. I'll let you if I encounter the same issue. Thank you

The images should have the same spacing (as far as I understand)

[bip5]

It turns out I made a mistake when porting the code into a single file. The parallel training is working with the decathlon data just fine. Thank you for the replies.