The convergence speed of 3D CNN training is too slow

[lanslotttTT]

Thanks for your gerat job,I foud that when I not use CropForeground in my transform ，my network doesn't seem to be converging, the only effect was to increase the number of training epochs, which seemed unacceptable for actual production situation.How can this problem be solved？

[Nic-Ma]

Hi @lanslotttTT ,

I think there are possible reasons that affect a training process, maybe you can share your training program and some loss / metrics curves for further analysis?
@ericspod @rijobro @wyli @dongyang0122 are research experts on this topic.

Thanks.

[lanslotttTT]

Our training data preprocessing is like the following, we found that after removing the CropForegroundd ,the network seems unable to optimize

            [
                LoadImaged(keys=["image", "label"]),
                AddChanneld(keys=["image", "label"]),
                Orientationd(keys=["image", "label"], axcodes="RAS"),
                Spacingd(
                    keys=["image", "label"],
                    pixdim=(1.5, 1.5, 2.0),
                    mode=("bilinear", "nearest"),
                ),
                FromMetaTensord(keys=["image", "label"]),
                ScaleIntensityRanged(
                    keys=["image"], a_min=-57, a_max=164,
                    b_min=0.0, b_max=1.0, clip=True,
                ),
                CropForegroundd(keys=["image", "label"], source_key="image"),
                # randomly crop out patch samples from
                # big image based on pos / neg ratio
                # the image centers of negative samples
                # must be in valid image area
                RandCropByPosNegLabeld(
                    keys=["image", "label"],
                    label_key="label",
                    spatial_size=(96, 96, 96),
                    pos=1,
                    neg=1,
                    num_samples=4,
                    image_key="image",
                    image_threshold=0,
                ),
                # user can also add other random transforms
                #                 RandAffined(
                #                     keys=['image', 'label'],
                #                     mode=('bilinear', 'nearest'),
                #                     prob=1.0,
                #                     spatial_size=(96, 96, 96),
                #                     rotate_range=(0, 0, np.pi/15),
                #                     scale_range=(0.1, 0.1, 0.1)),
                EnsureTyped(keys=["image", "label"]),
            ]
        )```

[lanslotttTT]

Here is our loss curve, we use dice loss,The curve shows that the network is not fully trained. The network appears to be randomly assigned 0 and 1 values with a probability of 0.5

[rijobro]

Hi, I don't think you should need to use FromMetaTensord unless you're using the feature/MetaTensor branch.

Are you saying that with CropForegroundd, you converge in what you consider to be an acceptable number of epochs, but when you remove CropForegroundd, it doesn't converge?

If that is the case, what is the rationale for wanting to remove CropForegroundd?

Also if one transform is having such a big difference, does that imply that the majority of your images are background?

[lanslotttTT]

Thanks for your reply! Yes，our training data is composed of small objects (artery tumor), the training process can converge quickly in the foreground, but some objects in the background will be judged as segmentation targets in the prediction. We wanted to add background to the training process, but it seemed that it's difficult for traning

[dongyang0122]

What are the issues in the predictions, false positives or false negatives? How large is gap between the prediction and your expected performance?

[dongyang0122]

If the task is tumor segmentation, be careful with Spacingd transform. You do not want to have a very coarse re-sampling, otherwise the foreground regions would be gone in the re-sampled ground truth masks, which makes training much harder.

[ericspod]

Our experience with segmenting small objects is that the task is just hard and takes time to get a good result when training on patches. If you're not cropping by foreground of course you'll get more patches that don't have tumour in them, and so the network has less chance to learn what to look for since it's fed so many negative cases with no label. My experience is that training a segmentation network with images having no labels (background) makes for poor results, but then you have the problem of what to do in the background areas, so perhaps a strategy of starting out with cropping to the foreground initially then introducing more background images over time will get to better results. It's hard to say anything more specific without seeing your data though our colleagues have worked on small object segmentation and how to evaluate the results in "Lesion-wise evaluation for effective performance monitoring of small object segmentation" which might be helpful to you.

[lanslotttTT]

Most of them are false positives. I expect dice score to be 0.8 above，but now I just get 0.1 dice score

…

---- Replied Message ---- | From | ***@***.***> | | Date | 05/25/2022 00:02 | | To | ***@***.***> | | Cc | ***@***.******@***.***> | | Subject | Re: [Project-MONAI/MONAI] The convergence speed of 3D CNN training is too slow (Discussion #4328) | What are the issues in the predictions, false positives or false negatives? How large is gap between the prediction and your expected performance? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[dongyang0122]

also check if the intensity range works for your data with some visualization tools (e.g., ITK-Snap).

[lanslotttTT]

Thank you for your advice. Our next plan is to increase the number of training epoches to test whether the network can converge or not.

…

---- Replied Message ---- | From | ***@***.***> | | Date | 05/25/2022 00:05 | | To | ***@***.***> | | Cc | ***@***.******@***.***> | | Subject | Re: [Project-MONAI/MONAI] The convergence speed of 3D CNN training is too slow (Discussion #4328) | If the task is tumor segmentation, be careful with Spacingd transform. You do not want to have a very coarse re-sampling, otherwise the foreground regions would be gone in the re-sampled ground truth masks, which makes training much harder. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>