information regarding training lung cancer nodule detection model on multiple or single GPUs

[DSRajesh]

Hello
It is required to train a 3D object detection model (using approximately around 20000 3D images) for which we want to use MONAI. We just wanted to know if it is possible to achieve the following:

Use 2 GPUs of GE Force RTX 4090
or
use a single GPU i.e GE Force RTX 6000
choice 1 has more number of cores but less memory and choice 2 has more memory but less number of cores.

It would be helpful to proceed ahead if we come to know the feasibility of both above approaches and also the better of these two choices. Or if there are other better options, it would be helpful too...

Thanking You

Rajesh

[DSRajesh]

Also

Given a task of detecting small objects from Lung CT such as lung nodules, from over 10000 scans what kind of a GPU might work best ?

A. High VRAM and moderate compute speed - the old A6000 48GB, 10752 CUDA cores, 1.45GHz clock

or

(B) High speed and lower VRAM - RTX 4090 24GB, 16384 CUDA cores, 2.2GHz clock

What are the subtleties involved in the nndetect parameters in training with a very large number of datasets. eg. Batch size, etc.

Would we push the memory limits on a RTX4090 ?

Would MONAI RetinaNet detector support multi-GPU or multiple workstations for detection with federated learning etc ?

Thanking You

Rajesh

[KumoLiu]

Hi @DSRajesh, thanks for the interest here.
We didn't try the RetinaNet detector with multi-GPU, could you please try it on your side?

• A single A6000 offers more GPU memory, which can be beneficial when handling large 3D volumes, especially in medical imaging tasks.
• Two RTX 4090 GPUs have more CUDA cores and may provide faster training due to their higher computing power.

There may be some points to consider:

1. Consider the size of your dataset. Larger datasets might benefit more from additional GPU memory.
2. Evaluate the training time requirements. If time is critical, the RTX 4090 GPUs may offer faster training.
3. Factor in budget constraints, as you mentioned that a single A6000 costs twice as much as an RTX 4090.

So the choice between a single A6000 and two RTX 4090 GPUs depends on your specific needs and budget. Careful consideration of these factors will help you make an informed decision for your nodule detection task.

You can also refer to the setting of the existing detection model which uses RetinaNet in the MONAI model zoo.
https://github.com/Project-MONAI/model-zoo/tree/dev/models/lung_nodule_ct_detection

If you have more detailed questions or need further assistance, feel free to ask.

[DSRajesh]

Thank you for your answer. As I understand,

1. we have not yet tried RetinaNet detector on multiple GPUs to know that it works yet, although we hypothesize that it should.
2. This uses Distributed Data Parallelism as we understand so as to fully utilize the GPU VRAM in both GPUs and cores in both, splitting the data across both GPUs.

Is this right ? Does DDP affect model accuracy as compared to no DDP ?

Thanks

[KumoLiu]

Yes, Distributed Data Parallel (DDP) can have an impact on model accuracy when compared to training without DDP.