Detection model resnet18 or resnet50 does not work with GPU

[suheylkiris]

Hi, i try to train a detection model on my own dataset. I tried multiple times with different backbones on both CPU and GPU. All of the backbones work with CPU but some of them does not work on GPU without any ERROR.

MobileNetV3_large_x0_5 works with CPU and GPU without any problem.
BUT
ResNet18_vd and ResNet50_vd_ssld works on CPU but does not work on GPU. I set the config files for each backbones. When usegpu=false, each of them works correctly. But when i change usegpu=true, two of them does not work. I put the logs below. After this, iterations will not start and it closes.

My anaconda prompt screen:

(paddle_last) C:\Users****\Desktop\Paddle\PaddleOCR\PaddleOCR>python tools/train.py -c configs/det_res18_db_v2.0.yml

.
someconfigparameters...
.
[2023/02/11 23:47:37] ppocr INFO: keep_keys : ['image', 'threshold_map', 'threshold_mask', 'shrink_map', 'shrink_mask']
[2023/02/11 23:47:37] ppocr INFO: loader :
[2023/02/11 23:47:37] ppocr INFO: batch_size_per_card : 8
[2023/02/11 23:47:37] ppocr INFO: drop_last : False
[2023/02/11 23:47:37] ppocr INFO: num_workers : 0
[2023/02/11 23:47:37] ppocr INFO: shuffle : True
[2023/02/11 23:47:37] ppocr INFO: profiler_options : None
[2023/02/11 23:47:37] ppocr INFO: train with paddle 2.4.1 and device Place(gpu:0)
[2023/02/11 23:47:37] ppocr INFO: Initialize indexs of datasets:['./train_data/2/real_products.txt']
[2023/02/11 23:47:37] ppocr INFO: Initialize indexs of datasets:['./train_data/2/real_products.txt']
W0211 23:47:37.292974 5272 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 12.0, Runtime API Version: 10.2
W0211 23:47:37.540496 5272 gpu_resources.cc:91] device: 0, cuDNN Version: 7.6.
[2023/02/11 23:47:38] ppocr INFO: train dataloader has 75 iters
[2023/02/11 23:47:38] ppocr INFO: valid dataloader has 600 iters
[2023/02/11 23:47:39] ppocr INFO: load pretrain successful from ./pretrain_models/ResNet18_vd_pretrained
[2023/02/11 23:47:39] ppocr INFO: During the training process, after the 300th iteration, an evaluation is run every 200 iterations

(paddle_last) C:\Users*****\Desktop\Paddle\PaddleOCR\PaddleOCR>

---

It stops without any reason. When i checked on nvidia-smi, GPU was busy and process is shown on the screen.
I will repeat again, this problem just occurs when use ResNet. When i train with MobileNet on GPU, there is no problem.

Could anybody help me about this problem?

[LDOUBLEV]

MobileNetV3_large_x0_5 works with CPU and GPU without any problem.
BUT
ResNet18_vd and ResNet50_vd_ssld works on CPU but does not work on GPU.

Maybe your GPU memory is too small to train larger models.
The memory occupied by MV3backbone is much smaller than that of ResNet.

[2023/02/11 23:47:37] ppocr INFO: batch_size_per_card : 8

Try to set a smaller batchsize. emmmm, 4 or 2.

[suheylkiris]

It is possible. I used Nvidia GTX 1650Ti for train which is older GPU. I will set smaller batch size, i hope it will works.

[suheylkiris]

Yes, you are right! This problem occurs due to small GPU memory. When batch size changed, it works! Thank you so much.

[LDOUBLEV]

Hope PaddleOCR can help you.
Please mark the discussion as answered if there are no more questions.