aistudio BML环境启用混合精度训练报错

[kwanxian]

请提供下述完整信息以便快速定位问题/Please provide the following information to quickly locate the problem

• 系统环境/System Environment：aistudio BML
• 版本号/Version：Paddle：2.5.1-gpupost112 PaddleOCR：2.7 问题相关组件/Related components：启用AMP混合精度训练报错
• 运行指令/Command Code：python3 tools/train.py -c train_data/config/en_PP-OCRv4_rec.yml -o Global.use_amp=True Global.scale_loss=1024.0 Global.use_dynamic_loss_scaling=True
• 完整报错/Complete Error Message：

我们提供了AceIssueSolver来帮助你解答问题，你是否想要它来解答(请填写yes/no)?/We provide AceIssueSolver to solve issues, do you want it? (Please write yes/no):

请尽量不要包含图片在问题中/Please try to not include the image in the issue.

Tue Sep  5 09:11:20 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:04:00.0 Off |                    0 |
| N/A   37C    P0    41W / 300W |      0MiB / 16160MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

[2023/09/05 09:14:29] ppocr INFO: train with paddle 2.5.1 and device Place(gpu:0)
[2023/09/05 09:14:29] ppocr INFO: Initialize indexs of datasets:['./train_data/train/Label.txt']
[2023/09/05 09:14:29] ppocr INFO: Initialize indexs of datasets:['./train_data/val/Label.txt']
W0905 09:14:29.766885  4502 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.6
W0905 09:14:29.774154  4502 gpu_resources.cc:149] device: 0, cuDNN Version: 8.4.
[2023/09/05 09:14:32] ppocr INFO: train dataloader has 266 iters
[2023/09/05 09:14:32] ppocr INFO: valid dataloader has 476 iters
[2023/09/05 09:14:32] ppocr INFO: load pretrain successful from ./train_data/model/en_PP-OCRv3_det_train/best_accuracy
[2023/09/05 09:14:32] ppocr INFO: During the training process, after the 50th iteration, an evaluation is run every 100 iterations
Traceback (most recent call last):
  File "/home/aistudio/PaddleOCR/tools/train.py", line 227, in <module>
    main(config, device, logger, vdl_writer)
  File "/home/aistudio/PaddleOCR/tools/train.py", line 198, in main
    program.train(config, train_dataloader, valid_dataloader, device, model,
  File "/home/aistudio/PaddleOCR/tools/program.py", line 292, in train
    preds = model(images)
  File "/home/aistudio/PaddleOCR/.venv/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1254, in __call__
    return self.forward(*inputs, **kwargs)
  File "/home/aistudio/PaddleOCR/ppocr/modeling/architectures/distillation_model.py", line 59, in forward
    result_dict[model_name] = self.model_list[idx](x, data)
  File "/home/aistudio/PaddleOCR/.venv/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1254, in __call__
    return self.forward(*inputs, **kwargs)
  File "/home/aistudio/PaddleOCR/ppocr/modeling/architectures/base_model.py", line 86, in forward
    x = self.backbone(x)
  File "/home/aistudio/PaddleOCR/.venv/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1254, in __call__
    return self.forward(*inputs, **kwargs)
  File "/home/aistudio/PaddleOCR/ppocr/modeling/backbones/det_mobilenet_v3.py", line 146, in forward
    x = self.conv(x)
  File "/home/aistudio/PaddleOCR/.venv/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1254, in __call__
    return self.forward(*inputs, **kwargs)
  File "/home/aistudio/PaddleOCR/ppocr/modeling/backbones/det_mobilenet_v3.py", line 179, in forward
    x = self.conv(x)
  File "/home/aistudio/PaddleOCR/.venv/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 1254, in __call__
    return self.forward(*inputs, **kwargs)
  File "/home/aistudio/PaddleOCR/.venv/lib/python3.10/site-packages/paddle/nn/layer/conv.py", line 710, in forward
    out = F.conv._conv_nd(
  File "/home/aistudio/PaddleOCR/.venv/lib/python3.10/site-packages/paddle/nn/functional/conv.py", line 133, in _conv_nd
    pre_bias = _C_ops.conv2d(
OSError: (External) CUDNN error(6), CUDNN_STATUS_ARCH_MISMATCH. 
  [Hint: 'CUDNN_STATUS_ARCH_MISMATCH'.  The function requires a feature absent from the current GPU device. Note that cuDNN only supports devices with compute capabilities greater than or equal to 3.0.  To correct, compile and run the application on a device with appropriate compute capability.  ] (at ../paddle/phi/kernels/gpudnn/conv_cudnn_v7.h:121)