ppdet.engine WARNING: Evaluation results empty, this may be due to training iterations being too few or not loading the correct weights.

[FreemanTang]

问题确认 Search before asking

• 我已经查询历史issue，没有发现相似的bug。I have searched the issues and found no similar bug report.

Bug组件 Bug Component

No response

Bug描述 Describe the Bug

您好，请教一下，使用ppyoloe+训练的时候，修改了一些配置参数，第一次出现了这个ppdet.engine WARNING: Evaluation results empty, this may be due to training iterations being too few or not loading the correct weights.，第二次训练又正常了，这是什么原因？第一次和第二次训练配置参数都是一样的。

第一次训练报错：
ppdet.engine WARNING: Evaluation results empty, this may be due to training iterations being too few or not loading the correct weights.

第二次又正常了：

复现环境 Environment

• OS: Windows -
• PaddlePaddle: 2.4.2
• PaddleDetection: release/2.7.0
• Python: 3.10.18
• CUDA: 11.1
• CUDNN: 8.4

Bug描述确认 Bug description confirmation

• 我确认已经提供了Bug复现步骤、代码改动说明、以及环境信息，确认问题是可以复现的。I confirm that the bug replication steps, code change instructions, and environment information have been provided, and the problem can be reproduced.

是否愿意提交PR？ Are you willing to submit a PR?

• 我愿意提交PR！I'd like to help by submitting a PR!

[FreemanTang]

主要修改的配置如下：

[FreemanTang]

问题确认 Search before asking

• 我已经查询历史issue，没有发现相似的bug。I have searched the issues and found no similar bug report.

Bug组件 Bug Component

No response

Bug描述 Describe the Bug

您好，请教一下，使用ppyoloe+训练的时候，修改了一些配置参数，第一次出现了这个ppdet.engine WARNING: Evaluation results empty, this may be due to training iterations being too few or not loading the correct weights.，第二次训练又正常了，这是什么原因？第一次和第二次训练配置参数都是一样的。

第一次训练报错： ppdet.engine WARNING: Evaluation results empty, this may be due to training iterations being too few or not loading the correct weights.

第二次又正常了：

### 复现环境 Environment * OS: Windows - * PaddlePaddle: 2.4.2 * PaddleDetection: release/2.7.0 * Python: 3.10.18 * CUDA: 11.1 * CUDNN: 8.4

Bug描述确认 Bug description confirmation

• 我确认已经提供了Bug复现步骤、代码改动说明、以及环境信息，确认问题是可以复现的。I confirm that the bug replication steps, code change instructions, and environment information have been provided, and the problem can be reproduced.

是否愿意提交PR？ Are you willing to submit a PR?

• 我愿意提交PR！I'd like to help by submitting a PR!

经过我反复的debug，我找到了问题所在以及可能的解决方法。

正常地打开cmd，使用以下命令,有时正常，有时不正常。

python tools/train.py -c configs/ppyoloe/ppyoloe_plus_crn_l_80e_coco.yml --eval -o use_gpu=true

用管理员权限打开cmd，使用以下命令，目前测了都是正常的。

python -u tools/train.py -c configs/ppyoloe/ppyoloe_plus_crn_l_80e_coco.yml --eval -o use_gpu=true

具体原因还不清楚，目前我是使用-u去训练。暂时没有发现报ppdet.engine WARNING: Evaluation results empty, this may be due to training iterations being too few or not loading the correct weights.错

[FreemanTang]

问题确认 Search before asking

• 我已经查询历史issue，没有发现相似的bug。I have searched the issues and found no similar bug report.

Bug组件 Bug Component

No response

Bug描述 Describe the Bug

您好，请教一下，使用ppyoloe+训练的时候，修改了一些配置参数，第一次出现了这个ppdet.engine WARNING: Evaluation results empty, this may be due to training iterations being too few or not loading the correct weights.，第二次训练又正常了，这是什么原因？第一次和第二次训练配置参数都是一样的。
第一次训练报错： ppdet.engine WARNING: Evaluation results empty, this may be due to training iterations being too few or not loading the correct weights.
第二次又正常了：

复现环境 Environment

• OS: Windows -
• PaddlePaddle: 2.4.2
• PaddleDetection: release/2.7.0
• Python: 3.10.18
• CUDA: 11.1
• CUDNN: 8.4

Bug描述确认 Bug description confirmation

• 我确认已经提供了Bug复现步骤、代码改动说明、以及环境信息，确认问题是可以复现的。I confirm that the bug replication steps, code change instructions, and environment information have been provided, and the problem can be reproduced.

是否愿意提交PR？ Are you willing to submit a PR?

• 我愿意提交PR！I'd like to help by submitting a PR!

经过我反复的debug，我找到了问题所在以及可能的解决方法。

正常地打开cmd，使用以下命令,有时正常，有时不正常。

python tools/train.py -c configs/ppyoloe/ppyoloe_plus_crn_l_80e_coco.yml --eval -o use_gpu=true

用管理员权限打开cmd，使用以下命令，目前测了都是正常的。

python -u tools/train.py -c configs/ppyoloe/ppyoloe_plus_crn_l_80e_coco.yml --eval -o use_gpu=true

具体原因还不清楚，目前我是使用`-u`去训练。暂时没有发现报`ppdet.engine WARNING: Evaluation results empty, this may be due to training iterations being too few or not loading the correct weights.`错

又出现一些新问题了，

loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
[08/14 17:26:39] ppdet.data.source.coco INFO: Load [16 samples valid, 0 samples invalid] in file E:\user\mytest\master\storage\train\train20250814172409\trainning_dataset\coco\train.json.
W0814 17:26:39.637744 31572 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 11.1, Runtime API Version: 11.6
W0814 17:26:39.647708 31572 gpu_resources.cc:91] device: 0, cuDNN Version: 8.0.
W0814 17:26:39.647708 31572 gpu_resources.cc:117] WARNING: device: 0. The installed Paddle is compiled with CUDA 11.6, but CUDA runtime version in your machine is 11.1, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDA version.
[08/14 17:26:43] ppdet.utils.checkpoint INFO: The shape [365] in pretrained weight yolo_head.pred_cls.0.bias is unmatched with the shape [1] in model yolo_head.pred_cls.0.bias. And the weight yolo_head.pred_cls.0.bias will not be loaded
[08/14 17:26:43] ppdet.utils.checkpoint INFO: The shape [365, 768, 3, 3] in pretrained weight yolo_head.pred_cls.0.weight is unmatched with the shape [1, 768, 3, 3] in model yolo_head.pred_cls.0.weight. And the weight yolo_head.pred_cls.0.weight will not be loaded
[08/14 17:26:43] ppdet.utils.checkpoint INFO: The shape [365] in pretrained weight yolo_head.pred_cls.1.bias is unmatched with the shape [1] in model yolo_head.pred_cls.1.bias. And the weight yolo_head.pred_cls.1.bias will not be loaded
[08/14 17:26:43] ppdet.utils.checkpoint INFO: The shape [365, 384, 3, 3] in pretrained weight yolo_head.pred_cls.1.weight is unmatched with the shape [1, 384, 3, 3] in model yolo_head.pred_cls.1.weight. And the weight yolo_head.pred_cls.1.weight will not be loaded
[08/14 17:26:43] ppdet.utils.checkpoint INFO: The shape [365] in pretrained weight yolo_head.pred_cls.2.bias is unmatched with the shape [1] in model yolo_head.pred_cls.2.bias. And the weight yolo_head.pred_cls.2.bias will not be loaded
[08/14 17:26:43] ppdet.utils.checkpoint INFO: The shape [365, 192, 3, 3] in pretrained weight yolo_head.pred_cls.2.weight is unmatched with the shape [1, 192, 3, 3] in model yolo_head.pred_cls.2.weight. And the weight yolo_head.pred_cls.2.weight will not be loaded
[08/14 17:26:43] ppdet.utils.checkpoint INFO: Finish loading model weights: E:\user\mytest\master\ppyoloe_crn_l_obj365_pretrained.pdparams
W0814 17:26:43.579831 31572 gpu_resources.cc:217] WARNING: device:  . The installed Paddle is compiled with CUDNN 8.4, but CUDNN version in your machine is 8.0, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDNN version.
[08/14 17:26:45] ppdet.engine INFO: Epoch: [0] [0/8] learning_rate: 0.000000 loss: 4.578032 loss_cls: 2.198289 loss_iou: 0.592584 loss_dfl: 1.796569 loss_l1: 2.348179 eta: 2:35:08 batch_cost: 2.4242 data_cost: 0.0000 ips: 0.8250 images/s
[08/14 17:26:53] ppdet.engine INFO: Epoch: [1] [0/8] learning_rate: 0.000200 loss: 4.291834 loss_cls: 1.612181 loss_iou: 0.592584 loss_dfl: 1.527257 loss_l1: 2.210172 eta: 0:57:50 batch_cost: 0.9057 data_cost: 0.1761 ips: 2.2082 images/s
[08/14 17:27:06] ppdet.engine INFO: Epoch: [2] [0/8] learning_rate: 0.000400 loss: 4.167111 loss_cls: 1.061333 loss_iou: 0.592584 loss_dfl: 1.527257 loss_l1: 2.700724 eta: 1:14:13 batch_cost: 1.1647 data_cost: 0.5444 ips: 1.7172 images/s
[08/14 17:27:17] ppdet.engine INFO: Epoch: [3] [0/8] learning_rate: 0.000600 loss: 3.720631 loss_cls: 0.570342 loss_iou: 0.592584 loss_dfl: 1.510540 loss_l1: 3.147440 eta: 1:13:21 batch_cost: 1.1534 data_cost: 0.6384 ips: 1.7340 images/s
[08/14 17:27:25] ppdet.engine INFO: Epoch: [4] [0/8] learning_rate: 0.000800 loss: 3.846039 loss_cls: 0.698097 loss_iou: 0.618988 loss_dfl: 1.527257 loss_l1: 3.147440 eta: 1:08:35 batch_cost: 1.0807 data_cost: 0.6045 ips: 1.8506 images/s
[08/14 17:27:36] ppdet.utils.checkpoint INFO: Save checkpoint: E:\user\mytest\master\storage\train\train20250814172409\output\ppyoloe_plus_crn_l_80e_coco\model_final
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
[08/14 17:27:36] ppdet.data.source.coco INFO: Load [4 samples valid, 0 samples invalid] in file E:\user\mytest\master\storage\train\train20250814172409\trainning_dataset\coco\val.json.
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
[08/14 17:27:36] ppdet.engine INFO: Eval iter: 0
[08/14 17:27:36] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
[08/14 17:27:36] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=0.01s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.03s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
[08/14 17:27:36] ppdet.engine INFO: Total sample number: 4, average FPS: 7.185426325380798
[08/14 17:27:36] ppdet.engine INFO: Best test bbox ap is 0.000.
[08/14 17:27:38] ppdet.utils.checkpoint INFO: Save checkpoint: E:\user\mytest\master\storage\train\train20250814172409\output\ppyoloe_plus_crn_l_80e_coco\model_final
[08/14 17:27:39] ppdet.engine INFO: Epoch: [5] [0/8] learning_rate: 0.001000 loss: 4.125414 loss_cls: 0.826641 loss_iou: 0.657247 loss_dfl: 1.531641 loss_l1: 3.147440 eta: 1:05:52 batch_cost: 1.0401 data_cost: 0.5739 ips: 1.9230 images/s
[08/14 17:27:45] ppdet.engine INFO: Epoch: [6] [0/8] learning_rate: 0.001000 loss: 4.125414 loss_cls: 0.868579 loss_iou: 0.657247 loss_dfl: 1.527488 loss_l1: 3.259805 eta: 1:00:37 batch_cost: 0.9594 data_cost: 0.5057 ips: 2.0847 images/s
[08/14 17:27:58] ppdet.engine INFO: Epoch: [7] [0/8] learning_rate: 0.001000 loss: 4.128360 loss_cls: 1.001173 loss_iou: 0.658620 loss_dfl: 1.527488 loss_l1: 3.367609 eta: 1:04:06 batch_cost: 1.0166 data_cost: 0.5615 ips: 1.9674 images/s
[08/14 17:28:09] ppdet.engine INFO: Epoch: [8] [0/8] learning_rate: 0.001000 loss: 4.167111 loss_cls: 1.069529 loss_iou: 0.658620 loss_dfl: 1.527488 loss_l1: 3.471909 eta: 1:04:31 batch_cost: 1.0252 data_cost: 0.5750 ips: 1.9508 images/s
[08/14 17:28:18] ppdet.engine INFO: Epoch: [9] [0/8] learning_rate: 0.001000 loss: 4.242053 loss_cls: 1.117389 loss_iou: 0.658620 loss_dfl: 1.531641 loss_l1: 3.708829 eta: 1:03:35 batch_cost: 1.0125 data_cost: 0.5682 ips: 1.9753 images/s
[08/14 17:28:30] ppdet.utils.checkpoint INFO: Save checkpoint: E:\user\mytest\master\storage\train\train20250814172409\output\ppyoloe_plus_crn_l_80e_coco\model_final
[08/14 17:28:30] ppdet.engine INFO: Eval iter: 0
[08/14 17:28:30] ppdet.engine INFO: Total sample number: 4, average FPS: 15.893113862080202
[08/14 17:28:30] ppdet.engine WARNING: Evaluation results empty, this may be due to training iterations being too few or not loading the correct weights.
[08/14 17:28:31] ppdet.engine INFO: Epoch: [10] [0/8] learning_rate: 0.001000 loss: 4.269426 loss_cls: 1.283434 loss_iou: 0.668217 loss_dfl: 1.539893 loss_l1: 3.790606 eta: 1:03:46 batch_cost: 1.0177 data_cost: 0.5679 ips: 1.9651 images/s
[08/14 17:28:40] ppdet.engine INFO: Epoch: [11] [0/8] learning_rate: 0.001000 loss: 4.330941 loss_cls: 1.293206 loss_iou: 0.668217 loss_dfl: 1.606811 loss_l1: 3.790606 eta: 1:03:10 batch_cost: 1.0102 data_cost: 0.5649 ips: 1.9799 images/s
[08/14 17:28:50] ppdet.engine INFO: Epoch: [12] [0/8] learning_rate: 0.000999 loss: 4.503972 loss_cls: 1.506878 loss_iou: 0.687088 loss_dfl: 1.630966 loss_l1: 3.708829 eta: 1:02:46 batch_cost: 1.0060 data_cost: 0.5661 ips: 1.9881 images/s
[08/14 17:29:00] ppdet.engine INFO: Epoch: [13] [0/8] learning_rate: 0.000999 loss: 4.769010 loss_cls: 1.750995 loss_iou: 0.691123 loss_dfl: 1.619261 loss_l1: 3.672219 eta: 1:02:53 batch_cost: 1.0141 data_cost: 0.6005 ips: 1.9722 images/s
[08/14 17:29:09] ppdet.engine INFO: Epoch: [14] [0/8] learning_rate: 0.000999 loss: 5.729881 loss_cls: 2.777561 loss_iou: 0.697929 loss_dfl: 1.879496 loss_l1: 3.708584 eta: 1:02:09 batch_cost: 0.9693 data_cost: 0.5638 ips: 2.0633 images/s
[08/14 17:29:19] ppdet.utils.checkpoint INFO: Save checkpoint: E:\user\mytest\master\storage\train\train20250814172409\output\ppyoloe_plus_crn_l_80e_coco\model_final
[08/14 17:29:19] ppdet.engine INFO: Eval iter: 0
[08/14 17:29:19] ppdet.engine INFO: Total sample number: 4, average FPS: 13.604982942294109
[08/14 17:29:19] ppdet.engine WARNING: Evaluation results empty, this may be due to training iterations being too few or not loading the correct weights.
[08/14 17:29:20] ppdet.engine INFO: Epoch: [15] [0/8] learning_rate: 0.000999 loss: 7.059696 loss_cls: 3.822260 loss_iou: 0.707298 loss_dfl: 1.927931 loss_l1: 3.604548 eta: 1:01:00 batch_cost: 0.9481 data_cost: 0.5362 ips: 2.1094 images/s
[08/14 17:29:29] ppdet.engine INFO: Epoch: [16] [0/8] learning_rate: 0.000999 loss: 8.875206 loss_cls: 6.297404 loss_iou: 0.757171 loss_dfl: 1.968593 loss_l1: 3.563791 eta: 1:00:23 batch_cost: 0.9367 data_cost: 0.5184 ips: 2.1352 images/s
[08/14 17:29:40] ppdet.engine INFO: Epoch: [17] [0/8] learning_rate: 0.000998 loss: 12.002896 loss_cls: 8.529802 loss_iou: 0.770650 loss_dfl: 2.023767 loss_l1: 3.540546 eta: 1:00:47 batch_cost: 0.9500 data_cost: 0.5302 ips: 2.1052 images/s
[08/14 17:29:47] ppdet.engine INFO: Epoch: [18] [0/8] learning_rate: 0.000998 loss: 13.436544 loss_cls: 9.961493 loss_iou: 0.770650 loss_dfl: 2.051093 loss_l1: 3.463388 eta: 0:59:19 batch_cost: 0.9544 data_cost: 0.5345 ips: 2.0955 images/s
[08/14 17:29:55] ppdet.engine INFO: Epoch: [19] [0/8] learning_rate: 0.000998 loss: 14.687655 loss_cls: 10.840700 loss_iou: 0.767634 loss_dfl: 2.064041 loss_l1: 3.387630 eta: 0:58:32 batch_cost: 0.9323 data_cost: 0.5157 ips: 2.1452 images/s
[08/14 17:30:08] ppdet.utils.checkpoint INFO: Save checkpoint: E:\user\mytest\master\storage\train\train20250814172409\output\ppyoloe_plus_crn_l_80e_coco\model_final
[08/14 17:30:08] ppdet.engine INFO: Eval iter: 0
[08/14 17:30:09] ppdet.engine INFO: Total sample number: 4, average FPS: 14.697478675045094
[08/14 17:30:09] ppdet.engine WARNING: Evaluation results empty, this may be due to training iterations being too few or not loading the correct weights.
[08/14 17:30:09] ppdet.engine INFO: Epoch: [20] [0/8] learning_rate: 0.000998 loss: 16.549195 loss_cls: 13.154790 loss_iou: 0.743597 loss_dfl: 2.064041 loss_l1: 3.295240 eta: 0:59:07 batch_cost: 0.9406 data_cost: 0.5288 ips: 2.1263 images/s
[08/14 17:30:21] ppdet.engine INFO: Epoch: [21] [0/8] learning_rate: 0.000997 loss: 18.098598 loss_cls: 15.063297 loss_iou: 0.749230 loss_dfl: 2.012804 loss_l1: 3.225075 eta: 0:59:40 batch_cost: 0.9385 data_cost: 0.5234 ips: 2.1311 images/s
[08/14 17:30:35] ppdet.engine INFO: Epoch: [22] [0/8] learning_rate: 0.000997 loss: 20.424107 loss_cls: 16.528803 loss_iou: 0.739455 loss_dfl: 1.969931 loss_l1: 3.004658 eta: 1:00:45 batch_cost: 0.9786 data_cost: 0.5630 ips: 2.0437 images/s
[08/14 17:30:50] ppdet.engine INFO: Epoch: [23] [0/8] learning_rate: 0.000996 loss: 22.716744 loss_cls: 18.733273 loss_iou: 0.728854 loss_dfl: 1.933586 loss_l1: 2.837401 eta: 1:02:06 batch_cost: 1.0154 data_cost: 0.5988 ips: 1.9696 images/s
[08/14 17:31:01] ppdet.engine INFO: Epoch: [24] [0/8] learning_rate: 0.000996 loss: 24.184759 loss_cls: 21.281372 loss_iou: 0.728854 loss_dfl: 1.914146 loss_l1: 2.587654 eta: 1:02:21 batch_cost: 1.0387 data_cost: 0.6157 ips: 1.9255 images/s
[08/14 17:31:14] ppdet.utils.checkpoint INFO: Save checkpoint: E:\user\mytest\master\storage\train\train20250814172409\output\ppyoloe_plus_crn_l_80e_coco\model_final
[08/14 17:31:14] ppdet.engine INFO: Eval iter: 0
[08/14 17:31:14] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
[08/14 17:31:14] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.00s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
[08/14 17:31:14] ppdet.engine INFO: Total sample number: 4, average FPS: 7.475131928119893
[08/14 17:31:14] ppdet.engine INFO: Best test bbox ap is 0.000.
[08/14 17:31:17] ppdet.utils.checkpoint INFO: Save checkpoint: E:\user\mytest\master\storage\train\train20250814172409\output\ppyoloe_plus_crn_l_80e_coco\model_final
[08/14 17:31:17] ppdet.engine INFO: Epoch: [25] [0/8] learning_rate: 0.000996 loss: 27.426736 loss_cls: 24.178626 loss_iou: 0.684768 loss_dfl: 1.858913 loss_l1: 2.401643 eta: 1:02:09 batch_cost: 1.0416 data_cost: 0.6158 ips: 1.9201 images/s
[08/14 17:31:27] ppdet.engine INFO: Epoch: [26] [0/8] learning_rate: 0.000995 loss: 28.619655 loss_cls: 25.829391 loss_iou: 0.682556 loss_dfl: 1.823565 loss_l1: 2.313807 eta: 1:01:53 batch_cost: 1.0400 data_cost: 0.6141 ips: 1.9230 images/s
[08/14 17:31:36] ppdet.engine INFO: Epoch: [27] [0/8] learning_rate: 0.000995 loss: 28.583591 loss_cls: 25.829391 loss_iou: 0.670669 loss_dfl: 1.764116 loss_l1: 2.271499 eta: 1:01:15 batch_cost: 1.0488 data_cost: 0.6237 ips: 1.9070 images/s
[08/14 17:31:47] ppdet.engine INFO: Epoch: [28] [0/8] learning_rate: 0.000994 loss: 29.372814 loss_cls: 26.563274 loss_iou: 0.670669 loss_dfl: 1.739482 loss_l1: 2.175653 eta: 1:01:18 batch_cost: 1.0620 data_cost: 0.6365 ips: 1.8833 images/s
[08/14 17:31:59] ppdet.engine INFO: Epoch: [29] [0/8] learning_rate: 0.000994 loss: 29.097139 loss_cls: 26.284186 loss_iou: 0.669174 loss_dfl: 1.720188 loss_l1: 2.186833 eta: 1:01:43 batch_cost: 1.0996 data_cost: 0.6672 ips: 1.8188 images/s

时而
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
时而
ppdet.engine WARNING: Evaluation results empty, this may be due to training iterations being too few or not loading the correct weights.

求官方大佬，解决方法。

[FreemanTang]

而且，这个出现后，后面Epoch数值，loss等也慢慢不正常了。
[08/14 17:46:06] ppdet.engine WARNING: Evaluation results empty, this may be due to training iterations being too few or not loading the correct weights.
[08/14 17:46:07] ppdet.engine INFO: Epoch: [100] [0/8] learning_rate: 0.000905 loss: 0.000000 loss_cls: 0.000000 loss_iou: 0.000000 loss_dfl: 0.000000 loss_l1: 4.469059 eta: 0:55:23 batch_cost: 1.0134 data_cost: 0.6188 ips: 1.9735 images/s
[08/14 17:46:18] ppdet.engine INFO: Epoch: [101] [0/8] learning_rate: 0.000903 loss: 0.000000 loss_cls: 0.000000 loss_iou: 0.000000 loss_dfl: 0.000000 loss_l1: 4.482659 eta: 0:55:15 batch_cost: 1.0272 data_cost: 0.6310 ips: 1.9471 images/s

[FreemanTang]

问题似乎解决了，并且似乎与warning提示的training iterations being too few或者not loading the correct weights没什么关系。

我增加了数据集数量，以及生成更加严格的coco格式，并使用PaddleX去划分数据集。训练似乎没问题了。

[08/18 11:41:39] ppdet.data.source.coco INFO: Load [321 samples valid, 0 samples invalid] in file E:/user/mytest/master/storage/train/train20250818113644/trainning_dataset/coco\train.json.
W0818 11:41:39.111899  2248 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 11.1, Runtime API Version: 11.6
W0818 11:41:39.123009  2248 gpu_resources.cc:91] device: 0, cuDNN Version: 8.0.
W0818 11:41:39.124007  2248 gpu_resources.cc:117] WARNING: device: 0. The installed Paddle is compiled with CUDA 11.6, but CUDA runtime version in your machine is 11.1, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDA version.
[08/18 11:41:41] ppdet.utils.checkpoint INFO: The shape [365] in pretrained weight yolo_head.pred_cls.0.bias is unmatched with the shape [1] in model yolo_head.pred_cls.0.bias. And the weight yolo_head.pred_cls.0.bias will not be loaded
[08/18 11:41:41] ppdet.utils.checkpoint INFO: The shape [365, 768, 3, 3] in pretrained weight yolo_head.pred_cls.0.weight is unmatched with the shape [1, 768, 3, 3] in model yolo_head.pred_cls.0.weight. And the weight yolo_head.pred_cls.0.weight will not be loaded
[08/18 11:41:41] ppdet.utils.checkpoint INFO: The shape [365] in pretrained weight yolo_head.pred_cls.1.bias is unmatched with the shape [1] in model yolo_head.pred_cls.1.bias. And the weight yolo_head.pred_cls.1.bias will not be loaded
[08/18 11:41:41] ppdet.utils.checkpoint INFO: The shape [365, 384, 3, 3] in pretrained weight yolo_head.pred_cls.1.weight is unmatched with the shape [1, 384, 3, 3] in model yolo_head.pred_cls.1.weight. And the weight yolo_head.pred_cls.1.weight will not be loaded
[08/18 11:41:41] ppdet.utils.checkpoint INFO: The shape [365] in pretrained weight yolo_head.pred_cls.2.bias is unmatched with the shape [1] in model yolo_head.pred_cls.2.bias. And the weight yolo_head.pred_cls.2.bias will not be loaded
[08/18 11:41:41] ppdet.utils.checkpoint INFO: The shape [365, 192, 3, 3] in pretrained weight yolo_head.pred_cls.2.weight is unmatched with the shape [1, 192, 3, 3] in model yolo_head.pred_cls.2.weight. And the weight yolo_head.pred_cls.2.weight will not be loaded
[08/18 11:41:41] ppdet.utils.checkpoint INFO: Finish loading model weights: E:/user/mytest/master/ppyoloe_crn_l_obj365_pretrained.pdparams
W0818 11:41:41.749217  2248 gpu_resources.cc:217] WARNING: device: . The installed Paddle is compiled with CUDNN 8.4, but CUDNN version in your machine is 8.0, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDNN version.
[08/18 11:41:43] ppdet.engine INFO: Epoch: [0] [  0/160] learning_rate: 0.000000 loss: 4.634274 loss_cls: 2.620645 loss_iou: 0.454774 loss_dfl: 1.753388 loss_l1: 2.096961 eta: 5:13:13 batch_cost: 1.4683 data_cost: 0.0009 ips: 1.3621 images/s
[08/18 11:42:25] ppdet.engine INFO: Epoch: [0] [100/160] learning_rate: 0.000125 loss: 4.132266 loss_cls: 1.662648 loss_iou: 0.600738 loss_dfl: 1.859573 loss_l1: 1.313805 eta: 1:08:00 batch_cost: 0.3098 data_cost: 0.0001 ips: 6.4559 images/s
[08/18 11:42:51] ppdet.engine INFO: Epoch: [1] [  0/160] learning_rate: 0.000200 loss: 4.169550 loss_cls: 1.620539 loss_iou: 0.610048 loss_dfl: 1.843160 loss_l1: 1.579870 eta: 1:05:59 batch_cost: 0.3029 data_cost: 0.0003 ips: 6.6034 images/s
[08/18 11:43:36] ppdet.engine INFO: Epoch: [1] [100/160] learning_rate: 0.000325 loss: 4.547754 loss_cls: 2.187004 loss_iou: 0.566590 loss_dfl: 1.752433 loss_l1: 1.968284 eta: 1:04:57 batch_cost: 0.3070 data_cost: 0.0001 ips: 6.5155 images/s
[08/18 11:44:03] ppdet.engine INFO: Epoch: [2] [  0/160] learning_rate: 0.000400 loss: 10.941847 loss_cls: 8.189565 loss_iou: 0.592943 loss_dfl: 1.852427 loss_l1: 2.200756 eta: 1:04:29 batch_cost: 0.3093 data_cost: 0.0006 ips: 6.4656 images/s
[08/18 11:44:48] ppdet.engine INFO: Epoch: [2] [100/160] learning_rate: 0.000525 loss: 20.617307 loss_cls: 17.955611 loss_iou: 0.559789 loss_dfl: 1.747882 loss_l1: 1.383454 eta: 1:03:15 batch_cost: 0.2956 data_cost: 0.0001 ips: 6.7667 images/s
[08/18 11:45:15] ppdet.engine INFO: Epoch: [3] [  0/160] learning_rate: 0.000600 loss: 19.717917 loss_cls: 17.112093 loss_iou: 0.503556 loss_dfl: 1.656234 loss_l1: 1.085369 eta: 1:03:05 batch_cost: 0.3101 data_cost: 0.0004 ips: 6.4502 images/s
[08/18 11:46:00] ppdet.engine INFO: Epoch: [3] [100/160] learning_rate: 0.000725 loss: 23.231203 loss_cls: 20.675051 loss_iou: 0.558885 loss_dfl: 1.755815 loss_l1: 1.614756 eta: 1:02:14 batch_cost: 0.2977 data_cost: 0.0001 ips: 6.7174 images/s
[08/18 11:46:28] ppdet.engine INFO: Epoch: [4] [  0/160] learning_rate: 0.000800 loss: 19.666744 loss_cls: 17.208502 loss_iou: 0.598462 loss_dfl: 1.744513 loss_l1: 1.424759 eta: 1:01:57 batch_cost: 0.3050 data_cost: 0.0003 ips: 6.5581 images/s
[08/18 11:47:14] ppdet.engine INFO: Epoch: [4] [100/160] learning_rate: 0.000925 loss: 25.738511 loss_cls: 23.534376 loss_iou: 0.518710 loss_dfl: 1.731643 loss_l1: 1.095961 eta: 1:01:15 batch_cost: 0.2986 data_cost: 0.0001 ips: 6.6979 images/s
[08/18 11:47:43] ppdet.utils.checkpoint INFO: Save checkpoint: E:\user\mytest\master\storage\train\train20250818113644\output
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
[08/18 11:47:43] ppdet.data.source.coco INFO: Load [17 samples valid, 0 samples invalid] in file E:/user/mytest/master/storage/train/train20250818113644/trainning_dataset/coco\val.json.
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
[08/18 11:47:43] ppdet.engine INFO: Eval iter: 0
[08/18 11:47:44] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
[08/18 11:47:44] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=0.14s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.08s).
Accumulating evaluation results...
DONE (t=0.02s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
[08/18 11:47:44] ppdet.engine INFO: Total sample number: 17, average FPS: 16.829589683093786
[08/18 11:47:44] ppdet.engine INFO: Best test bbox ap is 0.000.
[08/18 11:47:45] ppdet.utils.checkpoint INFO: Save checkpoint: E:\user\mytest\master\storage\train\train20250818113644\output
[08/18 11:47:46] ppdet.engine INFO: Epoch: [5] [  0/160] learning_rate: 0.001000 loss: 32.984369 loss_cls: 31.163313 loss_iou: 0.517642 loss_dfl: 1.679661 loss_l1: 1.168083 eta: 1:00:57 batch_cost: 0.3004 data_cost: 0.0000 ips: 6.6578 images/s
[08/18 11:48:31] ppdet.engine INFO: Epoch: [5] [100/160] learning_rate: 0.001000 loss: 39.220213 loss_cls: 37.059324 loss_iou: 0.554986 loss_dfl: 1.739641 loss_l1: 1.218256 eta: 1:00:14 batch_cost: 0.2956 data_cost: 0.0001 ips: 6.7665 images/s
[08/18 11:48:59] ppdet.engine INFO: Epoch: [6] [  0/160] learning_rate: 0.001000 loss: 42.211636 loss_cls: 40.237928 loss_iou: 0.508136 loss_dfl: 1.707087 loss_l1: 1.045107 eta: 0:59:53 batch_cost: 0.2970 data_cost: 0.0005 ips: 6.7332 images/s
[08/18 11:49:44] ppdet.engine INFO: Epoch: [6] [100/160] learning_rate: 0.000999 loss: 39.839064 loss_cls: 37.654276 loss_iou: 0.530166 loss_dfl: 1.709676 loss_l1: 1.171320 eta: 0:59:20 batch_cost: 0.3015 data_cost: 0.0002 ips: 6.6343 images/s
[08/18 11:50:12] ppdet.engine INFO: Epoch: [7] [  0/160] learning_rate: 0.000999 loss: 40.398403 loss_cls: 38.108433 loss_iou: 0.552604 loss_dfl: 1.631385 loss_l1: 1.109246 eta: 0:58:58 batch_cost: 0.2988 data_cost: 0.0002 ips: 6.6944 images/s
[08/18 11:50:59] ppdet.engine INFO: Epoch: [7] [100/160] learning_rate: 0.000998 loss: 43.462008 loss_cls: 41.216431 loss_iou: 0.524070 loss_dfl: 1.648772 loss_l1: 0.998712 eta: 0:58:43 batch_cost: 0.3195 data_cost: 0.0001 ips: 6.2591 images/s
[08/18 11:51:27] ppdet.engine INFO: Epoch: [8] [  0/160] learning_rate: 0.000997 loss: 42.826023 loss_cls: 40.764099 loss_iou: 0.516095 loss_dfl: 1.621301 loss_l1: 0.998712 eta: 0:58:28 batch_cost: 0.3207 data_cost: 0.0005 ips: 6.2360 images/s
[08/18 11:52:15] ppdet.engine INFO: Epoch: [8] [100/160] learning_rate: 0.000996 loss: 44.618443 loss_cls: 42.245604 loss_iou: 0.487860 loss_dfl: 1.624655 loss_l1: 0.937299 eta: 0:58:09 batch_cost: 0.3193 data_cost: 0.0002 ips: 6.2645 images/s
[08/18 11:52:44] ppdet.engine INFO: Epoch: [9] [  0/160] learning_rate: 0.000995 loss: 43.005487 loss_cls: 41.018473 loss_iou: 0.502286 loss_dfl: 1.632755 loss_l1: 0.933446 eta: 0:57:52 batch_cost: 0.3101 data_cost: 0.0006 ips: 6.4491 images/s
[08/18 11:53:35] ppdet.engine INFO: Epoch: [9] [100/160] learning_rate: 0.000994 loss: 45.243492 loss_cls: 42.966446 loss_iou: 0.493885 loss_dfl: 1.636969 loss_l1: 0.972122 eta: 0:57:41 batch_cost: 0.3324 data_cost: 0.0002 ips: 6.0176 images/s
[08/18 11:54:06] ppdet.utils.checkpoint INFO: Save checkpoint: E:\user\mytest\master\storage\train\train20250818113644\output
[08/18 11:54:07] ppdet.engine INFO: Eval iter: 0
[08/18 11:54:07] ppdet.metrics.metrics INFO: The bbox result is saved to bbox.json.
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
[08/18 11:54:07] ppdet.metrics.coco_utils INFO: Start evaluate...
Loading and preparing results...
DONE (t=0.02s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.10s).
Accumulating evaluation results...
DONE (t=0.02s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
[08/18 11:54:08] ppdet.engine INFO: Total sample number: 17, average FPS: 19.468000588655602
[08/18 11:54:08] ppdet.engine INFO: Best test bbox ap is 0.000.
[08/18 11:54:09] ppdet.utils.checkpoint INFO: Save checkpoint: E:\user\mytest\master\storage\train\train20250818113644\output

但我仍有一个疑惑，为什么使用百来张数据集，训练会异常，无法收敛？具体时什么原因导致的？

[Sunting78]

您好，数据量比较少确实会无法学习到具体的特征，可以试下训练集和验证集完全一致的时候，能够收敛，如果不能或者降低学习率尝试下，看看能否拟合。

[FreemanTang]

您好，数据量比较少确实会无法学习到具体的特征，可以试下训练集和验证集完全一致的时候，能够收敛，如果不能或者降低学习率尝试下，看看能否拟合。

感谢您的回复！
在我的Windows服务器上，我将默认参数
Thank you for your reply!
On my Windows server, I put the default parameter

epoch: 80
LearningRate:
  base_lr: 0.001
  schedulers:
    - name: CosineDecay
      max_epochs: 96
    - name: LinearWarmup
      start_factor: 0.
      epochs: 5

改为
change into

epoch: 480

LearningRate:
  base_lr: 0.0001
  schedulers:
    - name: CosineDecay
      max_epochs: 480
    - name: LinearWarmup
      start_factor: 0.1
      epochs: 20

可以收敛。
It can be converged.

但我仍有一个疑惑，我使用默认参数，在linux服务器上训练，依旧也是会收敛的，但Windows服务器上，如果不改学习率，就会出现不收敛、不拟合的情况。这是什么原因导致的？
（linux服务器有4个卡，但我没有用多卡训练，只指定一个卡训练；Windows服务器上，只有一个卡。除此，其他训练条件参数基本一致。）
But I still have a doubt, I use the default parameters and train on linux server, still it converges as well, but on Windows server, if I don't change the learning rate, it doesn't converge and doesn't fit. What is the reason for this?
(The linux server has 4 cards, but I didn't train with multiple cards, and only specified one card for training; on the Windows server, there is only one card. Other than that, the other training condition parameters are basically the same.)