Ignored boxes without labels seem to be treated as positive samples in YOLO training example

[alexandreofbh]

Hi dears,

I’ve been testing the dnn_yolo_train_ex.cpp example, and I noticed something unexpected. It looks like rectangles that are marked as "ignored" and no labeleds are still being treated as positive samples during training. They end up being detected by the trained model as if they were real objects.

Has anyone else observed this behavior? Is it the intended behavior, or could there be something wrong in how ignored boxes are being handled during training?

Thanks!

[arrufat]

Hi!

It would be helpful to have a bit more details, but I have a few questions:

• What kind of samples are you trying to ignore? Maybe they're really similar to non-ignored samples, which might be more abundant, and the detector it's just doing its job?
• How did you notice they're treated as positive samples, the ignore logic is there, did you debug it?:
  • https://github.com/arrufat/dlib/blob/d0f41798be7a64cb15f94611d960475cf3097b0c/dlib/dnn/loss.h#L3747
  • https://github.com/arrufat/dlib/blob/d0f41798be7a64cb15f94611d960475cf3097b0c/dlib/dnn/loss.h#L3768

[alexandreofbh]

Hi !
You are always willing to help people, I have no words to thank you, Arrufat!

I'll give you some details

I'm trying to create an ALPR of Brazilian license plates.

I used Imagelab to make the markings

Note the unlabeled boxes, they are all marked as ignored.

imglab --stats train.xml
Number of images: 490
Number of different labels: 37
Number of non-ignored boxes: 4511

Unlabeled Boxes:
number of images: 442
number of occurrences: 2051
min box area: 30
max box area: 7.39424e+06
mean box area: 97408.2
stddev box area: 504585
mean width/height ratio: 2.48678
stddev width/height ratio: 2.33541

Label: 0
number of images: 17
number of occurrences: 19
min box area: 216
max box area: 24871
mean box area: 2461.37
stddev box area: 5490.18
mean width/height ratio: 0.600275
stddev width/height ratio: 0.0887874

Label: 1
number of images: 42
number of occurrences: 62
min box area: 119
max box area: 19080
mean box area: 2161.82
stddev box area: 3283.67
mean width/height ratio: 0.546173
stddev width/height ratio: 0.130633

Label: 2
number of images: 156
number of occurrences: 195
min box area: 88
max box area: 127765
mean box area: 6229.99
stddev box area: 12186.8
mean width/height ratio: 0.684434
stddev width/height ratio: 0.131996

Label: 3
number of images: 155
number of occurrences: 209
min box area: 216
max box area: 49413
mean box area: 4098.91
stddev box area: 6450.51
mean width/height ratio: 0.675104
stddev width/height ratio: 0.110344

Label: 4
number of images: 145
number of occurrences: 182
min box area: 170
max box area: 39445
mean box area: 3740.88
stddev box area: 4837.96
mean width/height ratio: 0.642994
stddev width/height ratio: 0.0911963

Label: 5
number of images: 170
number of occurrences: 194
min box area: 160
max box area: 41440
mean box area: 5471.46
stddev box area: 7304.5
mean width/height ratio: 0.690463
stddev width/height ratio: 0.5633

Label: 6
number of images: 173
number of occurrences: 209
min box area: 126
max box area: 79576
mean box area: 5682.52
stddev box area: 8944.64
mean width/height ratio: 0.668295
stddev width/height ratio: 0.113481

Label: 7
number of images: 135
number of occurrences: 166
min box area: 165
max box area: 38608
mean box area: 4431.51
stddev box area: 6599.69
mean width/height ratio: 0.638453
stddev width/height ratio: 0.144194

Label: 8
number of images: 184
number of occurrences: 218
min box area: 88
max box area: 66732
mean box area: 5197.86
stddev box area: 8857.51
mean width/height ratio: 0.669833
stddev width/height ratio: 0.130577

Label: 9
number of images: 137
number of occurrences: 164
min box area: 48
max box area: 77861
mean box area: 5541.88
stddev box area: 10057.8
mean width/height ratio: 0.668693
stddev width/height ratio: 0.126802

Label: A
number of images: 91
number of occurrences: 149
min box area: 72
max box area: 65500
mean box area: 2886.21
stddev box area: 7163.93
mean width/height ratio: 0.70937
stddev width/height ratio: 0.145252

Label: B
number of images: 62
number of occurrences: 77
min box area: 117
max box area: 25400
mean box area: 2230.69
stddev box area: 4086.83
mean width/height ratio: 0.660226
stddev width/height ratio: 0.106486

Label: C
number of images: 54
number of occurrences: 59
min box area: 77
max box area: 41666
mean box area: 2379.07
stddev box area: 6437.52
mean width/height ratio: 0.698545
stddev width/height ratio: 0.124472

Label: D
number of images: 56
number of occurrences: 76
min box area: 77
max box area: 27800
mean box area: 2354.88
stddev box area: 3940.66
mean width/height ratio: 0.689918
stddev width/height ratio: 0.120181

Label: E
number of images: 93
number of occurrences: 143
min box area: 70
max box area: 27948
mean box area: 1829.55
stddev box area: 3968.96
mean width/height ratio: 0.732213
stddev width/height ratio: 0.555304

Label: F
number of images: 51
number of occurrences: 54
min box area: 60
max box area: 26499
mean box area: 3061.67
stddev box area: 5316.3
mean width/height ratio: 0.67299
stddev width/height ratio: 0.156233

Label: G
number of images: 65
number of occurrences: 76
min box area: 77
max box area: 37147
mean box area: 3455.14
stddev box area: 6208.9
mean width/height ratio: 0.711665
stddev width/height ratio: 0.158931

Label: H
number of images: 100
number of occurrences: 119
min box area: 63
max box area: 64222
mean box area: 4272.07
stddev box area: 8415.87
mean width/height ratio: 0.68363
stddev width/height ratio: 0.0966648

Label: I
number of images: 155
number of occurrences: 253
min box area: 40
max box area: 35000
mean box area: 2694
stddev box area: 4493.94
mean width/height ratio: 0.481298
stddev width/height ratio: 0.094845

Label: J
number of images: 96
number of occurrences: 111
min box area: 140
max box area: 36656
mean box area: 2057.97
stddev box area: 4215.22
mean width/height ratio: 0.666183
stddev width/height ratio: 0.0780568

Label: K
number of images: 168
number of occurrences: 173
min box area: 126
max box area: 56317
mean box area: 8011.4
stddev box area: 8947.38
mean width/height ratio: 0.737441
stddev width/height ratio: 0.207827

Label: L
number of images: 63
number of occurrences: 74
min box area: 77
max box area: 65660
mean box area: 3226.14
stddev box area: 9933.72
mean width/height ratio: 0.655199
stddev width/height ratio: 0.115363

Label: M
number of images: 62
number of occurrences: 77
min box area: 88
max box area: 51948
mean box area: 4291.52
stddev box area: 7737.65
mean width/height ratio: 0.85476
stddev width/height ratio: 0.278309

Label: N
number of images: 94
number of occurrences: 111
min box area: 66
max box area: 27772
mean box area: 2863.4
stddev box area: 4451.28
mean width/height ratio: 0.728565
stddev width/height ratio: 0.173305

Label: O
number of images: 199
number of occurrences: 358
min box area: 63
max box area: 331792
mean box area: 4229.97
stddev box area: 18451.2
mean width/height ratio: 0.699921
stddev width/height ratio: 0.132168

Label: P
number of images: 134
number of occurrences: 151
min box area: 80
max box area: 45236
mean box area: 2515.58
stddev box area: 5459.85
mean width/height ratio: 0.661735
stddev width/height ratio: 0.110732

Label: Q
number of images: 61
number of occurrences: 77
min box area: 260
max box area: 26625
mean box area: 3504.35
stddev box area: 4754.31
mean width/height ratio: 0.69924
stddev width/height ratio: 0.116743

Label: R
number of images: 115
number of occurrences: 190
min box area: 77
max box area: 29580
mean box area: 2577.62
stddev box area: 5026.07
mean width/height ratio: 0.698018
stddev width/height ratio: 0.108127

Label: S
number of images: 65
number of occurrences: 90
min box area: 77
max box area: 27105
mean box area: 1932.22
stddev box area: 4107.15
mean width/height ratio: 0.70534
stddev width/height ratio: 0.136413

Label: T
number of images: 72
number of occurrences: 84
min box area: 104
max box area: 50135
mean box area: 3037.58
stddev box area: 6496.95
mean width/height ratio: 0.664148
stddev width/height ratio: 0.111718

Label: U
number of images: 64
number of occurrences: 72
min box area: 117
max box area: 30163
mean box area: 4071.03
stddev box area: 5723.09
mean width/height ratio: 0.699972
stddev width/height ratio: 0.146174

Label: V
number of images: 79
number of occurrences: 81
min box area: 153
max box area: 51615
mean box area: 4685.07
stddev box area: 7863.48
mean width/height ratio: 0.657851
stddev width/height ratio: 0.145156

Label: W
number of images: 57
number of occurrences: 61
min box area: 154
max box area: 40002
mean box area: 6516.41
stddev box area: 7791.68
mean width/height ratio: 0.688893
stddev width/height ratio: 0.149251

Label: X
number of images: 102
number of occurrences: 110
min box area: 260
max box area: 52961
mean box area: 5284.73
stddev box area: 7937.26
mean width/height ratio: 0.679771
stddev width/height ratio: 0.106489

Label: Y
number of images: 35
number of occurrences: 35
min box area: 368
max box area: 37513
mean box area: 5642.43
stddev box area: 7873.88
mean width/height ratio: 0.648968
stddev width/height ratio: 0.113728

Label: Z
number of images: 38
number of occurrences: 38
min box area: 108
max box area: 18054
mean box area: 2310.37
stddev box area: 3698.5
mean width/height ratio: 0.679422
stddev width/height ratio: 0.112871

==============================================================
I completed the training

Training data:

images: 490

labels: 37

• : 2051 (31.2272%)
• 0: 19 (0.289281%)
• 1: 62 (0.943971%)
• 2: 195 (2.96894%)
• 3: 209 (3.1821%)
• 4: 182 (2.77101%)
• 5: 194 (2.95371%)
• 6: 209 (3.1821%)
• 7: 166 (2.52741%)
• 8: 218 (3.31912%)
• 9: 164 (2.49695%)
• A: 149 (2.26857%)
• B: 77 (1.17235%)
• C: 59 (0.898295%)
• D: 76 (1.15713%)
• E: 143 (2.17722%)
• F: 54 (0.822168%)
• G: 76 (1.15713%)
• H: 119 (1.81181%)
• I: 253 (3.85201%)
• J: 111 (1.69001%)
• K: 173 (2.63398%)
• L: 74 (1.12667%)
• M: 77 (1.17235%)
• N: 111 (1.69001%)
• O: 358 (5.45067%)
• P: 151 (2.29903%)
• Q: 77 (1.17235%)
• R: 190 (2.89281%)
• S: 90 (1.37028%)
• T: 84 (1.27893%)
• U: 72 (1.09622%)
• V: 81 (1.23325%)
• W: 61 (0.928745%)
• X: 110 (1.67479%)
• Y: 35 (0.532887%)
• Z: 38 (0.578563%)
  dnn_trainer details:
  net_type::num_layers: 286
  net size: 235.859 MiB (61717594 parameters)
  net architecture hash: 4bfc22564cc06c290c947006840bf230
  loss: loss_yolo (3 outputs:(tag4008,tag4016,tag4032), anchor_boxes:(tag4008:8x8,9x11,23x8;tag4016:18x27,53x11,42x39;tag4032:104x23,73x79,164x76), 37 labels:(,0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z), iou_ignore_threshold: 0.8, iou_anchor_threshold: 0.2, lambda_obj:1, lambda_box:1, lambda_cls:1, gamma_obj:0, gamma_cls:0, overlaps_nms:(0.45,1), classwise_nms:true)
  get_train_one_step_calls(): 238101
  synchronization file: .//caracteres_placas20250718.sync
  trainer.get_solvers()[0]: sgd: weight_decay=0.0005, momentum=0.9
  mini batch size: 20
  learning rate: 1e-07
  learning rate shrink factor: 0.1
  min learning rate: 1e-06
  iterations without progress threshold: 10000
  test iterations without progress threshold: 500

=============================================================
Inference

inference using a sample from the same training dataset for this test:

detections: 9

S: [(195.652, 104.653) (211.01, 128.064)] 0.999671
: [(215.158, 29.5952) (266.386, 76.7039)] 0.999375
Y: [(213.481, 103.967) (228.812, 126.939)] 0.999226
O: [(252.55, 103.803) (268.902, 127.369)] 0.999146
H: [(176.816, 104.878) (193.183, 127.943)] 0.998871
7: [(269.466, 103.775) (284.507, 127.722)] 0.998696
3: [(235.97, 104.283) (251.717, 127.247)] 0.998377
7: [(288.772, 102.896) (303.175, 125.9)] 0.989116
: [(207.123, 92.6843) (275.074, 102.014)] 0.984007

=============================================================
Conclusion

That's how I deduced that the trainer was training the ignored boxes. I reviewed them several times and also tested labeling the ignored boxes as NONELABELLED, and the NONELABELLED labeling showed up as detected.

[arrufat]

Can you try training by removing all ignored boxes? I you don't want to detect, you should just not label. Ignore means just I don't care what you do here. So I doesn't penalize detections but also doesn't penalize non-detections.
I think it will work if you just remove the ignored boxes from the dataset.

[alexandreofbh]

I think the objects marked as "ignore" would be interpreted as objects to avoid being negative and give weight to the marked objects. And if they were marked as valid objects, they would interfere with the marked objects. Like these figures, for example, which have the existing characters, but I'm ignoring them. If they were present with other labels, wouldn't they confuse the trainer?

[alexandreofbh]

I thought of this because of the comment written in dnn_mmod_train_find_cars_ex.cpp

// When I was creating the dlib vehicle detection dataset I had to label all the cars
// in each image.  MMOD requires all cars to be labeled, since any unlabeled part of an
// image is implicitly assumed to be not a car, and the algorithm will use it as
// negative training data.  So every car must be labeled, either with a normal
// rectangle or an "ignore" rectangle that tells MMOD to simply ignore it (i.e. neither
// treat it as a thing to detect nor as negative training data).  
// 
// In our present case, many images contain very tiny cars in the distance, ones that
// are essentially just dark smudges.  It's not reasonable to expect the CNN
// architecture we defined to detect such vehicles.  However, I erred on the side of
// having more complete annotations when creating the dataset.  So when I labeled these
// images I labeled many of these really difficult cases as vehicles to detect.   

Ah, I'm doing the training without the ignored boxes, it will take time. I tell you the result!

[arrufat]

I think that explanation applies mostly to the MMOD loss and not the YOLO loss.
MMOD is great because it works with few samples, but requires a very clean dataset. In my experience, YOLO is more robust to noisy datasets, but requires way more data.

Also the case mentioned there is different: it's labeling all hard cases as ignored, whereas your are labeling negative samples as ignored. That's not what you want: you want to not detect those texts and logos, so they shouldn't be labeled.

[arrufat]

I feel like, if all your texts are in plates, the network will figure that out and implicitly detect plates. I haven't tried plate number detection, though.

[alexandreofbh]

Thank you very much for the tips Arrufat! I removed all the boxes marked as ignored and put them into training. It will take a while to finish. When I finish and test it, I'll post the results.

[alexandreofbh]

Hi Arrufat,

Thank you so much for your help! I completed the training and ran some inference tests. Your suggestions made a huge difference and significantly improved the system.

My dataset is still quite small, so I'll expand it next. I'm also planning to do post-processing and full-plate detection to better isolate the important characters.

I really appreciate your willingness to help and the way you (and Davis King) always take the time to respond, not just to me, but to everyone.

ps: Attached are some test inferences, in case you want to look at them.

Thanks again, and best regards!

[arrufat]

Nice! It's great that it's working. Actually, these days, much to my regret, I don't do this kind of stuff any more and I miss it a bit.

By the way, did you know that there's a full-fledged repository to play with YOLO here?
https://github.com/dlibml/yolo-object-detector/

Feel free to check it out and apply to your needs. For this task you might want to use the yolov7_tiny, which is defined in the repo, but not enabled by default. It will be much faster than YOLOv3 (from the example) and might even work better.

[alexandreofbh]

I'm going to test it with the yolov7_tiny!
Thank you again!