Bad performance for object detection

[aisensiy]

Hi, I followed the example in https://lightning-flash.readthedocs.io/en/latest/reference/object_detection.html and the performance is bad:

Precision (IoU=0.50:0.95,area=all)=0.0941, Recall (IoU=0.50:0.95,area=all,maxDets=100)=0.148

I know this is a example with only 128 images. But I also tried a bigger coco subset with 10 classes and 500 images per class. The precision is about 0.17 which is also not a very good result. The code looks like below:

datamodule = ObjectDetectionData.from_coco(
    train_folder="minicoco/train2017/",
    train_ann_file="minicoco/annotation/train.json",
    test_folder="minicoco/test2017/",
    test_ann_file="minicoco/annotation/test.json",
    transform_kwargs={"image_size": 256},
    batch_size=256,
    num_workers=8
)

model = ObjectDetector(
    head="efficientdet", 
    backbone="d0", 
    num_classes=datamodule.num_classes, 
    image_size=256)

trainer = flash.Trainer(max_epochs=300, gpus=1)
trainer.finetune(model, datamodule=datamodule, strategy="no_freeze")
trainer.test(model, datamodule=datamodule)

So is there some one help me to figure out the problem.

[ethanwharris]

Hey @aisensiy Thanks for the question. I expect that this comes down to hyper-parameter tuning. Some suggestions that may help:

• As the model is pretrained, you will likely need to use either strategy="freeze" or strategy="freeze_unfreeze" so that the head weights have a chance to catch up with the pretrained weights
• As you are finetuning, you may want to use a lower LR. The default for the adam optimizer is 1e-3, so perhaps something like learning_rate=1e-4 would be better
• I would recommend adding a learning rate scheduler. As you have 300 epochs there, something like lr_scheduler=("multisteplr", {"milestones": [100, 200, 250]}) perhaps
• You could also increase the image size. You can use this table to see the recommended image size for each efficientdet backbone: https://github.com/rwightman/efficientdet-pytorch#models - e.g. for d0 it says to use image size of 512

Hope some of those tips help 😃 I'd be interested if you have any ideas to increase visibility for these kinds of tricks within Flash or how we could add them to our docs.

[aisensiy]

This a very fast and detail answer. Thanks very much.

[aisensiy]

I firstly will have a try and then give this a feedback.

[aisensiy]

Hi, after some trial. I have some progress below:

1. The default example have a good improve after using image size 512 and learning_rate=1e-4. The Precision of IoU[50:95] is about 0.4.
2. I also tried a coco subdataset with 10 classes with 5000+ images for traning (with 10% as validation) and 1400+ images for test. The validation result is good of Precision 0.26 after 30 epochs and the test result is Precision 0.2.