DeepSORT tracker performing worse than SORT

[JonathanSamelson]

Hi,

I'm posting this issue because I observed that I get worse scores with a DeepSORT tracker compared to what I get with a simple SORT tracker. I used your DETRAC tool to create the training images for the cosine metric learning. I also tried to re-train my DeepSORT model but I got equivalent results.

I compared the HOTA scores obtained on several cameras of AICity Challenge (Track 3) using the same detector (YOLO v4). I tried to tweak the max_consine_distance parameter of nn_matching and it seems that decreasing the distance increases the score. I obtained the best result when I set it to 0, I guess it means that I'm much closer to SORT in this case. For instance, on one camera, I obtained those scores:

Tracker	HOTA
SORT	43,13
Your DeepSORT - cosine distance 0.3	39,68
DeepSORT - cosine distance 0.5	40,18
DeepSORT - cosine distance 0.2	40,21
DeepSORT - cosine distance 0.1	40,36
DeepSORT - cosine distance 0.05	40,77
DeepSORT - cosine distance 0	41,22

There's not much things to tweak in the training parameters of the cosine metric. However, I was forced to use the triplet loss though, like this issue reports , did you have to use the triplet loss as well? At first, I guessed it was related to the dataset, but now I wonder if this may be the reason why my model does not work very well.

Thank you in advance!

[LeonLok]

Interesting. Unfortunately I'm not sure if I was forced to use triplet loss but I don't think I was.

Did you manage to solve the issue? And would you suggest I change the cosine distance?

[JonathanSamelson]

Unfortunately, I haven't managed to solve the issue. Actually I don't see much parameters that can be changed for training for further investigation. At the moment, I don't see the added value of using DeepSORT in comparison with SORT which is faster.

Based on your comments in your code, the cosine distance results in disregarding associations with cost larger than this parameter. I guess that having this parameter set to 0 (which gave the best result) is equivalent to ignore the cosine distance and so the model. This seems true, I just ran your model with the parameter set to 0 and obtained 41,22, same score as with my model. Hare is what I observed with your model on the video sequence, it follows the same trend:

• cosine distance 0.3 : 39,679%
• cosine distance 0.1 : 40,364%
• cosine distance 0 : 41,220% (exactly the same scores as mine)

Have you obtained quantitative results on any annotated data? AICity video sequences are nice but their annotations are sometimes inconsistent and it could create a biais.