Tracking Metrics Remain Identical Despite Different ReID Models Producing Different Embeddings

[Li-Ruiqi777]

Search before asking

• I have searched the BoxMOT issues and found no similar bug report.

Question

When using BoxMot with different ReID models, I observe that the extracted embeddings change as expected, but the final tracking metrics (MOTA, IDF1, etc.) remain completely identical across all model variations.
I want to know what caused this.

[mikel-brostrom]

Which dataset are you trying this on?

[Li-Ruiqi777]

Which dataset are you trying this on?

I tried both on the MOT17-mini and my custom dataset.
The result on MOT17-mini is this:

(cv) PS E:\DeepLearning\track_utils> boxmot eval --yolo-model ./yolo11n.pt `
>> --reid-model osnet_x0_25_msmt17.pt `
>> --tracking-method botsort `
>> --source 'E:/DeepLearning/boxmot/assets/MOT17-mini/train' `
>> --verbose `
>> --project ./runs

...
HOTA: -pedestrian HOTA DetA AssA DetRe DetPr AssRe AssPr LocA OWTA HOTA(0) LocA(0)
HOTALocA(0)
MOT17-02-FRCNN 34.98 14.001 88.414 14.115 88.722 89.827 96.007 88.285 35.158 38.435 86.74
33.339
MOT17-04-FRCNN 27.085 7.6874 96.295 7.7851 81.743 97.184 97.184 87.686 27.273 30.861 80.399
24.812
COMBINED 28.914 8.9889 93.727 9.0988 83.867 94.78 96.82 87.747 29.108 32.578 82.329
26.821

CLEAR: -pedestrian MOTA MOTP MODA CLR_Re CLR_Pr MTR PTR MLR sMOTA CLR_TP CLR_FN
CLR_FP IDSW MT PT ML Frag
MOT17-02-FRCNN 15.909 86.74 15.909 15.909 100 13.636 4.5455 81.818 13.8 14 74
MOT17-04-FRCNN 4.7619 92.591 4.7619 7.1429 75 7.1429 0 92.857 4.2327 24 312
8 0 3 0 39 0
COMBINED 7.0755 90.436 7.0755 8.9623 82.609 9.375 1.5625 89.062 6.2183 38 386
8 0 6 1 57 0

Identity: -pedestrian IDF1 IDR IDP IDTP IDFN IDFP
MOT17-02-FRCNN 27.451 15.909 100 14 74 0
MOT17-04-FRCNN 13.043 7.1429 75 24 312 8
COMBINED 16.17 8.9623 82.609 38 386 8

Count: -pedestrian Dets GT_Dets IDs GT_IDs
MOT17-02-FRCNN 14 88 4 22
MOT17-04-FRCNN 32 336 4 42
COMBINED 46 424 8 64

2025-11-13 21:18:25.301 | MainProcess/MainThread | INFO | E:\DeepLearning\boxmot\boxmot\engine\val.py:431 | run_trackeval - {"HOTA": 28.914, "AssA": 93.727, "AssRe": 94.78, "MOTA": 7.0755, "IDSW": 0, "IDF1": 16.17, "IDs": 8}

Change to the other reid model:

(cv) PS E:\DeepLearning\track_utils> boxmot eval --yolo-model ./yolo11n.pt `
>> --reid-model lmbn_n_cuhk03_d.pt `
>> --tracking-method botsort `
>> --source 'E:/DeepLearning/boxmot/assets/MOT17-mini/train' `
>> --verbose `
>> --project ./runs

...
HOTA: -pedestrian HOTA DetA AssA DetRe DetPr AssRe AssPr LocA OWTA HOTA(0) LocA(0)
HOTALocA(0)
MOT17-02-FRCNN 34.98 14.001 88.414 14.115 88.722 89.827 96.007 88.285 35.158 38.435 86.74
33.339
MOT17-04-FRCNN 27.085 7.6874 96.295 7.7851 81.743 97.184 97.184 87.686 27.273 30.861 80.399
24.812
COMBINED 28.914 8.9889 93.727 9.0988 83.867 94.78 96.82 87.747 29.108 32.578 82.329
26.821

CLEAR: -pedestrian MOTA MOTP MODA CLR_Re CLR_Pr MTR PTR MLR sMOTA CLR_TP CLR_FN
CLR_FP IDSW MT PT ML Frag
MOT17-02-FRCNN 15.909 86.74 15.909 15.909 100 13.636 4.5455 81.818 13.8 14 74
0 0 3 1 18 0
MOT17-04-FRCNN 4.7619 92.591 4.7619 7.1429 75 7.1429 0 92.857 4.2327 24 312
8 0 3 0 39 0
COMBINED 7.0755 90.436 7.0755 8.9623 82.609 9.375 1.5625 89.062 6.2183 38 386
8 0 6 1 57 0

Identity: -pedestrian IDF1 IDR IDP IDTP IDFN IDFP
MOT17-02-FRCNN 27.451 15.909 100 14 74 0
MOT17-04-FRCNN 13.043 7.1429 75 24 312 8
COMBINED 16.17 8.9623 82.609 38 386 8

Count: -pedestrian Dets GT_Dets IDs GT_IDs
MOT17-02-FRCNN 14 88 4 22
MOT17-04-FRCNN 32 336 4 42
COMBINED 46 424 8 64

2025-11-13 21:19:26.436 | MainProcess/MainThread | INFO | E:\DeepLearning\boxmot\boxmot\engine\val.py:431 | run_trackeval - {"HOTA": 28.914, "AssA": 93.727, "AssRe": 94.78, "MOTA": 7.0755, "IDSW": 0, "IDF1": 16.17, "IDs": 8}

And the embeddings are this, they are different.

[mikel-brostrom]

For very small datasets like boxmot's custom MOT17-mini no differences are usually seen for different ReID models. The sequences are too short to have any effect on the metrics. If you want to see differences in metrics use MOT17.

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.
Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!