the evaluation results display COCO default class namesinstead of my custom Visdrone class names

[EthanWare0115]

Search before asking

• I have searched the BoxMOT issues and found no similar bug report.

Question

I'm working on multi-class MOT evaluation using BoxMOT (v16.0.2) with the Visdrone2019-MOT dataset and a custom-trained YOLO11n model. I've encountered an issue where the evaluation results display COCO default class names (e.g., person, bicycle) instead of my custom Visdrone class names, leading to misaligned metrics. Here's a detailed breakdown:

1. My Setup
Dataset: Visdrone2019-MOT (train split), 12 custom classes with IDs 0-11:
ignored regions(0),pedestrian (1), people (2), bicycle (3), car (4), van (5), truck (6), tricycle (7), awning-tricycle (8), bus (9), motor (10),others(11).
Model:OLO11n trained on Visdrone2019-DET
Tracker: ByteTrack
Evaluation Tool: TrackEval (integrated with BoxMOT)

2. Modifications I Made (to boxmot/engine/trackeval/trackeval/datasets/mot_challenge_2d_box.py)
I followed the multi-class evaluation guidelines and modified the following to adapt to Visdrone:
①Updated get_default_dataset_config():
CLASSES_TO_EVAL: Set to my 12 custom class names (as listed above)
BENCHMARK: Visdrone2019-MOT
GT_FOLDER: Path to my Visdrone train split (/home/zhaoyuntao/LQR/link_data1_LQR/data/Visdrone2019-MOT/train)
②Updated self.class_name_to_class_id :
self.class_name_to_class_id = {'ignored regions':0,'pedestrian': 1, 'people': 2, 'bicycle': 3, 'car': 4, 'van': 5,'truck': 6, 'tricycle': 7, 'awning-tricycle': 8, 'bus': 9, 'motor': 10, 'others':11}
③Commented out the pedestrian-only validation (lines 257-260) to allow multi-class evaluation:

# if len(tracker_classes) > 0 and np.max(tracker_classes) > 1:
#     raise TrackEvalException(...)

④Cleared distractor classes:
distractor_class_names = []

3. The Issue：When running the evaluation command below:
python -m boxmot.engine.cli eval --yolo-model /path/to/yolo11n-best.pt --tracking-method bytetrack --classes 0 1 2 3 4 5 6 7 8 9 10 11 --source /path/to/Visdrone2019-MOT/train --project /path/to/BoxMot_Processing/runs

The results still display COCO default class names (e.g., person, bicycle, car) instead of my custom Visdrone classes. For example:

This means the class mapping is not working

4. Questions
①Did I miss any modifications to ensure TrackEval uses my custom class names instead of COCO defaults?
②How can I correctly map my Visdrone class IDs (0-9) and names to the evaluation results?
③Is there a known issue with TrackEval's multi-class name display in BoxMOT v16.0.2?
Any guidance to resolve this class name misalignment would be greatly appreciated!

Thanks！

[mikel-brostrom]

Do not make modification to boxmot/engine/trackeval/trackeval/datasets/mot_challenge_2d_box.py.
What enables custom datasets is this file: boxmot/boxmot/utils/custom_mot_challenge_2d_box.py
Which inherits from this file that you are modifying:

boxmot/boxmot/utils/custom_mot_challenge_2d_box.py

Line 7 in f27dbac

class CustomMotChallenge2DBox(MotChallenge2DBox):

I recommend you to make all the modification to this file custom_mot_challenge_2d_box and leave the full trackeval folder untouched

[mikel-brostrom]

The relevant parts are:

Classes to evaluate on:

boxmot/boxmot/utils/custom_mot_challenge_2d_box.py

Line 20 in f27dbac

real_classes = config.get('CLASSES_TO_EVAL', ['person']) if config else ['person']

Classes supported by your model, right now COCO classes are hardcoded:

boxmot/boxmot/utils/custom_mot_challenge_2d_box.py

Lines 37 to 46 in f27dbac

	self.class_name_to_class_id = {
	'person': 1, 'bicycle': 2, 'car': 3, 'motorcycle': 4, 'airplane': 5, 'bus': 6, 'train': 7, 'truck': 8, 'boat': 9, 'traffic light': 10,
	'fire hydrant': 11, 'stop sign': 12, 'parking meter': 13, 'bench': 14, 'bird': 15, 'cat': 16, 'dog': 17, 'horse': 18, 'sheep': 19, 'cow': 20,
	'elephant': 21, 'bear': 22, 'zebra': 23, 'giraffe': 24, 'backpack': 25, 'umbrella': 26, 'handbag': 27, 'tie': 28, 'suitcase': 29, 'frisbee': 30,
	'skis': 31, 'snowboard': 32, 'sports ball': 33, 'kite': 34, 'baseball bat': 35, 'baseball glove': 36, 'skateboard': 37, 'surfboard': 38, 'tennis racket': 39, 'bottle': 40,
	'wine glass': 41, 'cup': 42, 'fork': 43, 'knife': 44, 'spoon': 45, 'bowl': 46, 'banana': 47, 'apple': 48, 'sandwich': 49, 'orange': 50,
	'broccoli': 51, 'carrot': 52, 'hot dog': 53, 'pizza': 54, 'donut': 55, 'cake': 56, 'chair': 57, 'couch': 58, 'potted plant': 59, 'bed': 60,
	'dining table': 61, 'toilet': 62, 'tv': 63, 'laptop': 64, 'mouse': 65, 'remote': 66, 'keyboard': 67, 'cell phone': 68, 'microwave': 69, 'oven': 70,
	'toaster': 71, 'sink': 72, 'refrigerator': 73, 'book': 74, 'clock': 75, 'vase': 76, 'scissors': 77, 'teddy bear': 78, 'hair drier': 79, 'toothbrush': 80
	}

Adapt distractor classes for VisDrone

boxmot/boxmot/utils/custom_mot_challenge_2d_box.py

Line 55 in f27dbac

distractor_classes = [12, 8, 6, 7, 2] if self.benchmark == 'MOT20' else [12, 8, 7, 2]

Let me know if you solve the issue @EthanWare0115 😄

[mikel-brostrom]

Custom class dataset evaluation is not well tested due to the lack of feedback. So any feedback is appreciated 😄

[mikel-brostrom]

I will try to improve this workflow today

[mikel-brostrom]

This should be fully solved now for visdrone @EthanWare0115

[mikel-brostrom]

# https://github.com/VisDrone/VisDrone-Dataset
# object_category ids: 0 ignored regions*, 1 pedestrian, 2 people, 3 bicycle, 4 car, 5 van, 6 truck,
# 7 tricycle, 8 awning-tricycle, 9 bus, 10 motor, 11 others*; *0 and 11 are ignored/distractors.
download:
  runs_url: ""
  dataset_url: "https://drive.google.com/uc?export=download&id=14z8Acxopj1d86-qhsF1NwS4Bv3KYa4Wu"

benchmark:
  source: "boxmot/engine/trackeval/VisDrone2019-MOT-test-dev"
  split: "sequences"
  eval_classes:
    1: pedestrian
    2: people
    3: bicycle
    4: car
    5: van
    6: truck
    7: tricycle
    8: awning-tricycle
    9: bus
    10: motor
  distractor_classes:
    0: ignored regions
    11: others

Now all the classes are gathered from each datasets' config. Both classes to be evaluated and distractors. No more custom modification to trackeval in order to get this working. But notice! trackeval's indexing for classes starts at 1 and you model will most certainly generate classes that are 0-indexed. This mismatch offset by:

boxmot/boxmot/utils/mot_utils.py

Line 121 in c9fd7a1

results[:, 6].astype(np.int32) + 1, # class

But it feels a bit awkward to specify 1-indexed classes in the configs

[EthanWare0115]

Thank you for your prompt reply and proactive updates. As I have been occupied with some urgent matters recently, I may not be able to conduct the test temporarily. I will provide you with timely feedback as soon as I complete the test.

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.
Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!