Clarification on ChatRex Evaluation Protocol

[SZhanZ]

Hi, thank you for the great work!

In Section 5.1 of the paper, it is mentioned that for MLLMs, all ground-truth categories of the current test image are provided as input during evaluation, and the model is prompted to generate the corresponding coordinate boxes.

We would like to confirm whether the same evaluation protocol applies to ChatRex. Specifically:
1. Does ChatRex also receive the ground-truth object categories of the test image during evaluation?
2. Could you please provide more details on how mAP on COCO and LVIS is computed for ChatRex under this setting?

[Mountchicken]

Hi @SZhanZ
Thank you for your questions.

Ground-truth categories during evaluation:

ChatRex does not take ground-truth boxes as input during evaluation. For each test image, we first feed the image into the UPN model and use fine-grained prompts to generate proposals. These proposals are filtered using a confidence threshold of 0.3. The remaining boxes are then passed into ChatRex. If the number of proposals exceeds 100, we keep the top 100 proposals based on UPN scores.

mAP computation on COCO and LVIS:

ChatRex outputs the indices of the selected boxes, and we use these indices to retrieve the corresponding proposals from the input as the final predicted boxes. The confidence scores for these boxes are taken directly from the UPN outputs. Thus, for each image, we obtain a set of detection boxes and scores for each ground-truth category, and compute mAP using the standard COCO protocol.

However, we note that our mAP evaluation differs slightly from conventional detection models. For example, in DETR, each image is associated with 900 queries, producing 900 × 80 class-specific boxes, from which the top 100 predictions are selected for evaluation. In contrast, ChatRex only produces predictions for the given ground-truth categories using a smaller and filtered set of proposals.