Annotation and Pre-processing of ScanNet for Data Curation

[akawincent]

Hello @jihanyang, Thank you for your outstanding work！

I have some questions about the data curation pipeline related to ScanNet.

• When curating the category information for ScanNet, which annotation protocol did you adopt: NYU40, ScanNet200, or did you use the full set of 550 categories?
• During the construction of QA pairs, is it necessary to remove background information such as wall and floor?

Looking forward to your reply~~

[jihanyang]

We have a filtered category list on ScanNet, we did some post-processing for both scannet and scannet200:

SCANNET200_VALID_CATEGORY_NAME = ['pillow', 'monitor', 'door', 'lamp', 'printer', 'towel',
                                  'tv', 'nightstand', 'plant', 'keyboard', 'backpack', 'printer',
                                  'trash bin', 'trash can', 'recycling bin', 'closet', 'microwave', 'fan',
                                  'washing machine', 'mirror', 'piano', 'radiator', 'telephone', 'clock',
                                  'guitar', 'bicycle', 'oven', 'mouse']

SCANNET200_CLASS_REMAPPER = {
    'trash bin': ['trash bin', 'trash can', 'recycling bin'],
    'computer mouse': ['mouse'],
}

SCANNETV2_VALID_CATEGORY_NAME = ['bed', 'chair', 'sofa', 'table',
                                 'window', 'bookshelf', 'counter',
                                 'desk', 'refrigerator']

[akawincent]

@jihanyang Sorry for the late reply.

According to the information you provided, the VSI-Bench's QA, which is built upon ScanNet, only involves 34 semantic labels.
This number seems quite small. Because the original ScanNet semantic annotations include up to 200 categories.

Could this limitation in the number of semantic labels potentially restrict the diversity and comprehensiveness of the evaluation, especially in assessing an MLLM's true visual-spatial intelligence, which often involves a richer understanding of scene contents and object interactions?