Inquiry Regarding Fairness of Comparison on 3D Public Benchmarks

[fanyan0411]

Dear Dr. Qingyong Hu,

Thanks for your excellent work on the weakly supervised 3D scene segmentation.

I am writing to inquire about the fairness of comparison on 3D public benchmarks.
As you noted in a previous talk, the label ratios reported in SQN are super-voxel level, and the point level label ratios ought to be lower than values presented in the published paper. For instance, the reported super-voxel level label ratio of S3DIS is 0.1%, and the corresponding point level ratio should be 0.0068%. Several works have followed SQN, such as HybridCR and PSD.

However, a number of works announced that the label ratios are raw point level, such as OTOC, ReDAL, HPAL, CPCM. We have confirmed the code and outlined the details as follows:

ReDAL: https://github1s.com/tsunghan-wu/ReDAL/blob/main/dataloader/s3dis/region_active_dataset.py line 48-66
CPCM: https://github.com/lizhaoliu-Lec/CPCM/blob/master/dataset/fully_supervised/stanford.py line196-210

I've also observed that these projects have conducted comparisons without explicitly distinguishing between point-level and super-voxel level. This leads me to the question of whether it is equitable to directly integrate and compare these performances, akin to the example in CPCM as follows:

Thank you for your time and consideration. I am hoping for your response sincerely!