Evaluation Script on Benchmark

[Runsong123]

Hi,

Thank you for sharing this great work.

I want to test the proposed method on the released benchmark. Could you please share the evaluation script for the proposed benchmark?

By the way, for scannet, I noticed that the segmentation IDs in the gt_seg folder do not align with the semantic_classes.txt file.

Looking forward to your response.

Runsong

[lziiid]

Same question too :)

[VictorBbt]

Hello,

Thank you for your work! If I understood correctly the article and the Fig. 13, you voxelize the ground truth RGBD point cloud and the RGBD predictions from your model. Then, you check for those voxels if they spatially overlap, and if yes, if the class is the same.

If that's correct, I have a few questions:

• how do you find the class of a voxel if there are multiple points falling into the same voxel?
• which voxel size do you use?
• For alignment, are you using Procrustes method? Do you use a confidence filter?

Thank you in advance for your response. That would be great to get the benchmark scripts to be able to run it on the method :)

Victor

[lifuguan]

• The voxel class ambiguity problem does not arise because voxelization is performed independently for each object category. For every object, we collect all its associated predicted (or ground truth) points across frames and convert them into a binary (boolean) occupancy mask, regardless of how many points fall into it. Since each object has its own separate voxel grid mask, there is no need to resolve multi-class conflicts within a single voxel.
• We set the voxel size to 0.1.
• We do use the Procrustes for alignment, and we also use a confidence filter, which is shown below:

            threshold = np.percentile(distances, 99)  # Use 99th percentile as threshold
            weights = np.where(distances < threshold, 1.0, 0.1)  # Reduce weight of outliers