IndexError: index X is out of bounds during inference when using reduced chunk size (Nvidia Titan X Pascal GPU, 12GB)

[anloganathan]

Hi,
I encountered an IndexError during inference when using a trained model on a Nvidia Titan X Pascal GPU (12GB). Because of memory limitations, I reduced the chunk size to 5000, but inference crashes with an out-of-bounds index inside unified_metric.py.

Command used:
CUDA_VISIBLE_DEVICES=1 python tools/test.py \ configs/oneformer3d_qs_radius16_qp300_2many.py \ work_dirs/clean_forestformer/epoch_3000_fix.pth

Exception has occurred: IndexError index 130170 is out of bounds for axis 0 with size 130170 File "oneformer3d/unified_metric.py", line 96, in compute_metrics gt_l, pred_l = int(sem_gt_i[j]), int(sem_pre_i[j]) File "tools/test.py", line 173, in main runner.test() File "tools/test.py", line 180, in <module> main() IndexError: index 130170 is out of bounds for axis 0 with size 130170

Could you please advise whether changing the chunk size requires additional modifications elsewhere? I added chunk = chunk in the model dict in the config file.

[bxiang233]

Hi, I encountered an IndexError during inference when using a trained model on a Nvidia Titan X Pascal GPU (12GB). Because of memory limitations, I reduced the chunk size to 5000, but inference crashes with an out-of-bounds index inside unified_metric.py.

Command used: CUDA_VISIBLE_DEVICES=1 python tools/test.py \ configs/oneformer3d_qs_radius16_qp300_2many.py \ work_dirs/clean_forestformer/epoch_3000_fix.pth

Exception has occurred: IndexError index 130170 is out of bounds for axis 0 with size 130170 File "oneformer3d/unified_metric.py", line 96, in compute_metrics gt_l, pred_l = int(sem_gt_i[j]), int(sem_pre_i[j]) File "tools/test.py", line 173, in main runner.test() File "tools/test.py", line 180, in <module> main() IndexError: index 130170 is out of bounds for axis 0 with size 130170

Could you please advise whether changing the chunk size requires additional modifications elsewhere? I added chunk = chunk in the model dict in the config file.

Hello, I haven’t seen this issue before.
Was the inference working correctly before you modified the chunk size?

[anloganathan]

Hi, I get CUDA OOM issue if i don't reduce the chunk size, so i could not check. At line 96 of oneformer3d/unified_metric.py, there is mismatch between the size of groundtruth and predicted.

[jfprieur]

I get the same error with your provided data files, it crashes when it is calculating the accuracy metrics. I have not modified anything else.

CUDA_VISIBLE_DEVICES=0 python tools/test.py
configs/oneformer3d_qs_radius16_qp300_2many.py
work_dirs/clean_forestformer/epoch_3000_fix.pth

(Previous output removed)
Processing regions: 100%|███████████████████████████████████████████████████████████████████████████
postprocessing 6: 3 ms
postprocessing 7: 0 ms
postprocessing 8: 0 ms
Traceback (most recent call last):
File "/workspace/tools/test.py", line 176, in
main()
File "/workspace/tools/test.py", line 172, in main
runner.test()
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/runner.py", line 1767, in test
metrics = self.test_loop.run() # type: ignore
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/loops.py", line 442, in run
metrics = self.evaluator.evaluate(len(self.dataloader.dataset))
File "/opt/conda/lib/python3.10/site-packages/mmengine/evaluator/evaluator.py", line 79, in evalua
_results = metric.evaluate(size)
File "/opt/conda/lib/python3.10/site-packages/mmengine/evaluator/metric.py", line 133, in evaluate
_metrics = self.compute_metrics(results) # type: ignore
File "/workspace/oneformer3d/unified_metric.py", line 96, in compute_metrics
gt_l, pred_l = int(sem_gt_i[j]), int(sem_pre_i[j])
IndexError: index 130170 is out of bounds for axis 0 with size 130170

[bxiang233]

I get the same error with your provided data files, it crashes when it is calculating the accuracy metrics. I have not modified anything else.

CUDA_VISIBLE_DEVICES=0 python tools/test.py configs/oneformer3d_qs_radius16_qp300_2many.py work_dirs/clean_forestformer/epoch_3000_fix.pth

(Previous output removed) Processing regions: 100%|███████████████████████████████████████████████████████████████████████████ postprocessing 6: 3 ms postprocessing 7: 0 ms postprocessing 8: 0 ms Traceback (most recent call last): File "/workspace/tools/test.py", line 176, in main() File "/workspace/tools/test.py", line 172, in main runner.test() File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/runner.py", line 1767, in test metrics = self.test_loop.run() # type: ignore File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/loops.py", line 442, in run metrics = self.evaluator.evaluate(len(self.dataloader.dataset)) File "/opt/conda/lib/python3.10/site-packages/mmengine/evaluator/evaluator.py", line 79, in evalua _results = metric.evaluate(size) File "/opt/conda/lib/python3.10/site-packages/mmengine/evaluator/metric.py", line 133, in evaluate _metrics = self.compute_metrics(results) # type: ignore File "/workspace/oneformer3d/unified_metric.py", line 96, in compute_metrics gt_l, pred_l = int(sem_gt_i[j]), int(sem_pre_i[j]) IndexError: index 130170 is out of bounds for axis 0 with size 130170

Hi!

This error is caused by the changes in PR #9 (https://github.com/SmartForest-no/ForestFormer3D/pull/9
). I didn’t realise they would affect the test stage as well. From a quick look at the code, the problem should only be in the evaluation part.

The good news is that your prediction outputs should already be saved in the output folder. You can open the .ply files in CloudCompare to visualise the results. This is confirmed because the following line has already been executed:

self.save_ply_withscore(
original_points.cpu().numpy(),
final_semantic_labels,
clean_all_pre_ins,
merged_instance_scores,
region_path,
pts_semantic_gt,
pts_instance_gt
)

So, for now I recommend that you run the evaluation using tools/final_eval.py:
https://github.com/SmartForest-no/ForestFormer3D/blob/main/tools/final_eval.py

This will let you compute the metrics from the saved outputs without relying on the buggy evaluation inside test.py.