the evaluation results on validation set is very low

[beyoursel]

I have trained the ForestFormer3d on my machine with a 4090 GPU, batch_size=1, radius=8m, and other params is the same with the original code, but I found that the evaluation results on val set is very low. However , the result on test set is normal. Is there any problem?

[bxiang233]

I have trained the ForestFormer3d on my machine with a 4090 GPU, batch_size=1, radius=8m, and other params is the same with the original code, but I found that the evaluation results on val set is very low. However , the result on test set is normal. Is there any problem?

Hi, I did not modify the original OneFormer3D code for computing validation metrics, so the validation results were actually incorrect and not really meaningful. Later, Atakan tried to fix this part, but I did not have time to rerun the experiments and verify the updated validation results myself.

During training, I mainly tracked the training loss. As long as the loss kept decreasing, I considered the training normal, and I stopped when it became stable. I always saved the last model, and I did not use a save-best strategy based on validation metrics.

So your model is already trained properly. Could you try testing your current trained model directly and also visualize the predictions? The results should be similar to the checkpoint I provided. If the test results are still poor, please feel free to post them here, and I can help analyze them.

Best,
Binbin