Semantic Segmentation using PointNet/PPFNet on own data

[WizKhalista]

Hi everyone,

i recently already had a question and I am not really progressing at the momemt, so i hope someone can help me.

Task:

My main task is semantic segmentation of a scene, more detailed i have many objects recorded by a camera and i want to segment the Pointcloud, coming from the Camera so that every Object is assigned the correct class.
The Objects are Bolt/Nut/Cylinder/Quader as shown here:

My Goal would be in a scene like this, to assign each object the correct class:

What i did:

I created a Dataset for the 4 Objects, where every Object is in different Rotations, since when predicting on a scene the orientation of the Objects varies. I split it into Train/Validation with 80/20 %.
Then i trained the Model and reached a Train accuracy of 98%-100% but the Validation accuracy is stagnating on 15%-25%. Since that was a sign of overfitting for me i already used Dropout etc. but i can get good results. Also when predicting multiple objects in one frame, every Point just gets the same class (as shown on the picture all Objects are Red=Nut).

Is my idea of training on single Objects (Bolt/Nut/Quader/Cylinder) and then expecting it to work on a scene with multiple Objects in it wrong?

I hope someone can help me and maybe already used the Pointnet/PPFNet for his own Segmentation purposes.
Thanks in advance!

Regards Marcel

[WizKhalista]

I think i figured it out, i just needed to use the PointConv Layer from the Pointnet instead of the PPFConv Layers and also there was an issue with building my Data Object from the Dataset. So i can see that the Model is clearly learning things.
Just one question left, when i train my Model with a single labeled pointcloud of an object at a time, will it be able to make a prediction for a whole scene with many objects of different classes in it?

Pointclouds for training would look like this:

The Scene i want to predict would look something like this:

Do you think that this would be possible?
Thank you in advance!

Best regards Marcel

[rusty1s]

I don't think this is possible without modification of your inference pipeline. One trivial approach would be to utilize a sliding window over your evaluation point cloud and try to predict the object in each window.

[WizKhalista]

Thank you @rusty1s for your answer, that definitely makes sense.
The sliding window approach would be a good idea for scenes with not that many objects in it i think. My main goal is to make a prediction for such a scene:

where the resulting pointcloud would look like this:

Since there are many objects at different positions and sometimes occluding each other i think i need another approach.
I have read a few papers about how others are creating their Datasets for such a problem and i think i need to generate many scenes with those objects in different configurations in it and then train the Model.
Since the Labeling of the pointclouds will be taking a lot of time i hope i don´t need too many scenes for it to learn.. 😄

[rusty1s]

Yes, that works as well. In that case, this problem is interpreted as a segmentation problem rather than a classification problem.