Problems with pointnet for custom Dataset

[WizKhalista]

Hey everyone,

so i was trying to get a Point cloud segmentation Task working. For that i used the provided pointnet segmentation model from the examples: https://github.com/pyg-team/pytorch_geometric/blob/master/examples/pointnet2_segmentation.py.
Training on the Shapenet Dataset worked perfectly fine. After that i wanted to train it for a custom Dataset.
First i got some Pointcloud files of a Screw and annotated it for two labels:
Class 1: The basic screw (Screw)
Class 2: A specific area, where a screw can ge gripped by a roboter for example. (Gripping area)
It looked like this

I followed the procedure of creating a custom Dataset from this link: https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html
For a first proof of Concept i added 20 annotated Point clouds of the screw viewed from different angles. I trained the Segmentation model for the two classes and visualized the outcome on the Files of the training Dataset. Sadly the prediction predictet every point to be the same class, like this:

So there was no segmentation into Screw/Gripping area.
I came to the idea, that the Net is predicting every point to be Class 1 since there are much more points of that Class in the Dataset. I tried out tuning on many parameters, also Weighting in the Loss function, but nothing really works.

If anyone got experience with training the Net for his own Datasets any help would be greatly appreciated.
Thank you in advance!

Best regards
Marcel

[rusty1s]

This is interesting. Thanks for sharing. One issue of PointNet is that it cannot really generalize across rotations, which might explain the issues you are encountering as well. The PPFConv tries to fix that.

[WizKhalista]

Thank you for your reply.
So i would have to add this specific layer to my Model? Since i used this model: https://github.com/pyg-team/pytorch_geometric/blob/master/examples/pointnet2_segmentation.py , could you tell me how i would achieve that?

[rusty1s]

I don't think there is anything more today than replacing all PointConv instances with PPFConv instances in pointnet2_classification.py. Let me know if that works for you!

[WizKhalista]

Okay, i replaced the PointConv with the PPFConv layer, now the forward() call is missing the argument: "edge_index".

Trying to figure that out at the moment. Following the Documentation, i think that the forward pass of the PPFConv Layer needs normal vectors, so i would have to adjust this:

pytorch_geometric/examples/pointnet2_classification.py

Line 25 in 141d7b6

x = self.conv((x, x_dst), (pos, pos[idx]), edge_index)

I wanted to try it with only zeros as normal vectors first like this::
normals = torch.zeros_like(pos)
x = self.conv((x, x_dst), (pos, pos[idx]), normals, edge_index)
but already get an Error:
ValueError: Encountered tensor with size 3250 in dimension -2, but expected size 650.

[WizKhalista]

I got it running! I can train with the PPFConv now. :). The results on a first run looked promising, atleast the screws get segmented into multiple areas. I´ll have to optimize that though for better results.

Thank you for your help @rusty1s !