pointnet2_segmentation.py question regarding shapenet.py

[MBrandt-NASA]

Hi,

Thanks for creating the amazing PyG :).

In shapenet.py you have the comment:
include_normals (bool, optional): If set to :obj:False, will not
include normal vectors as input features. (default: :obj:True)

And indeed in init() you have:
self.data.x = self.data.x if include_normals else None

That's great (I don't have normals for my data ;), but then later in process_filenames() you have:
pos = data[:, :3]
x = data[:, 3:6]
y = data[:, -1].type(torch.long)
data = Data(pos=pos, x=x, y=y, category=cat_idx[cat])

So the normals are getting read into "x" from the ShapeNet data files even tho' "include_normals" above was set to False.

I had assumed that I could just run pointnet2_segmentation.py with "x = None". But then in a module like SAModule you have:

def forward(self, x, pos, batch):
idx = fps(pos, batch, ratio=self.ratio)
row, col = radius(pos, pos[idx], self.r, batch, batch[idx],
max_num_neighbors=64)
edge_index = torch.stack([col, row], dim=0)
x = self.conv(x, (pos, pos[idx]), edge_index)
pos, batch = pos[idx], batch[idx]
return x, pos, batch

So that obviously references "x" a lot...

Does pointnet2_segmentation.py support not having any normals in the input data? Or should I try running with x=pos to duplicate pos, or something like that?

Thanks :)

[MBrandt-NASA]

So I found this issue, which showed when support was added for normals in shapenet:
#884

Then checking the history of pointnet2_segmentation.py, I noticed this update made to support normals:
d782f8b#diff-5134187ae97d9080b798f14053558d38d996a6ba8177454dcfccfbf05a0955b7 which contained:
self.sa1_module = SAModule(0.2, 0.2, MLP([3 + 3, 64, 64, 128]))

And later this one:
a4f51d3#diff-5134187ae97d9080b798f14053558d38d996a6ba8177454dcfccfbf05a0955b7

which had:
self.fp1_module = FPModule(3, MLP([128 + 3, 128, 128, 128]))

So I removed the 2 "+ 3"'s that were added, and I seem to be able to run without normals... in case anyone else runs into this.

Still, the include_normals flag isn't obeyed in shapenet.py's process_filenames(), so I think I'll make it into an issue ;).

[rusty1s]

Yes, you are right. One needs to decrement the number of input channels in order to account for non-existent node features. I'm not sure though what's the problem with shapenet.py. Clearly, normal vectors are not included in case include_normals=False. The processing still includes the addition of normals though, as this allows us to switch between using normals and not using normals without re-processing the whole dataset.

[MBrandt-NASA]

Thanks for the reply @rusty1s :).

Regarding ShapeNet, I see I didn't understand the order of execution in PyG. I see now that, in the init() of PyG's Dataset class, the data is processed, but then later in ShapeNet's init because of the include_normals flag set = False, x will = None.

I should have read through the init methods in the superclasses more carefully -- thanks for correcting me :). I did see the Dataset tutorial but I guess I didn't fully see the order of execution until now... still learning PyG ;).

[paulwingert865]

How important do you think are normals in general for segmentation tasks?

[paulwingert865]

Do you have any comparison/experience based on ShapeNet trained with and without normals?

[rusty1s]

I don't have benchmarks on this one, but the last time I checked they helped a bit in reaching better performance. I think that normal vectors help the neural network to better understand the surfaces of the underlying object, which may increase performance in return. However, the performance difference isn't that drastic that the model does not learn anything without including them. Normally, we don't have access to normal vectors in standard point cloud scenarios anyways.