Image-based graph dataset

[qmf2020]

Hi, I have an image data set with an image size of 256*256 pixels for semantic segmentation tasks. I want to convert an image into a graph to generate a graph data set, where each pixel is used as a node, the pixel value is used as a feature of the node, and the edge between adjacent pixels is used as an edge (superpixels are not used because of semantic segmentation). How do I use pyg to achieve it? How about using graph neural network to achieve semantic segmentation effect?

[rusty1s]

This should be possible to achieve via torch_geometric.utils.grid which returns you the edge indices of a two-dimensional grid. Node features can simply be obtained by reshaping data from [height, width, num_channels] to [1, num_channels].

GNNs on this grid graph will then perform a similar operation compared to a classical CNN (but should run way slower).

[qmf2020]

Thanks for your reply.

[Joazs]

If I could see this post earlier, my progress would be much faster.Thank you!

[tmquan]

@rusty1s : Thank you for sharing the insight of the image as a graph. I also implemented using networkx and visualize as follows:
https://github.com/tmquan/graph/blob/main/Networkx.ipynb

I would like to raise another concern that Message Passing on a large image (let us say 1024 x1024) is large and may not be fit on to the GPU.

Is there any forward pass and backward pass operating in chunk fashion as the aggregated function is embarrassingly independent per node?

[rusty1s]

Good point. I'm not aware of any such solution. You may get around that problem via torch.utils.checkpoint.

[kimzed]

[rusty1s]

I don't think it returns a fully-connected graph, it should only connect direct neighbors. Do you have a minimal example to reproduce?

[kimzed]

Yes after checking you are indeed correct

[Joazs]

Hi, I have an image data set with an image size of 256*256 pixels for semantic segmentation tasks. I want to convert an image into a graph to generate a graph data set, where each pixel is used as a node, the pixel value is used as a feature of the node, and the edge between adjacent pixels is used as an edge (superpixels are not used because of semantic segmentation). How do I use pyg to achieve it? How about using graph neural network to achieve semantic segmentation effect?

You can see the ViG code. It's good project!
Yours classmate.