Dataloader implementation for graph with 2D images as nodes

[Gaurangkarwande]

Hello,

I am fairly new to pytorch_geometric and graph learning. I am working on a task which requires me to construct a graph with images as nodes. The final task is then the comparison of 2 such graphs. A general overview of the model is as follows -

1. Each data-point consists of 2 graphs.
2. Each graph as 18 nodes. Each node is an 224x224 image.
3. First I pass each image through a traditional CNN which embeds the 2D image into a 1D feature vector of size 512. Hence there are 18 different CNNs.
4. Now one graph can be represented by a tensor of shape 18x512.
5. Next I do graph embedding using graph neural networks. Each graph is embedded into a 128 dimensional vector.
6. Lastly I compare the two graph embedding.
7. The 18 CNNs and the one GNN are trained together.

I am struggling to implement the dataloaders for this task. I am familiar with pytorch dataloaders. But I am unsure how to build my custom dataset and dataloader for this task. If anyone has ideas or tips, please let me know.

Thanks!

[rusty1s]

Sounds interesting. Why do you want to train 18 separate CNNs? Can't you utilize one CNN to embedd all 18 images? For data loading, I suggest to look into the PairData tutorial, see here, where

• edge_index1 and edge_index2 denote edge connectivity of the two graphs
• x1 and x2 denote the batched images with shape [18, 244, 244, 3] or [18, 3, 244, 244]

You can then use the torch_geometric.loader.DataLoader to process these pairs of graph in batch mode as well. Let me know if you need any further guidance :)

[Gaurangkarwande]

Hi, thanks for your response. I did figure out the dataloader setup using PairData. The entire graph represents one image and the nodes of the graph are specific cropped parts of that image. Each cropped part is a distinct sub-image. I need 18 differs CNNs as each CNN would be specific to a sub-image.
While training I see that passing data sequentially through these 18 CNNs is very inefficient and is the bottleneck. There is a lot tensor reshaping as well. As the dimensions of the batch from Dataloader is [batch_size*num_nodes, node_features], this needs to reshaped to [batch_size, num_nodes, node_features] before feeding into the CNN and then lastly CNN output has to be reshaped again to the earlier dimensions to be fed into GNN. I wonder if there is a better way to do this.
Can I perform operations on individual nodes (here cropped sub-images) inside the forward method of my GNN model and parallelize these 18 different CNNs ?

[rusty1s]

Reshaping shouldn't be a bottleneck since it does not modify the underlying memory storage. AFAIK, there exists no way to parallelize different CNNs in PyTorch.

Why do each sub-image desperately needs to be processed by individual CNNs? Can't you apply a shared CNN on each subimage, and then use different model heads (on top of the shared CNN) for each region?