Have trouble in connecting GCN to MLP and create a combi NN

[zzzLemon]

Hi,

I am developing a model for the node regression task, I have generated a heterogeneous graph dataset, which contains multiple graphs. I want to predict the value of a "wire" node, which is always set as the last node of this type of node during dataset generation, so I want to achieve the aggregated feature of this node after two GCNs, I write it like:
x_dict['wire'][0,-1]

And my plan is to connect two GCNs to an MLP, which contains 3 hidden layers. Here is my code

class GCN_MLP(torch.nn.Module):
    def __init__(self, hidden_channels_gcn, out_channels_gcn, num_layers_gcn, hidden_layer_size_mlp,out_layer_size_mlp,num_layers_mlp,activation_function_mlp):
        super().__init__()
        self.num_layers_gcn=num_layers_gcn
        self.layers = torch.nn.ModuleList()
        self.relu = torch.nn.ReLU()
        for i in range(num_layers_gcn):
            if i== num_layers_gcn-1: # last layer hidden_channels->out_channels, else hidden_channels->hidden_channels
                conv = HeteroConv({
                ('input', 'connect_to', 'wire'): GCNConv(-1, out_channels),
                ('wire', 'connect_to', 'input'): GCNConv(-1, out_channels),
                ('transistor', 'connect_to', 'wire'):  GCNConv(-1, out_channels),   
                ('wire', 'connect_to', 'transistor'): GCNConv(-1, out_channels),   
                ('power', 'connect_to', 'wire'): GCNConv(-1, out_channels),
                ('wire', 'connect_to', 'power'): GCNConv(-1, out_channels),  
                ('transistor', 'connect_to', 'transistor'): GCNConv(-1, out_channels),

            }, aggr='sum')
                self.layers.append(conv)
            else:

                conv = HeteroConv({
                ('input', 'connect_to', 'wire'): GCNConv(-1, hidden_channels),
                ('wire', 'connect_to', 'input'): GCNConv(-1, hidden_channels),
                ('transistor', 'connect_to', 'wire'):  GCNConv(-1, hidden_channels),   
                ('wire', 'connect_to', 'transistor'): GCNConv(-1, hidden_channels),   
                ('power', 'connect_to', 'wire'): GCNConv(-1, hidden_channels),
                ('wire', 'connect_to', 'power'): GCNConv(-1, hidden_channels),  
                ('transistor', 'connect_to', 'transistor'): GCNConv(-1, hidden_channels),

            }, aggr='sum')

                self.layers.append(conv)
        self.last_layer=num_layers_mlp-3
        self.activation_function=activation_function
        self.layers.append(torch.nn.Linear(out_channels,hidden_layer_size[0]))
        self.layers.append(self.activation_function)
        for i in range(0,num_layers_mlp - 3):
            self.layers.append(torch.nn.Linear(hidden_layer_size[i],hidden_layer_size[i+1]))
            self.layers.append(self.activation_function)
        self.layers.append(torch.nn.Linear(hidden_layer_size[self.last_layer],out_layer_size))

    def forward(self, x_dict, edge_index_dict,x_gcn_mlp):
        for conv in self.layers:
            x_dict = conv(x_dict, edge_index_dict)
            x_dict = {key: x.relu() for key, x in x_dict.items()}
        x_gcn_mlp=x_dict['wire'][0,-1]

        for i in range(len(self.layers)+self.num_layers_gcn):
            x_gcn_mlp = self.layers[i](x_gcn_mlp)     
        return x_gcn_mlp # the last wire node

In which "out_channels_gcn" of the second GCN is also the size of MLP's input layer, "hidden_layer_size_mlp" is defined as a list, e.g. [10,20] stands for the first hidden layer and the second hidden layer of
the MLP contains 10 and 20 neurons, respectively. "out_layer_size_mlp" of MLP is always set to "1", I just have one value to predict."num_layers_mlp" is currently set to 5, i.e. MLP contains 1 input layer, 1 output layer, and 3 hidden layers.

In "forward", x_gcn_mlp is initially set to 0, it will receive the tensors of "x["wire"][0,-1]" and propagate further through MLP.

I batch multiple graphs into the training and testing batches just use the train_test_split from sklearn, and create dataloader like:

train_loader = DataLoader(train_set, batch_size=2, shuffle=True)

And then train:

def train(model, device, dataset, optimizer, num_epochs,batch_size):
    for epoch in range(num_epochs):
        model.to(device)
        model.train()
        for batch_index, batch in enumerate(train_loader):
            x_batch=batch
            y_batch=batch.y
            x_batch,y_batch=x_batch.to(device),y_batch.to(device)
            optimizer.zero_grad()
            print(type(x_batch.x_dict['wire'][0,-1]))
            out = model(x_batch.x_dict, x_batch.edge_index_dict,0)
            y_batch = y_batch.view(out.shape)
            loss =torch.nn.MSELoss(y_batch,out)
            loss.backward()
            optimizer.step()
            train_loss += loss.item()*batch_size
        train_loss =  train_loss/train_set.__len__() # average train_loss of one epoch 
        print('Train Epoch: {} Loss: {:.6f}'.format(
                epoch, train_loss))
        train_loss_sum +=train_loss

I get errors "'tuple' object has no attribute 'size'", it points to out = model(x_batch.x_dict, x_batch.edge_index_dict,0). Is it wrong to call the model like this?
Another reason might be the "forward"? I write x_gcn_mlp=x_dict['wire'][0,-1]maybe here x_dict['wire'][0,-1] is tuple with shape [1,out_channels]?

I would appreciate it if anyone could give me a solution and many thanks in advance.

[rusty1s]

I think the issue is that you cannot use GCNConv for bipartite message passing. Can you swap out this operator for something like SAGEConv?

[rusty1s]

Do you have a minimal example to reproduce? What does data.validate() output?

[zzzLemon]

Do you have a minimal example to reproduce? What does data.validate() output?

Hi, I solved the problem, my dataset was rightly generated, the error came from how I defined the combi-NN. As I have posted, I tried to construct a net that consists of gnn and mlp, the way I did was bad, I rebuilt the net like:

Class HeteroGNN(torch.nn.Module):
   def _init_:
    ********************
        self.MLP=MLP(channel_list=self.mlp_layers,num_layers=len(self.mlp_layers),norm=None)

    def forward(self, x_dict, edge_index_dict):
        for conv in self.HeteroGNN_layers:
            x_dict = conv(x_dict, edge_index_dict)
            x_dict = {key: x.relu() for key, x in x_dict.items()}
        return (self.MLP((x_dict['wire'][-1]).view(-1,self.gnn_out_channel_size)))# the last wire node

And in train:

        for batch in data_loader:
            batch.to('cuda:0')
            optimizer.zero_grad()
            out = model(batch)
            print(out)
            y_predict = batch.y.view(out.shape)   
            loss =loss_fn(y_predict,out)
            loss.backward()
            optimizer.step()
            train_loss += loss.item()*batch_size

Hopefully it can help anyone needs it.

However, I have trouble now with the batch, the loss is wrongly calculated. I set the batch to 2 and print the out and predict:

            out = model(batch.x_dict,batch.edge_index_dict)
            print(out)
            y_predict = batch.y
            print(y_predict)

And you can see, "out" only has one value,

tensor([[-0.0463]], device='cuda:0', grad_fn=<AddmmBackward0>)
tensor([0.7972, 0.7606], device='cuda:0')

I think there is something wrong with "out" or dataloader, but I don't know why. I have tried to modeify the forward function, to:

    def forward(self, x_dict, edge_index_dict,batch):

But since Im using heterogenous dataset, and HeteroDataBatch doesn't have batch. Do you have any suggestion? Thanks

[rusty1s]

I think you can want to index the last node of every graph, so you could do something like this

ptr = data['wire'].ptr
ptr = ptr[1:] - 1
x = x_dict['wire'][ptr]

[zzzLemon]

I think you can want to index the last node of every graph, so you could do something like this

ptr = data['wire'].ptr
ptr = ptr[1:] - 1
x = x_dict['wire'][ptr]

Hi,

many thanks for your help! So for multiple graphs training it is recommended to use ptr, right?

[rusty1s]

In your case, you need the last node index of every graph, and data.ptr is the easiest way to get it.