plot_question

[rtwod2]

Hi all,

I'm training a GNN model using pytorch geometric for a regression problem. I'm using cross-validation. The mean plot for my target is the below one. Do you find it okay?

Thank you in advance for your time and consideration of my question.

[rusty1s]

This is hard to tell just by looking at the loss. I suggest you make use of some additional metrics to judge the quality of your model, e.g., accuracy, F1-score, etc, and compare it to baselines as well.

[rusty1s]

I think I misunderstood. It looks like you already have a set of features, while for different kernels only a single feature changes (I thought you only have one feature for a single graph). Correct? In this case, this looks fine to me, and the problem of bad model performance is likely stemming from something else.

[rtwod2]

Yes, I have already a set of features, but the same kernels have different graphs, with the only different feature to be the input bytes (all the other features are the same for the same kernels). While different kernels have different all the features in between. Is this more clear?

So, I 'll give you an example of some predictions (depicts one of my targets - the plot above is about the other target):
labels
16 tensor(15802.5859) rodinia-3.1-streamcluster-memset_kernel.ll
17 tensor(16260.0508) rodinia-3.1-streamcluster-memset_kernel.ll
18 tensor(16717.5176) rodinia-3.1-streamcluster-memset_kernel.ll
19 tensor(9398.0723) shoc-1.1.5-Triad-Triad.ll
20 tensor(11227.9336) shoc-1.1.5-Triad-Triad.ll
21 tensor(13057.7930) shoc-1.1.5-Triad-Triad.ll

predictions
16 tensor(25941.0840) rodinia-3.1-streamcluster-memset_kernel.ll
17 tensor(25941.0840) rodinia-3.1-streamcluster-memset_kernel.ll
18 tensor(25941.0840) rodinia-3.1-streamcluster-memset_kernel.ll
19 tensor(26215.4453) shoc-1.1.5-Triad-Triad.ll
20 tensor(26215.4453) shoc-1.1.5-Triad-Triad.ll
21 tensor(26215.4453) shoc-1.1.5-Triad-Triad.ll

For the above, I have the same prediction-number for all the same kernels, (which differ on only one feature) and different prediction-number in between the different kernels.

I have tried to change my model, I used relu, then elu, I changed my dropout rate, I used topkpooling and then I erased topkpooling. But I can see the same behavior on all these experiments.

Do you have any intuition on what's going on?

[rusty1s]

Can you show me the model that you are using?

[rtwod2]

Yes, of course! Sorry for the late answer, I just saw your feedback.

class GTModel(nn.Module):
    def __init__(self, num_layers, hidden_dim, heads, feat_dropout, top_k_pool, norm, beta=True):

        super().__init__()

        dense_neurons = 256
        self.top_k_every_n = top_k_pool # self.top_k_every_n
        self.num_layers = num_layers
        self.feat_dropout = feat_dropout
        self.norm = None

        self.conv_layers = nn.ModuleList([])
        self.transf_layers = nn.ModuleList([])
        self.pooling_layers = nn.ModuleList([])
        self.norm_layers = nn.ModuleList([])

    
        #we don't calculate the embeddings here, but we calculate them using the NodeFeatureEncoder and then pass them into our  
        #GNN model directly.
        #This is happening per batch of graphs!


        # Transformation Layer
        self.conv1 = TransformerConv(hidden_dim, hidden_dim, heads, dropout=0.3, edge_dim=hidden_dim, norm=self.norm, beta=False)
        self.bn1 = nn.BatchNorm1d(hidden_dim*heads)
        self.transf1 = Linear(hidden_dim*heads, hidden_dim)

         for i in range(self.num_layers):
            self.conv_layers.append(TransformerConv(hidden_dim, hidden_dim, heads, dropout=0.3, edge_dim=hidden_dim, norm=self.norm, beta=False))
            self.norm_layers.append(nn.BatchNorm1d(hidden_dim*heads))
            self.transf_layers.append(Linear(hidden_dim*heads, hidden_dim))         
            if i % self.top_k_every_n == 0:
                self.pooling_layers.append(TopKPooling(hidden_dim, ratio=0.5))    

         self.linear1 = Linear(hidden_dim*2, dense_neurons)
         self.linear2 = Linear(dense_neurons, int(dense_neurons/2))  
         self.linear3 = Linear(int(dense_neurons/2), 1) 

 
    def forward(self, batch_graphs): 
        out = []

        nodefeature = NodeFeatureEncoder.FeatureEncoder(batch_graphs, emb_dim=64) # emb_dim=hidden_dim=64
        renewed_data = nodefeature(batch_graphs) # forward pass

        edgefeature = EdgeFeatureEncoder.eFeatureEncoder(renewed_data, emb_dim=64)
        renewed_data = edgefeature(renewed_data) 

        x, edge_attr, edge_index, batch_index= renewed_data.x, renewed_data.edge_attr, renewed_data.edge_index, renewed_data.batch

        # Initial Transformation
        x = self.conv1(x, edge_index, edge_attr)
        x = self.bn1(x)
        x = torch.nn.functional.elu(self.transf1(x))
        
        global_representation = []

        for i in range(self.num_layers):
            x = self.conv_layers[i](x, edge_index, edge_attr)
            x = self.norm_layers[i](x)
            x = torch.nn.functional.elu(self.transf_layers[i](x))            
            if i % self.top_k_every_n == 0 or i == self.num_layers:
                x, edge_index, edge_attr, batch_index, _, _ = self.pooling_layers[int(i/self.top_k_every_n)](x, edge_index, edge_attr, batch_index) 
            
                global_representation.append(torch.cat([gmp(x, batch_index), gap(x, batch_index)], dim=1))

        x = sum(global_representation)

        print(f'out shape after gap, cat and sum {x.shape}') #[batchsize, hiddendim*lendatalist]

        # Output block
        x = torch.nn.functional.elu(self.linear1(x))
        x = F.dropout(x, p=0.8, training=self.training)
        x = torch.nn.functional.elu(self.linear2(x))
        x = F.dropout(x, p=0.8, training=self.training)
        x = self.linear3(x)

       
        return x

And for TransformerConv, I'm using the one available from Pytorch Geometric.

[rusty1s]

Thanks. I think your model looks good but also is pretty complex. Maybe we should start with a more simpler baseline, e.g., a simple MLP or a simple GraphSAGE model with global_mean_pooling at the end?

[rtwod2]

Thank you so much for everything. Sorry for my late reply, but I've had a lot going on. I'm looking at the GraphSage paper, and I'll implement this model to see if it can be improved. I'm thinking if it might help to have position encodings. I'll look into them, and send you anything new that comes up.
Thank you soooo much.