How to handle the dynamically generated graph in model construction

[595693085]

Assuming my model consists of CNN and GCN, the graphs are generated using the CNN output, for example:

class Mymodel(...)
     ....
    forward(data):
        x=.....(data)
        x=cnn(x)
        adj=F.sigmoid(torch.matmul(x, x.transpose(1, 2)))  # the grpah is generated by multiplying x and its transposition. Filterd by 0.5
        edge_index = dense_to_sparse(adj)
        x=gcn(x, edge_index)
        ....

So the graph (edge_index) is genereted by the front part of the network, but it is different to utilize "Data" class to implemente the edge and x with it in the "forward" function. In addition, it is different to utilize the gcn in a mini-batch since the graphs in a batch (the generated graphs by the cnn in a batch) should be concatenated into a big graph. So I am confused if the graphs is generated in a dynamical way such as the front network, how to implement it.

[rusty1s]

Can you share a bit more detail here about input features and the GNN part? It looks like your adj matrix is dense by design, so you probably want to take a look at our dense GNN layers.

[595693085]

Thanks for the replying. If the shape of CNN output (x) is [batch, 128, 8], then the adj is generated by multiplying x and its transposition(adj=x*x^t) ,followed by a sigmoid function and a filter with a 0.5 threshold, the adj could have a shape of (batch, 128,128), which indicates that the graph for each data has 128 nodes. Then the edge_index could be genereated, the feature for the graph node is the CNN output (x, which with a shape of [batch, 128, 8]), so each graph in a batch all have 128 nodes. Then each graph in a batch will be taken to the GCN for the further feature extraction, which the shape of gcn output will be [batch, 128, n] (n depends on the gcn setting), then a classier will be added to the end to classify each node. How to implement it when the graphs are dynamically generated in the model "forward" function, it seems that its difficult to utilize "Data" class and min-batch mechanism in the "forward". Thanks.

[rusty1s]

The dense GNN layers support [batch_size, num_nodes, num_nodes] tensors, so you can directly use them. If you wanna do sparse message passing via edge_index, you can do

batch, row, col = adj.nonzero()
batch = batch * num_nodes

row = row + batch
col = col + batch
edge_index = torch.stack([row, col], dim=0)

x = x.view(-1, x.size(-1))  # [batch_size * num_nodes, num_features]

[595693085]

Thanks, it works. But it need a little change, which changes "batch, row, col = adj.nonzero()" to "batch, row, col = adj.nonzero()[:,0], dj.nonzero()[:,1], dj.nonzero()[:,2]"