BERT features fine-tune with graph neural network

[piyush-J]

Thank you for the amazing library! I am a student and needed some help in implementing one of my ideas.

• My dataset contains 1000 data points.
• Every data point is a list of 8 graphs: [graph_1, graph_2, ..., graph_8]
• Every graph has an unequal number of nodes and edges.
• Every node consists of an English sentence, which I want to pass to a pre-trained BERT model so that every node can be represented in a 768-dimension feature vector.

Questions:

1. However, I want to keep the BERT model trainable, so I need this BERT module inside the training class. How do I represent the data points (as the tokenized sentence & attention map will be of unequal length for all the nodes) so that I can feed them to the training class?
2. How will I batch the data in this case?

I really appreciate any help you can provide.

[rusty1s]

Instead of saving numerical node features in data.x as a PyTorch Tensor, you should be able to simply assign each node a str attribute:

from itertools import chain

from torch_geometric.data import Data
from torch_geometric.loader import DataLoader

data = Data(sentence=['node1', 'node2', 'node3'], num_nodes=3)

loader = DataLoader([data, data, data], batch_size=3)

batch = next(iter(loader))
print(batch)
print(batch.sentence)
print(list(chain(*batch.sentence)))

DataBatch(sentence=[3], num_nodes=9, batch=[9], ptr=[4])
[['node1', 'node2', 'node3'], ['node1', 'node2', 'node3'], ['node1', 'node2', 'node3']]
['node1', 'node2', 'node3', 'node1', 'node2', 'node3', 'node1', 'node2', 'node3']

You should be able to input that input a BERT/Transformer model to obtain a feature vector per node, while PyG batching stays fully intact. Let me know if this helps you!

[piyush-J]

Thank you so much for your prompt response! :)