Train autoencoder with custom dataset

[arya-h]

Good evening, thank you for the library and the useful guides.
I'm trying to train an autoencoder for a graph (fully connected) that only has coordinates as features.

batch_feat =  env.get_batch_nodes(1400)
num_nodes = 10
data_list = []
for el in batch_feat:
    feat = el.double() #array of coordinates
    perm = itertools.permutations(range(num_nodes), 2)
    l1 = []
    l2 = []
    for x in perm:
        l1.append(x[0])
        l2.append(x[1])
    a1 = torch.LongTensor(l1)
    a2 = torch.LongTensor(l2)

    res = [a1,a2]
    res = torch.stack(res)

    #res has graph connectivity in COO format
    data = Data(x=feat, edge_index=res)
    data_list.append(data)
         
torch.save(data_list, "./datasets/train.pt")``

The generated dataset (random graphs with 10 nodes) is generated. I then create the associated class

import torch
from torch_geometric.data import InMemoryDataset, download_url


class MyOwnDataset(InMemoryDataset):
    def __init__(self, root="./datasets", transform=None, pre_transform=None, pre_filter=None):
        super().__init__(root, transform, pre_transform, pre_filter)
        self.data = torch.load("./datasets/train.pt")
    @property
    def processed_file_names(self):
        return ['data.pt']

    def process(self):
        # Read data into huge `Data` list.
        data_list = [...]

        if self.pre_filter is not None:
            data_list = [data for data in data_list if self.pre_filter(data)]

        if self.pre_transform is not None:
            data_list = [self.pre_transform(data) for data in data_list]

        data, slices = self.collate(data_list)
        torch.save((data, slices), self.processed_paths[0])

dataset = MyOwnDataset(transform=T.NormalizeFeatures())

As described in the autoencoder tutorial

for _ in range(len(data)):
    #print(_)
    #print(dataset[0][_])
    data[_].train_mask = data[_].val_mask = data[_].test_mask = None
    data[_] = data[_].to(device)
    data[_] = train_test_split_edges(data[_])

Ill skip the other lines, which are identical to your guide, and skip to the train / test portion

def train(cnt):
    model.train()
    optimizer.zero_grad()
    data[cnt].x = (data[cnt].x).float()
    z = model.encode(data[cnt].x, (data[cnt].train_pos_edge_index))
    loss = model.recon_loss(z,  torch.LongTensor(data[cnt].train_pos_edge_index))
    #if args.variational:
    #   loss = loss + (1 / data.num_nodes) * model.kl_loss()
    loss.backward()
    optimizer.step()
    #scheduler.step()
    return float(loss)


def test(pos_edge_index, neg_edge_index, cnt):
    model.eval()
    with torch.no_grad():
        z = model.encode(data[cnt].x, data[cnt].train_pos_edge_index)
    return model.test(z, pos_edge_index, neg_edge_index)

for epoch in range(1, 1500 + 1):
    loss = train(epoch)
    print(f"loss : {loss} , epoch {epoch}")
    #auc, ap = test(data.test_pos_edge_index, data.test_neg_edge_index   ,epoch)
    if loss < 0.2:
        torch.save(model.state_dict(), "./models/maybe.pt")
    #print(f'Epoch: {epoch:03d}, AUC: {auc:.4f}, AP: {ap:.4f}')

However

1. the loss doesnt seem to go down, i played around with the optimizer and its learning rate but to no avail.
2. calling the test function gives me the following error which i dont understand (excuse my ignorance)
   ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.
   I tried generating data with y=node_features but clearly that wasn't the answer.

Am i getting it all wrong? I just want to encode graphs with n_nodes, 2 features each into 16-dimensional nodes, but maybe i misunderstood what autoencoders are for?

[rusty1s]

How does your model look like? Is this task actually solvable by a GNN? Do features of neighbors give information about central node coordinates? In general, most GNN might also fail to learn anything meaningful on fully-connected graphs since each node will receive the same set of messages.

[arya-h]

Thanks for the swift response.

This is the model, same as your tutorial. I tried changing the values of the bottleneck but didnt change much.

class GCNEncoder(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super(GCNEncoder, self).__init__()
        self.conv1 = GCNConv(in_channels, 2 * out_channels, cached=True) # cached only for transductive learning
        self.conv2 = GCNConv(2 * out_channels, out_channels, cached=True) # cached only for transductive learning

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        return self.conv2(x, edge_index)

So you believe there is no efficient way to create multi-dimensional embeddings for (fully connected) graphs whose only changing feature are the coordinates?

[rusty1s]

There definitely is, but you will need more sophisticated GNN operators for this that can take spatial relations into account. For example, in GCNConv on a fully-connected graph, features across nodes will be the same (which restricts any learning). I suggest to look into torch_geometric.nn.conv.SplineConv and use the coordinates between different nodes as edge features.

[arya-h]

I have added the distance between nodes as edge features of the graph Data object, and then defined the encoder as follows

from torch_geometric.nn import SplineConv
class GraphEncoder(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super(GraphEncoder, self).__init__()
        self.conv1 = SplineConv(in_channels=in_channels, out_channels=2*out_channels, dim=2, kernel_size=3) #, space_dimensions=2, propagate_dimensions=2, k=3, num_workers=1)
        #self.conv1 = GCNConv(in_channels, 2 * out_channels, cached=True) # cached only for transductive learning
        self.conv2 = SplineConv(in_channels=2*out_channels, out_channels=out_channels, dim=2, kernel_size=3) #,space_dimensions=2, propagate_dimensions=2, k=3, num_workers=1)
        #self.conv2 = GCNConv(2 * out_channels, out_channels, cached=True) # cached only for transductive learning

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        return self.conv2(x, edge_index)

but i get the following error

RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "/home/students/s290510/.local/lib/python3.7/site-packages/torch_spline_conv/basis.py", line 10, in spline_basis
                 is_open_spline: torch.Tensor,
                 degree: int) -> Tuple[torch.Tensor, torch.Tensor]:
    return torch.ops.torch_spline_conv.spline_basis(pseudo, kernel_size,
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
                                                    is_open_spline, degree)
RuntimeError: tensor does not have a device

I realize this may be data specific but I thought I'd ask anyway. I tested other convolutional layers to see if there was any difference, but either the loss goes in the 30-40 range or it jumps values between 1.00 and 1.38 continuously.

Would you recommend any other layer for this purpose?

[rusty1s]

It looks like your are missing an edge_attr for SplineConv:

x = self.conv1(x, edge_index, edge_attr).relu()

Note that you can easily generate this by saving your coordinates in data.pos, and applying the transforms.Cartesian transform. See here for an example.

[arya-h]

Thank you for the help, much appreciated