classification task on unsupervised GCN and GAT

[jianchaoji]

These days I'm trying to implement unsupervised GCN and GAT model on classification task. To do so, I'm trying to modify autoencoder.py and infomax_transductive.py, since they are link prediction task. I have implemented unsupervised GCN on classification task which comes from autoencoder.py.But the accurancy is pretty bad(around 40% on Pubmed). I think I make some mistakes when I modified the model. Can I get some advice on how to modify them ?

Or, if I want to apply unsupervised GCN and GAT on classification task, it is a better idea to implement my new model.

Thank you!

[rusty1s]

I think that the infomax approach produces better embeddings than the link prediction approach when one wants to use embeddings for a down-stream classification task.

If you think you made some mistakes, it might be a good idea to post your example here :)

[jianchaoji]

import os.path as osp

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch_cluster import random_walk
from sklearn.linear_model import LogisticRegression

import torch_geometric.transforms as T
from torch_geometric.nn import SAGEConv,GCNConv,GATConv
from torch_geometric.datasets import Planetoid
from torch_geometric.data import NeighborSampler as RawNeighborSampler

from torch_geometric.utils import train_test_split_edges
from torch_geometric.utils import (negative_sampling, remove_self_loops,
                                   add_self_loops)

EPS = 1e-15

dataset = 'Pubmed'#Cora Citeseer Pubmed
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
dataset = Planetoid(path, dataset, transform=T.NormalizeFeatures())
data = dataset[0]


class GCNEncoder(torch.nn.Module):
    def __init__(self, in_channels, out_channels):
        super(GCNEncoder, self).__init__()
        self.conv1 = GCNConv(in_channels, 2 * out_channels, cached=True)
        self.conv2 = GCNConv(2 * out_channels, out_channels, cached=True)
        #self.conv1 = GATConv(in_channels, 2 * out_channels)#change to GATConv if unsupervise GAT model
        #self.conv2 = GATConv(2 * out_channels, out_channels)#

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        return self.conv2(x, edge_index)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = GCNEncoder(data.num_node_features, out_channels=64)
model = model.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.1)
x, edge_index = data.x.to(device), data.edge_index.to(device)

def InnerProduct(z,edge_index,sigmoid=True):
    value = (z[edge_index[0]] * z[edge_index[1]]).sum(dim=1)
    
    
    return torch.sigmoid(value) if sigmoid else value

def InnerProductNegative(z,edge_index,sigmoid=True):
    value = (-z[edge_index[0]] * z[edge_index[1]]).sum(dim=1)
       
    return torch.sigmoid(value) if sigmoid else value

start=torch.randperm(data.num_nodes).cuda()
pos_edge_index = random_walk(edge_index[0], edge_index[1], start, walk_length=1,
                                coalesced=False).t()
def train():
    model.train()

    total_loss = 0
    negative_loss = 0
    
    optimizer.zero_grad()

    out = model(x,edge_index)
    out = F.normalize(out,p=2,dim=-1)
    pos_edge_index = random_walk(edge_index[0], edge_index[1], start, walk_length=1,
                                coalesced=False).t()
    neg_edge_index = negative_sampling(pos_edge_index,out.size(0))
    #    
    pos_loss = torch.log(InnerProduct(out,pos_edge_index,sigmoid=True)+EPS).mean()
    neg_loss = torch.log(InnerProductNegative(out,neg_edge_index,sigmoid=True)+EPS).mean()#change from neg_edge_index to edge_index
        
    loss = - pos_loss - neg_loss
             
    loss.backward()
    optimizer.step()
    
    total_loss += float(loss) * out.size(0)
    #total_loss = float(loss)
    return total_loss / data.num_nodes
    #return total_loss


@torch.no_grad()
def test():
    model.eval()
    out = model.forward(x,edge_index).cpu()   
    clf = LogisticRegression(max_iter = 2000)
    clf.fit(out[data.train_mask], data.y[data.train_mask])
    #print(out[data.train_mask])

    val_acc = clf.score(out[data.val_mask], data.y[data.val_mask])
    test_acc = clf.score(out[data.test_mask], data.y[data.test_mask])

    return val_acc, test_acc


for epoch in range(1, 201):
    loss = train()
    val_acc, test_acc = test()
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, '
          f'Val: {val_acc:.4f}, Test: {test_acc:.4f}')

[jianchaoji]

Here is my code.I have do some changes and the results becomes better. But I'm not sure my understanding is correct or not. Could you help me to check it?
Thank you!

[rusty1s]

Your code looks correct to me. You may want to measure how good your link prediction actually performs, e.g., via ROC or AP. In the end, it might well be that embeddings learned for link prediction are not really suitable for down-stream tasks such as node classification.

[jianchaoji]

Well received! Thank you for your reply!