to_hetero fails with ValueError: Cannot generate a graph node

[nilskre]

Context

I am trying to learn general-purpose node embeddings by an auxiliary link prediction task in a self-supervised fashion on a heterogeneous graph.

Environment

pyg 2.5.2
pytorch 2.0.1

Problem description

Heterodata dataset

There are no node features in the graph (initial node features are initialized with data[node_type].x = torch.ones(data.num_nodes, 1)). The graph data model is that nodes are linked by either binary or weighted edges. There are different Person nodes in the graph. Through edges, they are connected with other nodes (which only exist once in the graph, e.g. node_name).

HeteroData(
  Person={
    num_nodes=7000,
    x=[7000, 1],
  },
  node_name={
    num_nodes=1,
    x=[7000, 1],
  },
  many_more_nodes ...

  (Person, HAS_FEATURE, node_name)={ edge_index=[2, 200] },
  (node_name, rev_HAS_TREATMENT, Person)={ edge_index=[2, 200] },
  many_more_edges_and_rev_edges ...
)

Split into train, val and test

split = T.RandomLinkSplit(
    num_val=0.1,
    num_test=0.1,
    neg_sampling_ratio=0.0,
    add_negative_train_samples=False,
    edge_types=edge_types,
    rev_edge_types=reverse_edge_types,
)
train_data, val_data, test_data = split(data)

Resulting in

HeteroData(
  Person={
    num_nodes=7000,
    x=[7000, 1],
  },
  node_name={
    num_nodes=1,
    x=[7000, 1],
  },
  many_more_nodes ...

  (Person, HAS_FEATURE, node_name)={
    edge_index=[2, 180],
    edge_label=[180],
    edge_label_index=[2, 180],
  },
  (node_name, rev_HAS_FEATURE, Person)={ edge_index=[2, 10] },
  many_more_edges_and_rev_edges ...
)

GNN model

class GNNEncoder(torch.nn.Module):
    def __init__(self, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = SAGEConv((-1, -1), hidden_channels)
        self.conv2 = SAGEConv((-1, -1), out_channels)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)
        return x 

class EdgeDecoder(torch.nn.Module):
    def __init__(self, hidden_channels):
        super().__init__()
        self.lin1 = Linear(2 * hidden_channels, hidden_channels)
        self.lin2 = Linear(hidden_channels, 1)

    def forward(self, z_dict, edge_label_index):
        row, col = edge_label_index
        z = torch.cat([z_dict['Person'][row], z_dict['SomeNode'][col]], dim=-1) # <- what should be put here as SomeNode?

        z = self.lin1(z).relu()
        z = self.lin2(z)
        return z.view(-1)

class Model(torch.nn.Module):
    def __init__(self, hidden_channels):
        super().__init__()
        self.encoder = GNNEncoder(hidden_channels, hidden_channels)
        self.encoder = to_hetero(self.encoder, data.metadata(), aggr='sum')
        self.decoder = EdgeDecoder(hidden_channels)
        # binary cross entropy loss with logits
        self.loss = torch.nn.BCEWithLogitsLoss()

    def forward(self, x_dict, edge_index_dict, edge_label_index):
        z_dict = self.encoder(x_dict, edge_index_dict)
        return self.decoder(z_dict, edge_label_index)

The error

embedding_dim = 10
hidden_channels = 32
model = Model(hidden_channels)

Results in ValueError: Cannot generate a graph node 'relu' for type 'node_name' since it does not exist. Please make sure that all node types get updated during message passing.
I am confused because the node node_name is part of the heterodata object.

Where is my fault?
Thanks in advance!

[rusty1s]

Do you mind sharing your data? It looks like node_name is not part of message passing, i.e., since it does not have a path to any other node type (although your example output suggests that this is not the case).

Btw, I feel like a lot of these node types may better be described as node features.

[nilskre]

Outdated - see last comment

Just progressed debugging one step further.
For my edge_types there are two different cases:

• Binary edges: The edge exists or not (only edge_index)
• Weighted edges: The edge exists and has a value assigned (edge_index and edge_weight)

When I remove all binary edges from my HeteroData object, the model initializes successfully using to_hetero. This raises the question: is it possible to handle both binary and weighted edges simultaneously? One potential solution is to assign a dummy weight (e.g., 1) to every binary edge, ensuring all edges have an edge_weight and thus avoiding issues with to_hetero. However, I am unsure if this is an optimal approach. Any advice would be greatly appreciated. Thank you in advance!

[nilskre]

Outdated - see last comment

After the successful model initialization (at least with the weighted edges) another question comes up in the training process.

All the examples I have seen so far (e.g. #3958 or the MovieLens example) are tailored towards predicting edges between specific nodes. The node types are hard-coded in the forward/train method. Can I learn general-purpose node embeddings in a self-supervised manner by an auxiliary link prediction task, that uses all possible edges?

Is this possible?
Unfortunately, I have not yet found an example of how to do this. Therefore any advice is greatly appreciated! Thanks in advance!

Currently, my training loop code looks like this:

# Due to lazy initialization, we need to run one model step so the number
# of parameters can be inferred:
with torch.no_grad():
    model.encoder(train_data.x_dict, train_data.edge_index_dict)

optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

def calculate_loss(loss_func, pred, target):
    output = loss_func(pred, target)
    return output

def train():
    model.train()
    optimizer.zero_grad()
    pred = model(train_data.x_dict, train_data.edge_index_dict, train_data['Person', 'i_442476006'].edge_label_index)
    target = train_data['Person', 'i_442476006'].edge_label
    loss = calculate_loss(model.loss, pred, target)
    #loss = weighted_mse_loss(pred, target, weight)
    loss.backward()
    optimizer.step()
    return float(loss)


@torch.no_grad()
def test(data):
    model.eval()
    pred = model(data.x_dict, data.edge_index_dict,
                 data['Person', 'i_442476006'].edge_label_index)
    pred = pred.clamp(min=0, max=5)
    target = data['Person', 'i_442476006'].edge_label.float()
    rmse = F.mse_loss(pred, target).sqrt()
    return float(rmse)


for epoch in range(1, 301):
    loss = train()
    train_rmse = test(train_data)
    val_rmse = test(val_data)
    test_rmse = test(test_data)
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Train: {train_rmse:.4f}, '
          f'Val: {val_rmse:.4f}, Test: {test_rmse:.4f}')

[nilskre]

When I remove all binary edges from my HeteroData object, the model initializes successfully using to_hetero. This raises the question: is it possible to handle both binary and weighted edges simultaneously? One potential solution is to assign a dummy weight (e.g., 1) to every binary edge, ensuring all edges have an edge_weight and thus avoiding issues with to_hetero. However, I am unsure if this is an optimal approach. Any advice would be greatly appreciated. Thank you in advance!

This is the only remaining question in this thread.

All other questions have been fixed in the meantime:

• to_hetero failed for me, because for some nodes, there have been too few nodes in my dataset. After removing them, it works without problems.
• The question regarding generalized training on all available edges was solved by looping over all edge types in the forward function of the EdgeDecoder.

[rusty1s]

You would need to use a dummy edge_weight if you wanna use to_hetero. Currently, to_hetero expects that all node types and all edge types share the same format.

[emremrah]

Hello. Could you elaborate how you solved the error for to_hetero? Because I'm also going through the same tutorial and I have a similar dataset, but I don't think there are too few nodes in any node type.

the data
HeteroData(
  user={
    x=[27203, 1],
    num_nodes=27203,
  },
  product={
    x=[19575, 1],
    num_nodes=19575,
    id=[19575],
  },
  location={
    x=[5035, 1],
    num_nodes=5035,
    id=[5035],
  },
  (user, interacts, product)={ edge_index=[2, 32017] },
  (user, searched, location)={ edge_index=[2, 32017] },
  (product, located_in, location)={ edge_index=[2, 32017] }
)

# the model
class GNN(torch.nn.Module):
    def __init__(self, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = SAGEConv((-1, -1), hidden_channels)
        self.conv2 = SAGEConv((-1, -1), out_channels)

    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index)
        return x


model = GNN(hidden_channels=64, out_channels=64)
model = to_hetero(model, data.metadata(), aggr="sum")

Thanks in advance!