NeighborLoader with dataset with >1 graph

[wgeul]

I would like to apply NeighborLoader to a dataset that consists of multiple graphs. As the implementation relies on Data or HeteroData objects (not Dataset) I think I should apply a two step procedure:

• Iterate over each graph and apply the NeighborLoader to the graph, collect each sample in a list (train_loader_set)
• Combine the list in a DataLoader object to obtain a single final loader

I've used the following approach, but receive an unexpected error:

from torch_geometric.datasets import FakeHeteroDataset
from torch_geometric.loader import DataLoader, NeighborLoader

dataset = FakeHeteroDataset(2, edge_dim=3)

train_loader_set = []
for train_data in dataset:
    train_loader_temp = NeighborLoader(
        train_data,
        # Sample 5 neighbors for each node, 5 layers
        num_neighbors=[5, 5, 5, 5],
        # Use a batch size of 128 for sampling training nodes of type paper
        batch_size=128,
        input_nodes=("v0"),
        shuffle=True,
    )
    train_loader_set += list(train_loader_temp)

train_loader = DataLoader(train_loader_set, batch_size=4, shuffle=True)

# AttributeError
batch = next(iter(train_loader))

When I run the last line I receive this error:
AttributeError: 'int' object has no attribute 'max'

Full specification:

~/anaconda3/envs/.../site-packages/torch/utils/data/dataloader.py in __next__(self)
    679                 # TODO(https://github.com/pytorch/pytorch/issues/76750)
    680                 self._reset()  # type: ignore[call-arg]
--> 681             data = self._next_data()
    682             self._num_yielded += 1
    683             if self._dataset_kind == _DatasetKind.Iterable and \

~/anaconda3/envs/.../site-packages/torch/utils/data/dataloader.py in _next_data(self)
    719     def _next_data(self):
    720         index = self._next_index()  # may raise StopIteration
--> 721         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    722         if self._pin_memory:
    723             data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)

~/anaconda3/envs/.../site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     50         else:
     51             data = self.dataset[possibly_batched_index]
---> 52         return self.collate_fn(data)
...
--> 290             return int(value.max()) + 1
    291         elif isinstance(store, EdgeStorage) and 'index' in key:
    292             return torch.tensor(store.size()).view(2, 1)

I'm unfortunately not able to debug the above.

When I call batch = next(iter(train_loader_temp)) for each train_loader_temp (step 1), I encounter no errors. So the issue seems to stem from passing the list of HeteroData objects (train_loader_set) to the DataLoader. But this is unexpected to me, as the documentation describes that creating a data loader from a list of data objects should not be a problem (https://pytorch-geometric.readthedocs.io/en/latest/tutorial/create_dataset.html).

What am I doing wrong here? And is there a better approach for the use case of having multiple graphs in the dataset object?

[rusty1s]

It is usually not a good idea to apply NeighborLoader on a set of graphs. Are you dealing with very large graphs? Otherwise, I suggest to use DataLoader instead to simply group graphs into batches. If you want to use NeighborLoader on a set of graphs, you should merge all graphs into a single graph, and then use NeighborLoader on top:

batch = Batch.from_data_list(data_list)
loader = NeighborLoader(batch, ...)

[wgeul]

Hi @rusty1s,

Thanks for your reply. I am dealing with moderately sized graphs, but I intend to create many graphs (100s). The solution works for now, but I have to see if it will scale well, still a bit puzzled by the error.

I'll test with fake data how far I can push things with the single graph approach.

Thank you,
Willem

[wgeul]

Hi @rusty1s,

Thanks so far for your help, really appreciate your work.

This may be a stupid question, but I am actually working on a link prediction problem. I used NeighborLoader for two reasons:

• I didn't understand why LinkNeighborLoader creates new edge_label and edge_label_index, I prefer to just create a loader that samples sub graphs based on link information without index masking or supervision - maybe this is possible ?
• I didn't want to impose supervision on the link prediction problem through "edge context", leaking partially the ground truth of other edges - this is not an option in the intended end application.

also, my target link is captured in the edge_attr element (unlike common edge_label) in target_edge.

Question:
I suspect layer normalization to be important, also for the classifier/decoder (I started of with https://github.com/pyg-team/pytorch_geometric/blob/60ea78bc2fbb07bf10080889ab27265e6da1ee31/examples/hetero/hetero_link_pred.py) I notice that when I access the batch_dict element of a batch created by NeighborLoader and apply the edge_index coordinates of target_edge on the respective target and destination nodes, the batch order differs. To clarify:

row, col = edge_index
target_edge = ('source', 'to', 'destination')
b_one = batch_dict[target_edge[0]][row]
b_two = batch_dict[target_edge[-1]][col]

yields:

tensor([1, 4, 1,  ..., 0, 0, 1])
tensor([1, 0, 0,  ..., 2, 1, 0])

I think the batch mismatch is problematic and conclude from this that NeighborLoader may not be an appropriate choice.. I'm using many graphs with different feature distributions and likely need to incorporate the batch attribute in Layer Normalization (LayerNorm(hidden_channels)(z_values, batch)).

Is it possible to implement LinkNeighborLoader without supervision? How do I address the auto generated edge_label and edge_label_index elements, can I just ignore them and use edge_index and edge_attr instead?

[rusty1s]

AFAIK, LayerNorm does not require sorted batch input, so that should be totally fine.

I don't think LinkNeighborLoader is restricted to supervision. In the end, edge_label_index just holds information about which subgraphs correspond to which input edge.

[timoverlaan]

I ran into this same error, doing a similar thing. First sampling from graphs using NeighborLoader, then combining sampled parts into batches with Batch.from_data_list()). This Batch.from_data_list() function is also the part raising the error in the original question.

What I found was causing the error in my case was the fact that the NeighborLoader adds a batch_size attribute to the Data object, which the Batch.from_data_list() function is not expecting. In that function, the AttributeError: 'int' object has no attribute 'max' error is raised, because there's not such thing as the max of the int batch_size.

I fixed it by manually deleting the batch_size from each Data object, using del data.batch_size. This is probably not the best solution, and in most cases the solution described above is more elegant, but I just wanted to put this out there in case someone just wants to get around the error, instead of changing their whole setup, because I think this approach should be possible.

[rusty1s]

This is a good catch, sorry about that. Fixed it via #8601