Create a large dataset from torch_geometric.data.Dataset

[Nokimann]

I read the torch_geometric.data.Dataset class for a large dataset in docs

I expect my dataset will be over ~50 GB. But, the example in docs seems suitable for a "single" massive graph data.

Since my dataset is composed of "multiple" graphs, I wonder if I can use torch_geometric.data.Dataset.
There is no part data_list with collate like in torch_geometric.data.InMemoryDataset

Does .len() and .get() solve these problems?

[rusty1s]

Yes, the standard torch_geometric.data.Dataset will only hold the data that is passed to the model as input. By specifiying len and get, you will tell the dataset how many examples it contain and how to load individual examples from file. As such, the collate method (that bundles all examples together to load them efficiently) is not needed here.

[Nokimann]

@rusty1s Thank you always.

Then, how can we assign the data_list to torch.save(data, osp.join(self.processed_dir, f'data_{idx}.pt'))?
Just replacing data with data_list (=[Data(...)])?

---

Oh, I've just understood. The individual graph with index idx will be saved as data_{idx}.pt in the same folder!

[rusty1s]

Yes, exactly.

[abitofalchemy]

I have list of graphs (torch_geometric.data objects), I want to put them into a dataset and then shuffle them.

class MyOwnDataset(torch_geometric.data.Dataset):
    def __init__(self, data_list, transform=None):
        super(Dataset, self).__init__()
        self.data_list = data_list
        self.transform = transform

    def __len__(self):
        return len(self.data_list)

    def __getitem__(self, idx):
        data = self.data_list[idx]
        data = data if self.transform is None else self.transform(data)
        return data

but the resulting object dataset = MyOwnDataset(list_of_graphs) does not have 'data' composed of all the graphs. I can only do dataset[0] and not dataset.data. if I replace torch_geometric.data.Dataset with InMemoryDataset, in the init section I have this line self.data, self.slices = self.collate(listOfDataObjects); this works but the number of slices 3 instead of the 1339 graphs in my original list_of_graphs variable that I started out with. Why is the # of slices 3?

Working with Python 3.10 and torch_geometric 2.6.1;

[abitofalchemy]

I figured it out, looking at a test in the pyg test folder, I figure I could just use this and it allows me to do shuffle(). Using this approach does show me that I have one large graph and len(dataset) is now correct it matches the nbr of input graphs:

class MyTestDataset(InMemoryDataset):
    def __init__(self, data_list, transform=None):
        super().__init__(None, transform=transform)
        self.data, self.slices = self.collate(data_list)
        
dataset = MyTestDataset(data_graphs)
dataset.shuffle()

[garyhsu29]

Hello, I have a follow-up question here, what would be the suggestion when I have a single large graph over 50GB? Should I need to cut the graph into multiple subgraphs first?

[rusty1s]

As long as you can hold the graph in CPU memory, you should be fine. Otherwise, splitting the graph into multiple sounds like a good idea. We are also working on letting PyG scale to out-of-memory graphs, which will make this process much smoother.