How to use DataLoader with simple dataset?

[lambdaofgod]

Using DataLoader with simple torch_geometric.data.Data gives me KeyError

Minimal reproducible example:

import numpy as np
import torch
from torch_geometric import loader
from torch_geometric.data import Data 



features = np.random.rand(10,2)
edges = np.random.permutation(np.arange(10))
data = Data(torch.tensor(features), torch.tensor(edges))
loader = DataLoader(data, batch_size=1)
for batch in loader:
    break

throws

---

KeyError Traceback (most recent call last)
in
10 data = Data(torch.tensor(features), torch.tensor(edges))
11 loader = DataLoader(data, batch_size=1)
---> 12 for batch in loader:
13 break

~/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py in next(self)
515 if self._sampler_iter is None:
516 self._reset()
--> 517 data = self._next_data()
518 self._num_yielded += 1
519 if self._dataset_kind == _DatasetKind.Iterable and \

~/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py in _next_data(self)
555 def _next_data(self):
556 index = self._next_index() # may raise StopIteration
--> 557 data = self._dataset_fetcher.fetch(index) # may raise StopIteration
558 if self._pin_memory:
559 data = _utils.pin_memory.pin_memory(data)

~/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
42 def fetch(self, possibly_batched_index):
43 if self.auto_collation:
---> 44 data = [self.dataset[idx] for idx in possibly_batched_index]
45 else:
46 data = self.dataset[possibly_batched_index]

~/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py in (.0)
42 def fetch(self, possibly_batched_index):
43 if self.auto_collation:
---> 44 data = [self.dataset[idx] for idx in possibly_batched_index]
45 else:
46 data = self.dataset[possibly_batched_index]

~/.local/lib/python3.8/site-packages/torch_geometric/data/data.py in getitem(self, key)
352
353 def getitem(self, key: str) -> Any:
--> 354 return self._store[key]
355
356 def setitem(self, key: str, value: Any):

~/.local/lib/python3.8/site-packages/torch_geometric/data/storage.py in getitem(self, key)
66
67 def getitem(self, key: str) -> Any:
---> 68 return self._mapping[key]
69
70 def setitem(self, key: str, value: Any):

KeyError: 0

Relevant library versions:

torch==1.8.0+cu111
torch-cluster==1.5.9
torch-geometric==2.0.1
torch-scatter==2.0.8
torch-sparse==0.6.12

[wsad1]

Dataloader expects a Dataset object not a Data object that's why you are getting an error. Dataloader merges multiple Data objects in a Dataset and returns a minibatch. The idea here is if you have multiple graphs or Data objects in a Dataset this loader returns a subset of graphs as a single Data object in a minibatch.

On the other hand if your goal is to sample a set of nodes for training/testing from a single graph check out NeigghborLoader.

[lambdaofgod]

Thanks. Are Data objects used in batching? I mean does the DataLoader only ever return batches from single Data object?

If not, what is the use of having separate Data objects?

[wsad1]

The Dataloader always returns a Data object (provided your graph is homogenous). This Data could be many Data objects merged together or it can be a single Data object if you set batch_size=1, like @QuanticDisaster mentioned in his comment.
Hope this clarifies things.

[QuanticDisaster]

You can also use a list of data instead

loader = DataLoader([data], batch_size=1)