SAGE , papers100M dataset, to GPU failed

[LukeLIN-web]

In the Reddit, obgn product dataset, it can work.

class SAGE(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels, num_layers=2):
        super(SAGE, self).__init__()
        self.num_layers = num_layers

        self.convs = torch.nn.ModuleList()
        self.convs.append(SAGEConv(in_channels, hidden_channels))
        for _ in range(self.num_layers - 2):
            self.convs.append(SAGEConv(hidden_channels, hidden_channels))
        self.convs.append(SAGEConv(hidden_channels, out_channels))

 dataset = PygNodePropPredDataset(name="ogbn-papers100M", root=root)
data = dataset[0]
    model = SAGE(data.num_features, conf.hidden_channels, dataset.num_classes,
                 layers).to("cuda:0")

But when I used the papers100M dataset, it failed.

Traceback (most recent call last):
    model = SAGE(data.num_features, conf.hidden_channels, dataset.num_classes,
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 907, in to
    return self._apply(convert)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 578, in _apply
    module._apply(fn)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 578, in _apply
    module._apply(fn)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 578, in _apply
    module._apply(fn)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 601, in _apply
    param_applied = fn(param)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 905, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: CUDA error: invalid argument

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

[rusty1s]

Mh, not sure if this is PyG related. Is cuda:0 occupied by something else on your end? What happens if you call we model.cuda()?

[LukeLIN-web]

I think maybe it is because GPU memory is full. because I move model first, then move data.y failed.
y = data.y.to(rank) RuntimeError: CUDA error: invalid argument CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Does there any PyG training big dataset demo? bigger than ogb-products.

[rusty1s]

There exists a few PyG examples in the OGB repo for training on larger datasets.