Data splitting and dataset.data

[anthonysirico]

Is there a better way to split the dataset besides doing training = dataset[:1000] val = dataset[1000:1500] etc? Because I ran into two issues doing it this way.

1. I want to be able to sort the subset, is there a way to do that? I have dataset.shuffle() within the code, but is there a way to do it without re-writing the InMemoryDataset class, i.e. from the console?
2. Why is dataset.data the same as training.data and val.data? Why would a subset of the large dataset still show .data of the large dataset and not its own?

[rusty1s]

1. Yes, you can sort based on a pre-defined permutation, e.g.:

dataset = dataset[torch.tensor([0, 2, 4, 6])]

2. Shuffling/Splitting does not modify the underlying data (the same as a tensor slice will not copy the tensor). This allows us to not modify/copy the underlying data. If you strictly want to enforce this, try dataset = dataset[:1000].copy().

[anthonysirico]

the sort worked, thanks.

[anthonysirico]

@rusty1s whats wrong with this code?

def add_graph_label(data):
    for i in range(0,10812):
        if i <= 10812*0.25:
            binary = np.asarray([1])
            data.y = torch.tensor(binary, dtype=torch.int64)
        else:
            binary = np.asarray([0])
            data.y = torch.tensor(binary, dtype=torch.int64)
    return data

known_training = ImbCircuitDataset('test_imb', split=0.25, transform=add_graph_label)

when I look up known_training[0].y, they are all 0. Also, when I run known_training.data, there is no y.

[rusty1s]

I am not sure what's the usage of your for-loop. add_graph_label will get called for each data object individually. If you want to perform transforms based on indices. It may be best to do

class MyTransform:
    def __init__(self):
         self.i = 0

    def __call__(self, data):
        if self.i <= 10812*0.25:
            binary = np.asarray([1])
            data.y = torch.tensor(binary, dtype=torch.int64)
        else:
            binary = np.asarray([0])
            data.y = torch.tensor(binary, dtype=torch.int64)
        self.i += 1
         return data

known_training = ImbCircuitDataset('test_imb', split=0.25, pre_transform=MyTransform())

(note the usage of pre_transform to only apply the transformation once).

[anthonysirico]

i tried it and now i have no way of checking.

known_training[0]

returns

TypeError                                 Traceback (most recent call last)
Input In [12], in <cell line: 1>()
----> 1 known_training[0]

File ~\anaconda3\lib\site-packages\torch_geometric\data\dataset.py:198, in Dataset.__getitem__(self, idx)
    193 if (isinstance(idx, (int, np.integer))
    194         or (isinstance(idx, Tensor) and idx.dim() == 0)
    195         or (isinstance(idx, np.ndarray) and np.isscalar(idx))):
    197     data = self.get(self.indices()[idx])
--> 198     data = data if self.transform is None else self.transform(data)
    199     return data
    201 else:

TypeError: __init__() takes 1 positional argument but 2 were given

My code:

class _add_graph_labels:
    def __init__(self):
        self.i = 0

    def __call__(self, data):
        if self.i <= 10812*0.25:
            binary = np.asarray([1])
            data.y = torch.tensor(binary, dtype=torch.int64)
        else:
            binary = np.asarray([0])
            data.y = torch.tensor(binary, dtype=torch.int64)
        self.i += 1
        return data

I originally wrote it like yours, but you didn't have self.i += 1 and the return indented and my IDE was throwing an error.

If this doesn't work, could I create a separate list for y using a loop?

y = []
for i in range(len(known_sorted)):
    if i <= len(known_sorted)*0.25:
        binary = np.asarray([1])
        y.append(torch.tensor(binary, dtype=torch.int64))
    else:
        binary = np.asarray([0])
        y.append(torch.tensor(binary, dtype=torch.int64))

I just don't know how that would be used in combination with the DataLoader.

[rusty1s]

Keep in mind that you need to pass the transform as transform=MyTransform(). I fixed my above code snippet.