Reduce nr. of elements of ModelNet dataset

[MatteoGioia]

Hello, I was trying to shrink ModelNet by reducing the number of elements in the dataset since I'm working on google Colab. Basically I want a fixed nr. of elements for each class (say 20). Is there any way to do it which is reproducible besides pre_filtering? I tried to write my own pre_filter function but it does not work (and I am also not sure about reproducibility).

Here is the function (and the error):

cl = [0 for x in range(10)]

def shrink_ModelNet(data):
  item_class = data.y
  nr_instances = cl[item_class]
  if nr_instances < 20:
    cl[item_class] += 1
    return True

Error:

Downloading http://vision.princeton.edu/projects/2014/3DShapeNets/ModelNet10.zip
Extracting training_data/ModelNet10.zip
Processing...

---------------------------------------------------------------------------

IndexError                                Traceback (most recent call last)

[<ipython-input-7-a8cdbda9f857>](https://localhost:8080/#) in <module>
     54 batch_size = 60
     55 train_ds = AugmentedDS(root = "training_data", augmentor=augmentor, 
---> 56                        transformer=transformer, train=True)
     57 test_ds = AugmentedDS(root = "test_data", augmentor=augmentor,
     58                        transformer=transformer, train=False)

7 frames

[<ipython-input-7-a8cdbda9f857>](https://localhost:8080/#) in __init__(self, root, train, augmentor, transformer)
     22   def __init__(self, root: str, train:bool, augmentor, transformer):
     23       #self.dataset = GeometricShapes(root = root, train = train)
---> 24       self.dataset = ModelNet(root=root, train=train, name='10', pre_filter=shrink_ModelNet)
     25       self.transformer = transformer
     26       self.augmentor_transformer = Compose([augmentor,transformer])

[/usr/local/lib/python3.7/dist-packages/torch_geometric/datasets/modelnet.py](https://localhost:8080/#) in __init__(self, root, name, train, transform, pre_transform, pre_filter)
     56         assert name in ['10', '40']
     57         self.name = name
---> 58         super().__init__(root, transform, pre_transform, pre_filter)
     59         path = self.processed_paths[0] if train else self.processed_paths[1]
     60         self.data, self.slices = torch.load(path)

[/usr/local/lib/python3.7/dist-packages/torch_geometric/data/in_memory_dataset.py](https://localhost:8080/#) in __init__(self, root, transform, pre_transform, pre_filter)
     48                  pre_transform: Optional[Callable] = None,
     49                  pre_filter: Optional[Callable] = None):
---> 50         super().__init__(root, transform, pre_transform, pre_filter)
     51         self.data = None
     52         self.slices = None

[/usr/local/lib/python3.7/dist-packages/torch_geometric/data/dataset.py](https://localhost:8080/#) in __init__(self, root, transform, pre_transform, pre_filter)
     85 
     86         if self.process.__qualname__.split('.')[0] != 'Dataset':
---> 87             self._process()
     88 
     89     def indices(self) -> Sequence:

[/usr/local/lib/python3.7/dist-packages/torch_geometric/data/dataset.py](https://localhost:8080/#) in _process(self)
    168 
    169         makedirs(self.processed_dir)
--> 170         self.process()
    171 
    172         path = osp.join(self.processed_dir, 'pre_transform.pt')

[/usr/local/lib/python3.7/dist-packages/torch_geometric/datasets/modelnet.py](https://localhost:8080/#) in process(self)
     86     def process(self):
     87         torch.save(self.process_set('train'), self.processed_paths[0])
---> 88         torch.save(self.process_set('test'), self.processed_paths[1])
     89 
     90     def process_set(self, dataset):

[/usr/local/lib/python3.7/dist-packages/torch_geometric/datasets/modelnet.py](https://localhost:8080/#) in process_set(self, dataset)
    107             data_list = [self.pre_transform(d) for d in data_list]
    108 
--> 109         return self.collate(data_list)
    110 
    111     def __repr__(self) -> str:

[/usr/local/lib/python3.7/dist-packages/torch_geometric/data/in_memory_dataset.py](https://localhost:8080/#) in collate(data_list)
    104 
    105         data, slices, _ = collate(
--> 106             data_list[0].__class__,
    107             data_list=data_list,
    108             increment=False,

IndexError: list index out of range

[Padarn]

My guess here would be that your dataset has no elements after filtering, but it's a bit hard to tell. Do you have a full example to reproduce the issue?

By the way this approach is for filtering data objects, Is that what you intend? If so I personally would suggest you just choose a 'split' of graphs and keep it fixed. Probably no need to even send them to the Dataset?

[rusty1s]

Besides via pre_filter, you can also do this post-processing:

data_list = []
num_classes = defaultdict(lambda: 0)
for data ind dataset:
   if num_classes[data.y] < 20:
       data_list.append(data)
       num_classes[data.y] += 1

[MatteoGioia]

I am gonna try this asap, thank you. I tought preprocessing would be more efficient though, as I would only have to run it once. Am I wrong?

[rusty1s]

This is similar in my example, where you create data_list once, and then use this list of data objects for training. Lemme know :)

[MatteoGioia]

This is similar in my example, where you create data_list once, and then use this list of data objects for training. Lemme know :)

Works fine, thank you! My only question would be if this returns the same data_list at each execution (i.e. if I run this I always get the same items in the data_list). Thanks again for the help. 😄

[rusty1s]

Yes, this should be deterministic.