Batching with ClusterLoader and ClusterData

[LuisaWerner]

Hello,

I have some questions about the ClusterLoader and ClusterData implementation based on the publication "Cluster-GCN: An Efficient Algorithm for Training Deep and Large Convolutional Networks"

(1) I would like to create batches based on Cluster GCN for an inductive setting and an transductive setting. So I have to make sure that the batches in the inductive setting only contain links between training nodes while the batches in the transductive setting may include links between all nodes in the graph. How can this be realized with ClusterLoader and DataLoader?

(2) In the publication, the authors describe Stochastic Multiple Partition where batches are composed of several partitions to reduce variance. Is this included in the implementation of ClusterLoader or does it only create batches from a single partition? Is there a possibility to control this in the ClusterLoader? Since sampling from several partitions is done randomly, is there the possibility to set a seed?

(3) Is there any possibility to control how the partitions are made? In the Cluster-GCN publication the authors talk about random partition or clustering partition with METIS. What is the default partitioning procedure?

It would be great if someone can answer my questions :-) Thanks in advance!
Luisa

[rusty1s]

1. For inductive learning, simply filter your graph before passing to the ClusterData via data = data.subgraph(data.train_mask).
2. Yes, this is supported via the batch_size argument.
3. we only support METIS right now.

[LuisaWerner]

Thanks for your fast and helpful reply!

The answer of point 2. is not 100% clear for me yet. How to you mean exactly that I can specify the composition of the batch by the batch_size? Are several partitions only used if batch_size is larger than the size of a partition or how does this work exactly?

Thanks!

[rusty1s]

batch_size refers to the number of partitions you want to group together.

[LuisaWerner]

thanks for your reply @Rusty, sorry for asking so many questions about the ClusterLoader :-)
I am still not fully sure if I got this point.

So if the argument batch_size quantifies how many partitions of the ClusterData object are chosen, then that would mean that I take 3 partitions for batch size 3, for example.
But this still doesn't tell me how to determine the final size of the batch, since it depends on the size of the partitions, right?

How exactly are the partitions created? There seem to be several algorithms in the documentation of the Metis package. Or are the partitions always approximately of the same size which would then be something like |N| / number of partitions ?

I am concerned that if (even if only few) unusually large partitions are randomly chosen, the batch would still be too large for the GPU memory. In addition, I have in mind that the contribution of a batch to the overall epoch loss should be normalized by the number of nodes taken into account.

[rusty1s]

Yes, METIS will try to make all partitions have approximately the same size - I never encountered any issues with that. If you still have concerns, it might be a good idea to look into the DynamicBatchSampler which lets you determine a "batch size" based on number of total nodes/edges within a batch.

I think you are absolutely right that theoretically one should normalize the loss by the total number of nodes.

[LuisaWerner]

Thank you :-)