ClusterLoader with multiple partitioned files

[timdnewman]

Hi,

I have a some large files that I have managed to partition using ClusterData.

Q1. Am I right in thinking cluster loader only works with one file at a time?

I think I am - if I assume that to be true

Q2. What do you think they best way of generating some sensible batches from multiple files is?

I think there are a few possibilities:

1. Just do each graph in turn - easy but bad, prone to catastrophic forgetting
2. Create one loader per file and call them in a sequence - easy and better, but not good
3. Since they are all in 500 partitions create a custom loader that batches one from each file in the training set
4. Multiple loaders and select one batch at random each time.
5. Other - I'm fully open to suggestions

Thanks

[timdnewman]

I should add I'm not aiming for good here - this is to decide if something is worth investigating down the line, and the bar for that is "does this work at all?". So I'm leaning towards 2 as I was hoping it would be quick. If I did is there any reason that's a bad idea? For example - that will still use a vast amount of ram, and I've misunderstood the point of the ClusterLoader!

[rusty1s]

I think supporting ClusterLoader for multiple graphs is a bit challenging, and that's why ClusterData is designed to operate on single graphs only. Hopefully, #9448 makes it a bit easier to use for multiple graphs. If you want to use ClusterLoader with multiple graphs, the best way is to operate on a mini-batch created from a single graph for each optimization step, but use a different graph in each step.

[timdnewman]

Thanks - what I'm doing has underlying physical meaning so I've just chunked it up initially to see if that works (x>0, x<=100, y... z...)

My plan was to have that approach but it still took up too much memory. I thought it only loaded partition on the fly but the amount of time and memory it was using suggests that's not the case.

[rusty1s]

No, ClusterData currently holds all the partitions in memory.