Opinion on using torchdata for graph data processing?

[vikasd22]

Hi everyone and @rusty1s,

I have been recently suggested to use torchdata for processing my graph data. I am currently using pandas and my data requires a ton of preprocessing to be in the shape acceptable for training. However, there is a possibility that I can not work with graph that will not fit in memory. Is torchdata recommended in such scenarios?

I have been diving in the documentation of torchdata but I feel like don't understand the syntax or purpose that well. Also, I can not find many tutorial about graph + torchdata except this from @rusty1s . Is anyone using torchdata successfully for graph projects? or is there any alternative for doing graph data processing?

Thanks,

[rusty1s]

Hi @vikasd22. I must admit that I haven't used it in a big project yet. I think you can leverage it to pre-process your graph, but you would still want to store the processed data on disk for faster access during training. Otherwise, you need to re-compute the data for every single epoch.