Data to Dataloader

[Chrisable]

I have data of Data type.How to do dataloader?
Data:
Data(x=[23974, 12, 207, 2], edge_index=[2, 1722], edge_attr=[1722], y=[23974, 12, 207, 2], num_nodes=207)

[rusty1s]

It depends on what your data represents? What is the data x holds and what does it shape represents (since it does not follow the usual convention of [num_nodes, num_features])?

[Chrisable]

x: (num_samples, input_length, num_nodes, input_dim) y: (num_samples, output_length, num_nodes, output_dim)
I plan to divide the training set into several subgraphs, then train a model for each subgraph.Then input the validation set into each subgraph model, and then aggregate the output of each subgraph model. However, the num_nodes dimension of the verification set is still the num_nodes dimension before the subgraph is divided, and the num_nodes dimension of each subgraph is the num_nodes dimension after the partition, so my idea is not feasible at present.

[rusty1s]

Can you clarify? Sorry :( I think you should be able to split your data into subgraphs during training, and just train on individual subgraphs. For inference though, I suggest that you perform it on the full-graph. This way, you do not need any aggregation logic.

[Chrisable]

But when i split the data into subgraphs during training,each input of subgraphs is (num_samples, input_length, sub_num_nodes, input_dim),for example:(2000,12,100,2).
For inference,the input is (1000,12,200,2),how to input validation set into subgraph model?

[rusty1s]

I think your model should be independent of the number of nodes, right? That is, it shouldn't assume a fixed number of nodes per sample. This way, the number of nodes can vary across training/prediction.

[Chrisable]

right

[Chrisable]

In traffic prediction, if a traffic network is divided into several partitions in the training stage, each partition trains a model, and the verification set is also changed into corresponding partition according to the division in the training stage. Finally,In the verification phase，each partition has a prediction.So,should these predictions be combined or aggregated to obtain the predicted value of each node ？

[rusty1s]

I think you only need to aggregate predictions across partitions in case these partitions overlap. Is that the case for you?

[Chrisable]

If there is no overlap in each partition, is it not necessary to aggregate? If so, the model in each partition will lead to poor results due to the small number of samples.Is there any way？

[rusty1s]

I don't think there is a need to aggregate in that case. I am not an expert on the task you want to solve, but you could try to increase the partition size during inference in order to let the model know about a larger number of samples.

[Chrisable]

Thank you