Custom dataset without graph structure information.

[cjfcsjt]

I have a tabular data with shape N*F, N denotes the training sample number and F denotes the features number. I want to construct complete graph (F nodes) for each training sample. Now I have two ways to do this:
First, construct all graphs in advance and then use Dataloader to generate batch graphs:

class GraphDataset(torch_geometric.data.Dataset):
    def __init__(self, darray):
        super().__init__(darray)
        self.darray = darray
        self.size = self.darray.shape[0] # N
        self.num_features = self.darray.shape[1] # F
        self.src_nodes, self.dst_nodes = zip(*list(product(range(self.num_fields), repeat=2)))
        self.graphs = [Data(x = darray[idx], edge_index = torch.tensor([list(self.src_nodes),list(self.dst_nodes)], dtype=torch.long)) for idx in range(self.size)]
    
    def get(self,index):
        graphs = self.graphs[index]
        y = self.darray[index, -1]
        return graphs, y

Second, construct graphs when loading batches:

class GraphDataset(torch_geometric.data.Dataset):
    def __init__(self, darray):
        super().__init__(darray)
        self.darray = darray
        self.size = self.darray.shape[0] # N
        self.num_features = self.darray.shape[1] # F
    def get(self,index):
        X = self.darray[index, :]
        src_nodes, dst_nodes = zip(*list(product(range(self.num_features), repeat=2))) # fully connected graph
        graphs = Data(x = torch.tensor(X), edge_index = torch.tensor([list(src_nodes),list(dst_nodes)], dtype=torch.long)) # 
        y = self.darray[index, -1]
        return graphs, y

When N is huge, the first method is out-of-memory while the second method is very slow (If I do not generate the Data object and utilize the (X,y) purely, it will cost 30 minutes/epoch. If I generate the Data object, it will cost 50 minutes/epoch).

Is there are any memory/computation friendly way for handling batches of complete graphs?

[rusty1s]

As far as I understand, you want to represent each feature as a node, such that you have N graphs, each with F nodes that are fully-connected. Is that correct? I don't think a GNN makes sense here, and the same task is better achieved by a simple MLP (which already aggregates from all F nodes by default).

Edit: You can construct the fully-connected edge_index in __init__, and simply use it during get() (as it is always the same):

def __init__(self, ...)
    self.edge_index = ...
    
def get(self, index):
    return Data(x=self.darray[index],  edge_index=self.edge_index)