missing keys in dict when batching custom heterogeneous data

[dinhhoi]

Hi @rusty1s ! I am trying ti set up a heterogeneous graph of agent_id and core_tran_id nodes as below:

I successfully used HGTLoader as below:

train_loader = HGTLoader(data, num_samples=[1024] * 4, shuffle=False, input_nodes=train_input_nodes, **kwargs)

val_loader = HGTLoader(data, num_samples=[1024] * 4,input_nodes=val_input_nodes, **kwargs)

test_loader = HGTLoader(data, num_samples=[1024] * 4, input_nodes=test_input_nodes, **kwargs)

I used the transform to make sure that there also exist edges from agent_id to cell via: T.ToUndirected()(data)

but when train still missing keys in dict when: as below.

Am I missing anything to define the ### HeteroData?

[wsad1]

Your train seems to run fine, but the test loop with val_loader fails, are the train and test loop similar? Does your val_input_nodes look ok how many non-zero entries does it have?

[dinhhoi]

If I replace the val_loader with train_loader it is running thorugh.
I am wondering if I did incorrectly define val_mask
Below is the code I defined train_mask, val_mask, and test_mask

print('num_nodes',num_nodes)
print('num_train',num_train)
print('num_val',num_val)
print('num_test',num_test)
num_nodes = data['core_tran_id'].x.shape[0]
print('num_nodes',num_nodes)
print('double_check',num_train+num_val+num_test)

num_train 378676
num_val 452776
num_test 481989
num_nodes 1313441
double_check 1313441

ones = torch.ones(num_nodes)
ones[num_train:]=0
data['core_tran_id'].train_mask= ones.bool()
data['core_tran_id'].train_mask

ones = torch.ones(num_nodes)
ones[:num_train+num_val]=0

data['core_tran_id'].test_mask= ones.bool()

ones = torch.ones(num_nodes)
ones[0:num_train]=0
ones[num_train+num_val:]=0
data['core_tran_id'].val_mask= ones.bool()

print('check number of train test', sum(data['core_tran_id'].train_mask))
print('check number of val test', sum(data['core_tran_id'].val_mask))
print('check number of test test', sum(data['core_tran_id'].test_mask))

check number of train test tensor(378676)
check number of val test tensor(452776)
check number of test test tensor(481989)

The code is running with replacing to train_loader

My idea is to assign the first num_tran to the test, then next num_val to val and last test_mask is the test. Please let me know the best way to do so.
Thanks

[dinhhoi]

Also, If I loaded more data , same issue:

[dinhhoi]

Also, If I change the loader to NeighborLoader my model was trained through.

train_loader = NeighborLoader(data, num_neighbors=[10] * 2, shuffle=True, input_nodes=train_input_nodes, **kwargs)
val_loader = NeighborLoader(data, num_neighbors=[10] * 2, input_nodes=val_input_nodes, **kwargs)
test_loader=NeighborLoader(data, num_neighbors=[10] * 2,input_nodes=test_input_nodes, **kwargs)

[rusty1s]

There was a bug in HGTLoader which is now fixed, see rusty1s/pytorch_sparse#263. Can you try to upgrade torch-sparse to the latest version and try again?