dataset question

[rtwod2]

Hi all,

I'm having a dataset of kernels, for which kernel there are more than one corresponding graphs. This depends on input bytes for example. I'm doing a graph regression using PyTorch geometric. For training I have 150 graphs, for validation 120 and for testing 39 graphs. Is this number of graphs enough?

I'm not sure if this is a problem, but for example in test set, the number of 39 is out of 3 kernels with different input bytes.

Thank u very much for your time and consideration.

[rusty1s]

This is totally hard to say. A similar sized graph dataset is the MUTAG dataset of the TUDataset suite (188 graphs in total), which sees heavy fluctuations in a 10-fold cross validation scenario, but works totally fine on average. I think it's best to just try it out and see how it performs in practice :)

[rtwod2]

Hello, indeed detaching was the solution. I ran some experiments, e.g., by computing the loss of the different targets individually, as I wrote in the code segment above, by taking the median loss of the output bytes (power-law distributed target) and the corresponding loss of the execution time, and also by re-weighting the individual losses as you mentioned.

• By re-weighting the individual losses I had big loss, of the order of 1.6
  ( I did it as you suggested

loss_1, loss_2 = loss_train_all.mean(dim=0)
total_loss = loss_1 + 4 * loss_2 # Weight loss_2 four times more)

• by the second one using the median loss of the power-law distributed target and the corresponding loss of the normal distribute target I have a loss
  Epoch: 0299 ['loss_train: 0.0029', 'loss_train: 0.8171']
  Epoch: 0299 ['loss_eval: 0.0023', 'loss_eval: 0.9885']

• finally by computing the loss of the different targets individually and taking it the mean out of each as in code snippet above, I have a loss

Epoch: 0299 ['loss_train: 0.0048', 'loss_train: 0.8215']
Epoch: 0299 ['loss_eval: 0.0021', 'loss_eval: 0.9812']

And the plots for the last one is:

As you can see for the execution time (normal distributed one) it seems fine. But for the outputbytes there's no improvement at all.
How could I deal with this?

[rtwod2]

Hi, I'm sorry for writing you back, but did you have the time to look at the above problem? Because I'm stuck at the moment on the bad predictions of the second variable. Do you have any ideas on how could I deal with it?

Hello, indeed detaching was the solution. I ran some experiments, e.g., by computing the loss of the different targets individually, as I wrote in the code segment above, by taking the median loss of the output bytes (power-law distributed target) and the corresponding loss of the execution time, and also by re-weighting the individual losses as you mentioned.

• By re-weighting the individual losses I had big loss, of the order of 1.6
  ( I did it as you suggested

loss_1, loss_2 = loss_train_all.mean(dim=0)
total_loss = loss_1 + 4 * loss_2 # Weight loss_2 four times more)

• by the second one using the median loss of the power-law distributed target and the corresponding loss of the normal distribute target I have a loss
  Epoch: 0299 ['loss_train: 0.0029', 'loss_train: 0.8171']
  Epoch: 0299 ['loss_eval: 0.0023', 'loss_eval: 0.9885']
• finally by computing the loss of the different targets individually and taking it the mean out of each as in code snippet above, I have a loss

Epoch: 0299 ['loss_train: 0.0048', 'loss_train: 0.8215'] Epoch: 0299 ['loss_eval: 0.0021', 'loss_eval: 0.9812']

And the plots for the last one is:

As you can see for the execution time (normal distributed one) it seems fine. But for the outputbytes there's no improvement at all. How could I deal with this?

[rusty1s]

This looks like there is zero correlation between the training and validation labels of the second variable. Do you know if there might be some out-of-distribution prediction going on for validation examples?

[rtwod2]

I'm not sure about that. I found the distribution of the second variable, for the training labels I have

for the validation labels I have:

and for the test labels I have:

How else could I check for OOD prediction?

[rusty1s]

One additional option to check for OOD is to compare the embeddings of train/val/test graphs produced by the GNN. Ideally, you would want these embeddings to be in the same space.

Nonetheless, this might only give you a better understanding of the problem, but doesn't fix your problem in the end. It is hard to give you advice without knowing your data in detail. One thing you should ask yourself is whether the model is really able to generalize and learn meaningful signals from the data to predict the correct test label (and what might be missing to achieve that). Currently, the model looks like all is does is trying to remember the training data.