GCNConv layer gives different results after TorchScript conversion

[btglr]

Hello,

I've noticed that the values produced by the GCNConv layer stay the same on each run whether I'm using the normal model or the TorchScript. However, every time I convert the model to TorchScript, the GCNConv layer produces different values. Those differences propagate all the way to the results of my model. Is that something to be expected?

Values produced by the normal model:

GCNConvJittable_43ff84(16, 128):

tensor([[ 0.6927,  0.0924, -0.0930,  0.0454, -0.3251,  0.0383,  0.0349, -0.2808,
          0.1160,  0.8954, -0.5267, -1.0392, -0.8184,  0.3648,  0.4481, -1.0397,
          0.3859, -0.5712, -0.8016, -0.2063, -0.1436,  0.8895,  0.1072, -0.1365,
         -0.4416,  0.0764, -0.8079, -0.4083,  1.0004,  0.6026, -0.3772,  0.7703,
         -0.0176,  0.4675,  0.5234,  0.0746,  0.1507, -0.6593, -0.0255,  0.7045,
         -0.0442, -1.0275, -0.0019,  0.3325,  0.2753,  0.8064, -0.7116, -0.1685,
         -0.7229, -0.0295, -0.0077,  0.8281, -0.5703, -1.0626,  0.9949,  0.2629,
          0.2104, -0.3522, -1.3992,  0.3707, -0.6950, -0.6135,  0.0961,  0.3160,
          0.0210,  0.6108,  0.1692, -0.4248, -0.1599, -0.2290, -0.7360,  1.1151,
         -0.1907, -0.5321,  0.2251, -0.0890, -0.0660, -0.8451, -0.7405, -0.2420,
         -0.5954, -0.1267, -0.0133, -0.2991, -0.6169, -0.7834, -0.1894, -0.2281,
          0.4442, -0.6677, -0.6881,  0.8022,  0.2414, -0.3440, -0.0906, -0.5560,
         -0.4846,  0.6239,  0.3619,  0.9581,  0.6867, -0.1558, -0.4392,  0.2911,
          0.7335,  0.2460,  0.1758,  0.5415,  0.0335, -0.6323, -0.2969, -0.3675,
         -0.2238,  0.2747,  0.3026, -0.5329, -0.2711,  0.1356,  0.0569, -0.5085,
          0.0131, -0.1172, -0.3914,  0.0117, -0.6645, -0.2705, -0.4051, -0.1271]],
       grad_fn=<AddBackward0>)

After TorchScript conversion:

<__torch__.GCNConvJittable_e2e461.GCNConvJittable_e2e461 object at 0x600003558910>: 

-0.0557 -0.0943 -0.5601  0.5444  1.0015  0.0629 -0.2132  0.7013  0.0390  0.1640
-1.4510  0.2455 -0.2719 -0.5601 -0.4455 -0.0575  0.4886 -0.3304 -0.0325 -0.1437
 0.5326  0.7250  0.1720  0.2150 -1.3447 -0.8265 -1.0737  0.7910  0.1092 -0.5185
-0.3685  0.2130 -0.6157 -0.1297 -0.4388  0.2599  0.1622 -0.0264  0.5830 -0.4394
 0.3086  0.2024 -0.5262 -1.0479 -0.5472 -0.9170  0.2938 -0.8818  0.8105  0.3199
 0.2135 -0.4537  0.3214 -0.1767 -1.0756  0.1881 -0.3581 -0.0586 -0.5617 -0.2842
-0.1465  0.0935  0.3231  0.0828  0.0665  0.7775  0.5885 -0.5465 -0.2300 -0.0467
-0.8554  0.5226 -0.6763 -0.0625 -0.1717 -0.0346  0.7810  0.2829  0.7645 -0.5535
 0.0348  0.4648  0.0241  0.6862  0.1655  0.1833 -0.4028 -0.0284  0.0898 -0.2194
-0.0107  0.5462  0.6610 -0.5715 -0.5160 -0.3711 -0.2881  0.4056  0.4144 -0.0980
-0.1956  0.5045 -0.7880  0.1464  1.3817  0.4274  0.4537  0.9082  0.7327 -0.1553
-1.2744  0.2254 -0.0353 -0.5511 -0.8738 -0.0550 -0.3105 -0.4174 -0.9022 -0.7263
 0.4351  0.1958  0.7976 -0.8130 -0.0277  0.1027 -0.6720 -0.4032

Now those values stay the same for every execution, but if I convert the same model again I'll get different values. Am I missing something?

Thanks!

[rusty1s]

Do you have a small example to reproduce? The set of parameters after calling jittable() should be the same as the original ones, and there definitely shouldn't be any difference in outcome (we actually test against this multiple times across all GNN layers).

[btglr]

Well, I had started working on a minimal example to recreate the issue which is when I realized that the values were changing on every run even without the torchscript conversion. The reason was that I actually wasn't loading my previously saved model when converting...

So I had

model = ModelHeterogeneous(input_dim=args.input_dim, hidden_dim=args.hidden_dim, ngcn=args.ngcn, nmlp=args.nmlp)
ts_model = torch.jit.script(model)

instead of

model = ModelHeterogeneous(input_dim=args.input_dim, hidden_dim=args.hidden_dim, ngcn=args.ngcn, nmlp=args.nmlp)

model.load_state_dict(torch.load(args.model_path))
model.eval()

ts_model = torch.jit.script(model)

Sorry for the trouble!

[rusty1s]

Super, thanks for confirming!