The global fixed random seed produces different training results every time

[Tyqfat]

I encountered a problem when using pyg's GeneralConv (using edge features) for deep learning training. I have set the global fixed random seed, but during multiple runs of training, each time I get different training results. However, if I only use DNN or Transformer (without using edge features), the results of each training are the same. What is the cause of this? How should I solve this problem?
utils.py

import math
import os
import random
import time
import numpy as np
import torch

def set_random_seed(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)  
    torch.cuda.manual_seed(seed)  
    torch.cuda.manual_seed_all(seed) 
    os.environ['PYTHONHASHSEED'] = str(seed)
    torch.backends.cudnn.deterministic = True  
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.enabled = True

network.py

import torch
import torch.nn as nn
from torch_geometric.nn import NNConv, GENConv, DeepGCNLayer, GeneralConv
import math
import torch.nn.functional as F


class GeneralConvNet(nn.Module):
    def __init__(self, args: dict, node_input_size, edge_input_size, node_output_size, edge_hidden_size,
                 node_hidden_size):
        super(GeneralConvNet, self).__init__()
        self.args = args
        self.node_output_size = node_output_size
        self.node_encoder = nn.Linear(node_input_size, node_hidden_size)
        self.edge_encoder = nn.Linear(edge_input_size, edge_hidden_size)
        self.general_conv_layers_1 = GeneralConv(in_channels=node_hidden_size, out_channels=node_output_size,
                                                 in_edge_channels=edge_hidden_size, aggr='mean', directed_msg=True,
                                                 heads=1)

    def forward(self, x, edge_index, edge_attr, num_graphs):
        x = self.node_encoder(x)
        edge_attr = self.edge_encoder(edge_attr)
        x = self.general_conv_layers_1(x, edge_index, edge_attr)
        x = x.view(num_graphs, -1, self.node_output_size)
        return x

main.py

if __name__ == '__main__':
    args = vars(get_params())
    set_random_seed(args['seed'])
    train_val_test(args)

[akihironitta]

Hi @Tyqfat, do you have your full code and env details to reproduce?

I haven't followed the status throughly, but looking at existing issues like #2788 (comment), aggregations are not deterministic unless you use SparseTensor.

[Tyqfat]

Why does using the adam optimizer lead to uncertain results? I'm confused.

[rusty1s]

Are you sure it's a problem with Adam? That would be new to me.

[Tyqfat]

After re-inspecting the code, I apologize for my previous understanding; the non-deterministic behavior was not caused by the Adam optimizer, but rather stemmed from the usage of CUDA. Upon adding the following statement within the set_random_seed function, consistent results are obtained across experiments regardless of whether GeneralConv employs SparseTensor or not.

os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8" # my cuda version = 11.8 
torch.use_deterministic_algorithms(True)

utils.py
new set_random_seed function

def set_random_seed(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)  
    torch.cuda.manual_seed(seed)  
    torch.cuda.manual_seed_all(seed)  
    os.environ['PYTHONHASHSEED'] = str(seed)
    torch.backends.cudnn.enabled = True   
    torch.backends.cudnn.deterministic = True  
    torch.backends.cudnn.benchmark = False 
    os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
    print(f'CUBLAS_WORKSPACE_CONFIG: {os.environ["CUBLAS_WORKSPACE_CONFIG"]}') # You can print it out to see if the value has been set successfully
    torch.use_deterministic_algorithms(True) 

[rusty1s]

Ah, that's good to know. Do you mind contributing your seed function to seed_everything in PyG?

[Tyqfat]

Of course, I'd love to!(●'◡'●)