How to get the perfect reproducibility

[devjwsong]

Hi, I'm currently trying to finetune a pretrained BERT model for intent classification using Huggingface's Transformers library and Pytorch Lightning.
The structure is simple where a linear classifier is simply put on the BERT encoder.
I want to get the same result at the same seed setting, but although the whole setting including the seed is identical, the result changes.
I thought if I pre-fix the seed using seed_everything and set the flag workers=True, I can get the exact same result, but I don't know what the problem is.
The fun fact is that all executions save the exact same best checkpoint with identical valid accuracy and actually the flow of the training seems also the same.
But after executing the test, the results are not same.

The main codes are as follows.

from transformers import BertConfig, BertTokenizer, BertModel
from pytorch_lightning import Trainer, seed_everything
from pytorch_lightning.callbacks import ModelCheckpoint

def run(args):
    # For directory setting
    ...

    # Tokenizer & Model => This model is a pre-trained encoder, so I think there is no need to fix a random seed.
    config = BertConfig.from_pretrained('bert-base-uncased')
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
    model = BertModel.from_pretrained('bert-base-uncased')

    ...
    
    print("Loading datasets...")
    # For data loading
    train_set = Dataset(...)
    valid_set = Dataset(...)
    test_set = Dataset(...)

    total_train_steps = int(len(train_set) / batch_size * num_epochs)
    warmup_steps = int(total_train_steps * warmup_prop)
    
    # Random seed fixing for intent classification layer
    seed_everything(0, workers=True)
    
    # Lightning Module setting  
    module = TrainModule(model)

    # Dataloaders
    ppd = PadCollate(...)
    
    # Reset random seed for data shuffle
    seed_everything(0, workers=True)

    train_loader = DataLoader(train_set, collate_fn=ppd.pad_collate, batch_size=batch_size, shuffle=True, num_workers=num_workers, pin_memory=True)
    valid_loader = DataLoader(valid_set, collate_fn=ppd.pad_collate, batch_size=batch_size, num_workers=num_workers, pin_memory=True)
    test_loader = DataLoader(test_set, collate_fn=ppd.pad_collate, batch_size=batch_size, num_workers=num_workers, pin_memory=True)

    print("Setting pytorch lightning callback & trainer...")
    # Model checkpoint callback
    filename = "best_ckpt_{epoch}_{train_all_acc:.4f}_{valid_all_acc:.4f}"
    monitor = "valid_all_acc"
    
    checkpoint_callback = ModelCheckpoint(
        dirpath=save_dir,
        filename=filename,
        verbose=True,
        monitor=monitor,
        mode='max',
        every_n_val_epochs=1,
    )
    
    # Trainer setting
    trainer = Trainer(
        check_val_every_n_epoch=1,
        gpus=gpu,
        auto_select_gpus=True,
        num_nodes=num_nodes,
        max_epochs=num_epochs,
        gradient_clip_val=max_grad_norm,
        num_sanity_val_steps=0,
        deterministic=True,
        callbacks=[checkpoint_callback]
    )
    
    print("Train starts.")
    trainer.fit(model=module, train_dataloader=train_loader, val_dataloaders=valid_loader)
    print("Training done.")
    
    print("Test starts.")
    trainer.test(model=module, test_dataloaders=test_loader, ckpt_path='best')
    
    print("GOOD BYE.")

Also, TrainModule is designed like below.

from torch import nn as nn
import pytorch_lightning as pl

class TrainModule(pl.LightningModule):
    def __init__(self, args, encoder):
        super().__init__()
        
        self.args = args
        self.save_hyperparameters(args)
        
        self.encoder = encoder
        self.output_layer = IntentDetection(args)
        self.output_layer.init_params()
        
        self.loss_func = nn.CrossEntropyLoss()
        
    def forward(self, input_ids, padding_masks=None):  # input_ids: (B, L), padding_masks: (B, L)
        hidden_states = self.encoder(input_ids=input_ids, attention_mask=padding_masks)[0]  # (B, L, d_h)
            
        return self.output_layer(hidden_states[:, 0])  # (B, L, C) or  (B, C)
    
    def training_step(self, batch, batch_idx): 
        ...

class IntentDetection(nn.Module):
    def __init__(self, args):
        super(IntentDetection, self).__init__()
        
        self.hidden_size = args.hidden_size
        self.num_classes = args.num_classes
        
        self.linear = nn.Linear(self.hidden_size, self.num_classes)
        
    def forward(self, hiddens):
        # hiddens: (B, d_h)
        
        return self.linear(hiddens)  # (B, C)
    
    def init_params(self):
        nn.init.xavier_uniform_(self.linear.weight)  

I also post the current training environment.

• OS: Ubuntu 18.04.5 LTS
• Python version: 3.8.5
• Pytorch version: 1.7.1+cu110
• Pytorch Lightning version: 1.3.0
• GPU: A100-SXM4-40GB (DGX)
• CUDA version: 11.0

And I got 3 different results when I run the same codes 3 times.

This is odd, since when I run the exact same code above in different environment, I could get the identical results from a same seed, even the number of workers in a dataloader is different.
I also attach the environment which I was able to get the perfect reproducibility.

• OS: CentOs Linux release 7.9.2009 (Core)
• Python version: 3.7.4
• Pytorch version: 1.7.1+cu110
• Pytorch Lightning version: 1.3.0
• GPU: RTX 3090
• CUDA version: 11.2

It will be really great if anyone can help me to solve this issue...
Thank you very much.

[carmocca]

Can you try seeding again right before the trainer.test call?

Not saying you should need to but to know if that makes any difference

I think your best bet is to try to create a reproducible snippet. You can get started with the https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/bug_report_model.py

[devjwsong]

Thanks for replying!!
I tried it, but it didn' t work. The results are still different.

I wonder this is an environment-specific issue, since I'm current doing this on DGX environment.
I don't think there is a certain problem in my code....

Do u have any idea?

[carmocca]

What I would do is:

First, check whether the first batch always produces the same value:

• yes: then the issue appears as you test
• no: the issue is in the testing initialization

Then, I would try adding seed_everything(0) calls around the suspect code to find which piece is modifying the RNG state unexpectedly.