Why does the python api training results differ drastically from direct use.

[lxrswdd]

I am doing an image classification task. I have 7 classes.
The accuracy of the model shows much difference if I train it through python API.
If I run MMpretrain train.py with configs I get validation with 85% accuracy. However, If I write my own training codes, Its training accuracy keeps growing but the validation accuracy stuck at 22%.

According to the configs file in the MMpretrain folder, the normalization values are copied, I tried 256 and 0.256, but the results are approximately the same.

preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0, 0, 0], std=[.256, .256, .256]),
])
training_set = datasets.ImageFolder(train_dir,preprocess)
train_loader = DataLoader(training_set, batch_size=32)

validation_set = datasets.ImageFolder(test_dir,preprocess)
validation_loader = DataLoader(validation_set, batch_size=32)



model = get_model('repvgg-A1_8xb32_in1k', pretrained=True)
model.head.fc = nn.Linear(model.head.fc.in_features, n_classes)

optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=0.0001) # type: ignore
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer,T_max=120) 
loss_function = nn.CrossEntropyLoss()  

train_acc_list, valid_acc_list,train_loss_list,valid_loss_list,F1s = Pytorch_trainer.train_model(model,epochs,train_loader,validation_loader,optimizer,scheduler,loss_function,earlystopper=False)


def cal_accuracy(predictions, labels):
    classes = torch.argmax(predictions, dim=1)
    return torch.mean((classes == labels).float())

def train_model(model,epoch,trainloader,validloader,optimizer,scheduler,loss_function,earlystopper=False):
    valid_loss_list = []
    train_loss_list = []
    valid_acc_list = []
    train_acc_list = []

    early_stopping = EarlyStopping(patience=10, verbose=True)
    print("Training Started")
    for epoch in range(epoch):
        
        model.train()
        train_loss = 0.0
        train_correct = 0
        valid_correct = 0
        valid_ypred_list = []
        valid_y_list= []
        for i, data in enumerate(trainloader):
            X, y = data
            y = y.long()
            if torch.cuda.is_available():
                X, y = X.cuda(), y.cuda()
                model.cuda()

            optimizer.zero_grad()
            y_pred = model(X)   
            
            # calculate loss function
            loss = loss_function(y_pred, y)    
            loss.backward()

            #Calculate Train Accuracy
            y_pred_cpu = y_pred.cpu().detach()
            y_detached = y.cpu().detach()
            # print(y_pred_cpu)

            train_correct += cal_accuracy(y_pred_cpu,y_detached)
            # update with current step regression parameters 
            optimizer.step()
            train_loss += loss.item()
        
        train_loss_list.append(train_loss/ len(trainloader))
        
        valid_loss = 0.0

        model.eval()  
        with torch.no_grad():  
            for X, y in validloader:
                y = y.long()
                
                if torch.cuda.is_available():
                    X, y = X.cuda(), y.cuda()
                    
                y_pred = model(X)
                
                # y_pred = y_pred.long()
                loss = loss_function(y_pred,y)
                valid_loss += loss.item() * X.size(0)

                # Caculate Validation Accuracy
                y_pred_cpu = y_pred.cpu().detach()
                y_detached = y.cpu().detach()
                valid_y_list.extend(y_detached)
                valid_ypred_list.extend(torch.argmax(y_pred_cpu, dim=1))
                valid_correct += cal_accuracy(y_pred_cpu,y_detached)
        
        scheduler.step(valid_loss)

        valid_loss_list.append(valid_loss/ len(validloader))
        train_acc = 100 * (train_correct / len(trainloader))
        valid_acc = 100 * (valid_correct / len(validloader))
        valid_acc_list.append(valid_acc)    
        train_acc_list.append(train_acc)

        F1s = precision_recall_fscore_support(valid_ypred_list, valid_y_list, average='macro')

        
        # F1 = 0

        if earlystopper == True:
            early_stopping(valid_loss, model)
            
            if early_stopping.early_stop:
                print("Early stopping")
                break
                   
        if epoch %1 ==0:
            print(f'Epoch {epoch+1} \t\t Training Accuracy: {train_acc} \t\t Validation Accuracy: {valid_acc}')
            print(f'\t\t\tTraining Loss: {train_loss / len(trainloader)} \t\t Validation Loss: {valid_loss / len(validloader)}')
            print("\t\t\tmacro F1 Score: ",F1s,"\n")


    return train_acc_list,valid_acc_list,train_loss_list,valid_loss_list,F1s

I have attached my codes for both self_written and the MMpretrain configs.
codes.zip
Is there any possible explanation for this?

[Ezra-Yu]

1. The data augmentation used in the training is different, and the optimizer and related settings used in the training are also different.
2. If you really want to align this accuracy, you may need to align the forward loss and backward gradients, which is a very complicated and cumbersome task.

[lxrswdd]

Would you mind sharing how can I access the data augmentation used in the training? This is because I don't see any of them in the configs files. I have tried my best to copy the configurations for the optimizer, transformation, etc. The point for me to do this is because I am going to add some other networks constructed by myself, using Python Api is easier to understand.

[Ezra-Yu]

python tools/misc/print_config.py your/config/path > my_cfg.py

And all the training data information will be in train_dataloader.

[chiguru1]

@lxrswdd if you get the preprocessing steps can you please share here thankyou .