Notebook: 01_pytorch_workflow.ipynb | Training and Testing in the same for loop

[bhandari-nitin]

Hi there,

In notebook: 01_pytorch_workflow.ipynb,
under Section 3: Train Model: Why did we use add the testing loop within the same loop where we are training the model?

torch.manual_seed(42)

# Set the number of epochs (how many times the model will pass over the training data)
epochs = 100

# Create empty loss lists to track values
train_loss_values = []
test_loss_values = []
epoch_count = []

for epoch in range(epochs):
    ### Training

    # Put model in training mode (this is the default state of a model)
    model_0.train()

    # 1. Forward pass on train data using the forward() method inside 
    y_pred = model_0(X_train)
    # print(y_pred)

    # 2. Calculate the loss (how different are our models predictions to the ground truth)
    loss = loss_fn(y_pred, y_train)

    # 3. Zero grad of the optimizer
    optimizer.zero_grad()

    # 4. Loss backwards
    loss.backward()

    # 5. Progress the optimizer
    optimizer.step()

    ### Testing

    # Put the model in evaluation mode
    model_0.eval()

    with torch.inference_mode():
      # 1. Forward pass on test data
      test_pred = model_0(X_test)

      # 2. Caculate loss on test data
      test_loss = loss_fn(test_pred, y_test.type(torch.float)) # predictions come in torch.float datatype, so comparisons need to be done with tensors of the same type

      # Print out what's happening
      if epoch % 10 == 0:
            epoch_count.append(epoch)
            train_loss_values.append(loss.detach().numpy())
            test_loss_values.append(test_loss.detach().numpy())
            print(f"Epoch: {epoch} | MAE Train Loss: {loss} | MAE Test Loss: {test_loss} ")

Don't we want to train the model, calculate loss, do the backpropagation, and update the parameters first and then run another loop to test the model?

[mrdbourke]

Hi @bhandari-nitin ,

In this case, we test the model every epoch.

As in, do a forward pass on all the training data, calculate the loss and then progress the optimizer (steps 1-5).

After those steps are done, we turn the model into eval() mode with model.eval() and then perform a forward pass on the test data and calculate the loss.

Although these steps are in the same for loop, the test calculations do not interfere with the training calculations.

You can test the model outside the training loop (e.g. only test every 5 epochs) by creating a specific function to test the model.

However, in this case, we've decided to do training & testing in one loop.

[Giubb8]

May i ask if this is a common practice in deep learning ?

I'm more familiar with ML classical models (RF,XGB,ADA,SVM,...) and using those i've always seen testing after the training loop.

Does DL common steps requires to test inside the training loop and ,in the end ,outside of it ?
or testing inside of the loop (and not outside of the loop) is the common practice ?

Thank you so much