108. Creating a train/test loop - Why model.train() command was not included while functionalizing the training loop?

[ShahidNawaz01]

def train_step(model: torch.nn.Module,
data_loader: torch.utils.data.DataLoader,
loss_fn: torch.nn.Module,
optimizer: torch.optim.Optimizer,
accuracy_fn,
device: torch.device = device):
train_loss, train_acc = 0, 0
model.to(device)
for batch, (X, y) in enumerate(data_loader):
# Send data to GPU
X, y = X.to(device), y.to(device)

    # 1. Forward pass
    y_pred = model(X)

    # 2. Calculate loss
    loss = loss_fn(y_pred, y)
    train_loss += loss
    train_acc += accuracy_fn(y_true=y,
                             y_pred=y_pred.argmax(dim=1)) # Go from logits -> pred labels

    # 3. Optimizer zero grad
    optimizer.zero_grad()

    # 4. Loss backward
    loss.backward()

    # 5. Optimizer step
    optimizer.step()

# Calculate loss and accuracy per epoch and print out what's happening
train_loss /= len(data_loader)
train_acc /= len(data_loader)
print(f"Train loss: {train_loss:.5f} | Train accuracy: {train_acc:.2f}%")

[jhoanmartinez]

If you are in the training loop and include model.eval() is because you also include model.train(), in this case is not necessary because you are not entering into evaluation mode, but if you enter to evaluation mode you must come back to train mode, and you add in that case the model.eval() to the loop, let me know if make sense fella.

If the question is answered please let me know, otherwise will try again, cheers my fella.

[ShahidNawaz01]

For what I learnt from the lecture is that this has nothing to do with model.eval(). Because it is the training loop not the testing or evaluation loop. While building the training loops, we were including model.train() command but the respective teacher didn't include the model.train() command while he was functionalizing the training and testing loops.

[otrturn]

I found something similar in lesson 85, after line 13. There the model_2.train() is missing.

[jhoanmartinez]

If you include model.eval() is just to ensure that the model is training, also is not mandatory to include model.eval() if you include model.train(), model.train() is like to make 100% sure that you are training and avoid anomalies from your model.

If model.eval() is not included the model will remain in the previous state, so add model.eval() is a good practice. for example if the model was in test mode before the loop and the loop does not explicitly change the mode to training maybe will lead to undesirably behavior and affect training.

If model.train() was not included while functionalizing the training loop maybe the professor was 100% sure that there were not another state before, but is not a recommended practice.

[ShahidNawaz01]

That makes sense. Thank you @jhoanmartinez

[otrturn]

The code snippet

The code looks like:

### Loop
for epoch in range(epochs):
### Training
  **model_4.train()  # This line is missing in the course example**

  y_logits = model_4(X_blob_train) # The raw output of the model (logits)
  y_preds = torch.softmax(y_logits, dim=1).argmax(dim=1) # From logits -> pred probs

  loss = loss_fn(y_logits, y_blob_train)
  acc = accuracy_fn(y_true=y_blob_train,y_pred=y_preds)
  
  optimiser.zero_grad()
  loss.backward();
  optimiser.step()

### Testing
  model_4.eval() # **Here the model is set to eval**

  with torch.inference_mode():
    test_logits = model_4(X_blob_test)
    test_preds = torch.softmax(test_logits, dim=1).argmax(dim=1)

    test_loss = loss_fn(test_logits, y_blob_test)
    test_acc = accuracy_fn(y_true=y_blob_test,y_pred=test_preds)

So, inside the loop the model is repeatedly set to model.eval().

My understanding is, that it should be set to model.train() at the beginning of the loop.

[otrturn]

I apologize for the horrible formatting

[otrturn]

For me the question is answered

[mrdbourke]

Good question!

Perhaps I can provide some clarity on this.

A model's default state is model.train().

However, when performing evaluation/testing, it's best to call model.eval() (to turn off certain functionality required for training.

So if I've made a mistake in the materials somewhere, the best practice is as follows:

When training (e.g. inside a training loop)

Call model.train().

For example:

def train_step(model, ...):
    model.train()

    # Perform training steps...

When testing/evaluating/making predictions (e.g. inside a testing loop)

Call model.eval().

For example:

def test_step(model, ...):
    model.eval()

    # Perform testing steps...

--

For more, see this resource: https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch