loss remains the same

[Pritush09]

class LinearRegressionModel2(nn.Module):
  def __init__(self):
    super().__init__()
    self.linear_layer = nn.Linear(in_features=1,out_features=1) # refer above 

  # Define the forward computation (input data x flows through nn.Linear())
  def forward(self, x: torch.Tensor) -> torch.Tensor:
      return self.linear_layer(x)

# Set the manual seed when creating the model (this isn't always need but is used for demonstrative purposes, try commenting it out and seeing what happens)
torch.manual_seed(42)
model_1 = LinearRegressionModel2()
model_1, model_1.state_dict()


# Create the loss function
loss_fn = nn.MSELoss() # MAE loss is same as L1Loss

# Create the optimizer
optimizer = torch.optim.SGD(params=model_0.parameters(), # parameters of target model to optimize
                            lr=0.01) # learning rate (how much the optimizer should change parameters at each step, higher=more (less stable), lower=less (might take a long time))


torch.manual_seed(42)

# Set the number of epochs 
epochs = 1000 

# Put data on the available device
# Without this, error will happen (not all model/data on device)
X_train = X_train.to(device)
X_test = X_test.to(device)
y_train = y_train.to(device)
y_test = y_test.to(device)

for epoch in range(epochs):
    ### Training
    model_1.train() # train mode is on by default after construction

    # 1. Forward pass
    y_pred = model_1(X_train)

    # 2. Calculate loss
    loss = loss_fn(y_pred, y_train)

    # 3. Zero grad optimizer
    optimizer.zero_grad()

    # 4. Loss backward
    loss.backward()

    # 5. Step the optimizer
    optimizer.step()

    ### Testing
    model_1.eval() # put the model in evaluation mode for testing (inference)
    # 1. Forward pass
    with torch.inference_mode():
        test_pred = model_1(X_test)
    
        # 2. Calculate the loss
        test_loss = loss_fn(test_pred, y_test)

    if epoch % 100 == 0:
        print(f"Epoch: {epoch} | Train loss: {loss} | Test loss: {test_loss}")

and after these steps loss is not going down

[Ishan-Sahu-01]

Try reducing the optimizer learning rate to 0.0001

[shisirkha]

NEVER DO IT THIS LOW learning rate

USE L1Loss not MSE loss they are different.

[sahuishan01]

What's wrong with having a low learning rate, except for the fact that the model will converge more slowly ?

[mrdbourke]

Hi @Pritush09 ,

Which notebook are you running when this happens?

What have you tried so far?

Have you tried what @Ishan-Sahu-01 suggested and lowering the learning rate to a different value?

Some things you can try when loss isn't changing:

• Changing the learning rate (try lower, then try higher)
• Change the optimizer
• Change the number of epochs
• Change the batch size (try to make it work for a batch size of 1)
• Change the model architecture (e.g. adjust the amount of layers/hidden units)

[Toyolo]

I am having the same issue, I have perfectly copied the code but my bias is greatly different and no matter what I do the loss never changes. Should I be running this in a different notebook (I have the entire course so far first two chapters up till now, in one notebook)

[sahuishan01]

Did you try doing the suggestions suggested by @mrdbourke ?

[Toyolo]

i have and nothing changes
The model learned the following values for weights and bias:
OrderedDict([('linear_layer.weight', tensor([[0.7645]])),
('linear_layer.bias', tensor([0.8300]))])

And the original values for weights and bias are:
weights: 0.7, bias: 0.3
this is with 100k epochs and adjusted learning rates
in reference to just adjusting model architecture etc seems to suggest anyone to do something they have not yet learned how or what so im unable to move beyond this loss not changing

# Set the number of epochs 
epochs = 300000

# Put data on the available device
# Without this, error will happen (not all model/data on device)
X_train = X_train.to(device)
X_test = X_test.to(device)
y_train = y_train.to(device)
y_test = y_test.to(device)

for epoch in range(epochs):
    ### Training
    model_1.train() # train mode is on by default after construction

    # 1. Forward pass
    y_pred = model_1(X_train)

    # 2. Calculate loss
    loss = loss_fn(y_pred, y_train)

    # 3. Zero grad optimizer
    optimizer.zero_grad()

    # 4. Loss backward
    loss.backward()

    # 5. Step the optimizer
    optimizer.step()

    ### Testing
    model_1.eval() # put the model in evaluation mode for testing (inference)
    # 1. Forward pass
    with torch.inference_mode():
        test_pred = model_1(X_test)
    
        # 2. Calculate the loss
        test_loss = loss_fn(test_pred, y_test)

    if epoch % 100 == 0:
        print(f"Epoch: {epoch} | Train loss: {loss} | Test loss: {test_loss}")```

![image](https://github.com/mrdbourke/pytorch-deep-learning/assets/32978117/bdc70ae4-20fa-4b7c-a307-dc887619c0e5)

[sahuishan01]

Can you also share your optimizer parameters?