Video 67: Comments on bad predictions of the model

[slawomirwojtas]

https://colab.research.google.com/drive/1sdLTBoyL5TONCta9UquLk3zi-Oq3R_De?usp=sharing Oh no, why the link, isn't working...

I am shocked how badly this model works with simple linear regression data.

The only change I've made is a little shuffling for train/test sets, I don't know if that way is conventional.

dataset = TensorDataset(X, y)
print(dataset[0])

example_index = 0
sample = dataset[example_index]
X_sample, y_sample = sample
print("Example X:", X_sample)
print("Example y:", y_sample)

Xy_train, Xy_test = train_test_split(dataset, test_size=0.2, random_state=42)
print("Train dataset sample")
print(Xy_train[:5])
print("Test dataset sample")
print(Xy_test[:5])

X_train_list = [item[0] for item in Xy_train]
y_train_list = [item[1] for item in Xy_train]
X_test_list = [item[0] for item in Xy_test]
y_test_list = [item[1] for item in Xy_test]

X_train = torch.cat(X_train_list, dim=0).unsqueeze(dim=1)
y_train = torch.cat(y_train_list, dim=0).unsqueeze(dim=1)
X_test = torch.cat(X_test_list, dim=0).unsqueeze(dim=1)
y_test = torch.cat(y_test_list, dim=0).unsqueeze(dim=1)
print(len(X_train), len(y_train), len(X_test), len(y_test))

Also, I changed the order of actions regarding optimizer - it make more sense to me to clear gradients after taking a step. This way I have two blocks in the loop: one about the loss, the other about the optimizer. Can it cause any issues?

model_2.train()
y_pred = model_2(X_train)
loss = loss_fn(y_train, y_pred)
loss.backward()
optimizer.step()
optimizer.zero_grad()

Anyways, 300 epochs and I had to decrease learning rate because the model was not converging at the default 0.01 - the loss had two values again and again:

Epoch: 242 | Train loss: 23.505 | Test loss: 9.782
Epoch: 243 | Train loss: 2.260 | Test loss: 2.621
Epoch: 244 | Train loss: 23.505 | Test loss: 9.782
Epoch: 245 | Train loss: 2.260 | Test loss: 2.621
Epoch: 246 | Train loss: 23.505 | Test loss: 9.782

Does it mean the learning ratio is too high? Event with smaller learning rate of 0.001 the quality of predictions is rather low. How come we had better results during the lessons?

[mrdbourke]

Hi @slawomirwojtas ,

There could be several reasons your model isn't working.

It's hard to tell without seeing the full model/training/data setup (the Google Colab link you shared isn't working).

--

The order of your optimizations shouldn't matter too much (e.g. optimizer.zero_grad() after optimizer.step()), however, I've always seen & used it in the reverse order.

--

The learning rate is one of the most important values you can tune.

But it depends on your data/model setup.

Generally, values of 0.01 or 0.001 are good starting points but with a simple dataset/model, the learning rate can highly effect results.

[shisirkha]

I could give a simple answer to it.

When you shuffle the data (aka train and test) you make it not serial (like tensor([0.234 , 0.432 , ....])). but this like serial (tensor([0.312 , 0.333 , 0.343 , ....]) is needed for a Linear ML model.

Linear means straight lines that mean no upside down. (like tensor([0.234 , 0.432 , ....]))

so, it can't predict on the data because it gets nonlinear data.

For this problem you can use Nonlinear regression methods. that can handle the nonlinear data.

(@mrdbourke does my answer is correct?)

For more information see this article - https://statisticsbyjim.com/regression/curve-fitting-linear-nonlinear-regression/

I hope it helps.

I hope it helps understand the meaning of linearity. (In machine learning Linear means serial data not shuffled data)