Model not updated following the tutorial example #6249

czxttkl · 2021-02-28T09:07:07Z

czxttkl
Feb 28, 2021

Hi, I am following the very basic example from https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html#minimal-example to train my first pytorch-lightning model. However, when I print the model parameters in training_step(), I find that they never changed.

Here is the script I am using:

import os
import torch
from torch import nn
import torch.nn.functional as F
from torchvision import transforms
from torchvision.datasets import MNIST
from torch.utils.data import DataLoader, random_split
import pytorch_lightning as pl
import torch.distributed as dist
from pytorch_lightning import loggers as pl_loggers


class LitModel(pl.LightningModule):

    def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(28 * 28, 10)

    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))

    def training_step(self, batch, batch_idx):
        # training_step defined the train loop.
        # It is independent of forward
        x, y = batch
        y_hat = self(x)
        loss = F.cross_entropy(y_hat, y)
        model_params = list(self.parameters())[0][0][:10].detach().cpu().numpy()
        print(f"step end: model_params={model_params}")
        return loss

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=1.0)
        return optimizer

dataset = MNIST(os.getcwd(), download=True, transform=transforms.ToTensor())
train_loader = DataLoader(dataset)
model = LitModel()

trainer = pl.Trainer()
trainer.fit(model, train_loader)

And the output is:

(base) czxttkl@czxttkl-mbp github % python pytorch-lightning.py
GPU available: False, used: False
TPU available: None, using: 0 TPU cores
/Users/czxttkl/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/Users/czxttkl/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/Users/czxttkl/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/Users/czxttkl/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/Users/czxttkl/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/Users/czxttkl/miniconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])

  | Name | Type   | Params
--------------------------------
0 | l1   | Linear | 7.9 K 
--------------------------------
7.9 K     Trainable params
0         Non-trainable params
7.9 K     Total params
0.031     Total estimated model params size (MB)
/Users/czxttkl/miniconda3/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py:50: UserWarning: The dataloader, train dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 16 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
Epoch 0:   0%|                      | 0/60000 [00:00<?, ?it/s]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 1/60000 [00:00<04:43, 211.99it/s, loss=2.14, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                        | 2/60000 [00:00<03:32, 282.81it/s, loss=37, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 3/60000 [00:00<03:07, 319.25it/s, loss=32.6, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 4/60000 [00:00<02:55, 341.70it/s, loss=32.5, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 5/60000 [00:00<02:47, 357.63it/s, loss=26.5, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 6/60000 [00:00<02:41, 370.89it/s, loss=22.5, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 7/60000 [00:00<02:37, 381.47it/s, loss=22.9, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 8/60000 [00:00<02:35, 385.52it/s, loss=20.4, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 9/60000 [00:00<02:32, 394.13it/s, loss=18.4, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 10/60000 [00:00<02:29, 400.46it/s, loss=16.7, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 11/60000 [00:00<02:28, 405.30it/s, loss=15.4, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 12/60000 [00:00<02:25, 412.75it/s, loss=14.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 13/60000 [00:00<02:22, 419.75it/s, loss=13.4, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 14/60000 [00:00<02:21, 425.23it/s, loss=12.6, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 15/60000 [00:00<02:19, 431.16it/s, loss=11.9, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 16/60000 [00:00<02:19, 429.93it/s, loss=11.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 17/60000 [00:00<02:18, 434.45it/s, loss=10.8, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 18/60000 [00:00<02:16, 439.96it/s, loss=10.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 19/60000 [00:00<02:14, 445.29it/s, loss=9.9, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 20/60000 [00:00<02:13, 450.39it/s, loss=9.52, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 21/60000 [00:00<02:11, 455.13it/s, loss=9.53, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 22/60000 [00:00<02:10, 459.58it/s, loss=6.05, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 23/60000 [00:00<02:09, 462.40it/s, loss=4.98, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 24/60000 [00:00<02:08, 465.59it/s, loss=3.48, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 25/60000 [00:00<02:08, 468.43it/s, loss=3.48, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                     | 26/60000 [00:00<02:07, 471.12it/s, loss=3.48, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 27/60000 [00:00<02:07, 471.47it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 28/60000 [00:00<02:06, 473.52it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 29/60000 [00:00<02:06, 475.55it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 30/60000 [00:00<02:05, 477.81it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 31/60000 [00:00<02:04, 480.34it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 32/60000 [00:00<02:04, 482.79it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 33/60000 [00:00<02:03, 484.68it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 34/60000 [00:00<02:03, 487.00it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 35/60000 [00:00<02:02, 488.78it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 36/60000 [00:00<02:02, 490.68it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 37/60000 [00:00<02:01, 492.17it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 38/60000 [00:00<02:01, 493.31it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 39/60000 [00:00<02:01, 494.05it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 40/60000 [00:00<02:01, 493.49it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 41/60000 [00:00<02:01, 493.58it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 42/60000 [00:00<02:01, 494.83it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 43/60000 [00:00<02:00, 496.00it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 44/60000 [00:00<02:00, 497.46it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 45/60000 [00:00<02:00, 499.02it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 46/60000 [00:00<01:59, 500.05it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 47/60000 [00:00<01:59, 501.18it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 48/60000 [00:00<01:59, 502.43it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 49/60000 [00:00<01:59, 503.76it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 50/60000 [00:00<01:58, 504.97it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 51/60000 [00:00<01:58, 505.21it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 52/60000 [00:00<01:58, 506.47it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 53/60000 [00:00<01:58, 507.56it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 54/60000 [00:00<01:57, 508.57it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 55/60000 [00:00<01:57, 508.74it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 56/60000 [00:00<01:57, 508.88it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 57/60000 [00:00<01:57, 509.42it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 58/60000 [00:00<01:57, 509.94it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 59/60000 [00:00<01:57, 510.41it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 60/60000 [00:00<01:57, 511.14it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 61/60000 [00:00<01:57, 511.93it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 62/60000 [00:00<01:57, 511.89it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 63/60000 [00:00<01:57, 510.88it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 64/60000 [00:00<01:57, 510.42it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 65/60000 [00:00<01:57, 511.17it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|                      | 66/60000 [00:00<01:57, 512.13it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 67/60000 [00:00<01:56, 512.76it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 68/60000 [00:00<01:56, 513.41it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 69/60000 [00:00<01:56, 514.08it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 70/60000 [00:00<01:56, 514.67it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 71/60000 [00:00<01:56, 515.34it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 72/60000 [00:00<01:56, 516.16it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 73/60000 [00:00<01:55, 517.13it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 74/60000 [00:00<01:55, 518.25it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 75/60000 [00:00<01:55, 519.70it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 76/60000 [00:00<01:54, 521.38it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 77/60000 [00:00<01:54, 523.12it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 78/60000 [00:00<01:54, 524.65it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 79/60000 [00:00<01:53, 526.02it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 80/60000 [00:00<01:53, 527.59it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 81/60000 [00:00<01:53, 528.62it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 82/60000 [00:00<01:53, 529.88it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 83/60000 [00:00<01:52, 531.25it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 84/60000 [00:00<01:52, 531.78it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
Epoch 0:   0%|▏                     | 85/60000 [00:00<01:52, 532.30it/s, loss=2.3, v_num=5]step end: model_params=[ 0.00233593  0.02456592 -0.02326157 -0.03403276 -0.0023836  -0.00719094
 -0.01075585  0.02806853  0.00197827 -0.02581473]
...

My pytorch-lightning version is 1.2.1, torch version is 1.7.1.

Any insight why?

i-aki-y · 2021-02-28T11:12:47Z

i-aki-y
Feb 28, 2021

You can see that your loss gradually decreases (from ~30 to 2.3 ).
So it seems to learn something.
You are monitoring only a limited area of parameters, so what happens if you look at all the parameters?

2 replies

czxttkl Feb 28, 2021
Author

Thanks for the reply! Indeed, when I look at all parameters, there are some parameters changed for every batch. Just feel confused why only some weights change but not others.

The updated training_step():

  def training_step(self, batch, batch_idx):
        # training_step defined the train loop.
        # It is independent of forward
        x, y = batch
        y = y.reshape(-1, 1)
        y_hat = self(x)
        loss = F.mse_loss(y_hat, y)
        # Logging to TensorBoard by default
        # self.log('train_loss', loss)
        model_params = torch.cat(
            list(map(lambda x: x.detach().cpu().flatten(), self.parameters()))
        ).numpy().copy()
        model_identical = np.all(model_params==self.last_model_param)
        print(f"step end: loss={loss} model_params_identical={model_identical}")
        self.last_model_param = model_params
        return loss

i-aki-y Mar 1, 2021

I’m not sure exactly, but I think the following is a reason.

In this special single-layer case, you can explicitly write down
the loss as a function of weight elements, like this:

loss = l(f(Wx+b)),

where l(...): loss function, f(...): activation, W: weight, b: bias, x: input data.

A derivative of loss with (i,j) element of weight (W_ij) is given by

dW_ij = (dl(z)/dz) * (df(y)/dy) * (d(Wx+b)/ dW_ij)

Note, (d(Wx+b)/ dW_ij) = x_j, where x_j is the j th element of the input.

Since the MNIST pixels are binarized and their nonzero values have been localized around the center.
If you select weight elements correspond to some peripheral area (ex. W_ij with j = 1..10 ) x_j will be
highly probably zero (so dW_ij = 0). Since parameter update is proportional to dW_ij,
some parameters will not change.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Model not updated following the tutorial example #6249

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Model not updated following the tutorial example #6249

Uh oh!

czxttkl Feb 28, 2021

Replies: 1 comment · 2 replies

Uh oh!

Uh oh!

i-aki-y Feb 28, 2021

Uh oh!

czxttkl Feb 28, 2021 Author

Uh oh!

Uh oh!

i-aki-y Mar 1, 2021

czxttkl
Feb 28, 2021

Replies: 1 comment 2 replies

i-aki-y
Feb 28, 2021

czxttkl Feb 28, 2021
Author