In the Workflow exercise solutions, the location of requires_grad is different to the lecutre notebook

[wittyalias]

In the 01_pytorch_workflow.ipynb in section 2. Build Model, the class initialization code has the requires_grad outside of the 'torch.randn()'. However, in the exercise answers to the related question, the requires_grad is inside the torch.randn() (code snippets below).

In trying to figure out what the difference between these two were, I had a look at the nn.Parameter documentation and it looks like requires_grad defaults to True. So unless it somehow takes on the requires_grad from the tensor that is passed to it, then putting the requires_grad in the torch.randn() won't do anything at all.

So, questions:

• is there any difference in practice between these two?
• does the requires_grad within the torch.randn() have any potentially ill effects?

from 01_pytorch_workflow.ipynb

self.weights = nn.Parameter(torch.randn(1, # <- start with random weights (this will get adjusted as the model learns)
                                                dtype=torch.float), # <- PyTorch loves float32 by default
                                   requires_grad=True) # <- can we update this value with gradient descent?)

from 01_pytorch_workflow_exercise_solutions.ipynb

self.weight = nn.Parameter(data=torch.randn(1, 
                                              requires_grad=True,
                                              dtype=torch.float
                                              ))

[mrdbourke]

Hi @wittyalias (good GitHub name too btw),

Good question!

As far as PyTorch is concerned, both of these are the same in terms of requires_grad.

Your assumption is right that nn.Parameter takes the requires_grad parameter of the tensor passed to it.

But also, even if we didn't set requires_grad=True, nn.Parameter has it set to True by default, see: https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html

I set it explicitly in the videos to showcase an example.

But you can also check the two above are the same via:

import torch
from torch import nn

# Grad outside torch.randn()
grad_outside = nn.Parameter(torch.randn(1, # <- start with random weights (this will get adjusted as the model learns)
                                        dtype=torch.float), # <- PyTorch loves float32 by default
                                        requires_grad=True) # <- can we update this value with gradient descent?)

# Grad inside torch.randn()
grad_inside = nn.Parameter(data=torch.randn(1, 
                                            requires_grad=True,
                                            dtype=torch.float))

assert grad_outside.requires_grad == grad_inside.requires_grad

(the code above will pass the assertion)

Or:

grad_outside, grad_inside

Output:

(Parameter containing:
 tensor([0.0031], requires_grad=True),
 Parameter containing:
 tensor([0.9672], requires_grad=True))

See a demo notebook here: https://colab.research.google.com/drive/1dge-FUOp9T06JGu92yNLEIPBgiMP5MFr?usp=sharing