Access model weights

[AdarshMJ]

Hi @rusty1s @EdisonLeeeee

I wanted to know if the following way is the right method to access the model weights and possibly modify them ?

from torch_geometric.nn import GCNConv
import torch
from torch.nn import Linear
import torch.nn.functional as F
from tqdm import tqdm
from torch import linalg as LA

class GCN(torch.nn.Module):
    def __init__(self, hidden_channels):
        super().__init__()
        self.conv1 = GCNConv(dataset.num_features, hidden_channels,bias=False)
        self.conv2 = GCNConv(hidden_channels, dataset.num_classes,bias=False)
    def forward(self, x, edge_index):
        x = self.conv1(x, edge_index)
        x = x.relu()
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.conv2(x, edge_index)
        return x


model = GCN(hidden_channels=2)
with torch.no_grad():  
    for param in model.parameters():
        print(param)
print("===========================")
with torch.no_grad():  
    for param in model.parameters():
        param = param/(LA.norm(param,2))
        print("AFter")
        print(param)

Here I am basically trying to modify the initial weights by dividing them with their norm. Thanks in advance!

[EdisonLeeeee]

You need to use "inplaced" operation instead:

param.div_(LA.norm(param,2))

[AdarshMJ]

Can I also do this in the train loop?

def train():
      for param in model.parameters():
          param = param/(LA.norm(param,2))
          model.train()
          optimizer.zero_grad()  # Clear gradients.
          out = model(data.x, data.edge_index)  # Perform a single forward pass.
          loss = criterion(out[data.train_mask], data.y[data.train_mask])  # Compute the loss solely based on the training nodes.
          loss.backward()  # Derive gradients.
          optimizer.step()
          return loss

[EdisonLeeeee]

It depends. The inplaced modification on param is not available for backward propagation, so it only works for the following case:

def train():
    with torch.no_grad():
        for param in model.parameters():
            param.div_(LA.norm(param, 2))
    model.train()
    optimizer.zero_grad()  # Clear gradients.
    out = model(data.x, data.edge_index)  # Perform a single forward pass.
    # Compute the loss solely based on the training nodes.
    loss = criterion(out[data.train_mask], data.y[data.train_mask])
    loss.backward()  # Derive gradients.
    optimizer.step()
    return loss

[AdarshMJ]

Yes I got an error saying - "RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation" if I use

param.div_(LA.norm(param,2))

But If I do simply,

for param in model.parameters():
          param = param/(LA.norm(param,2))

I am able to train but I am unsure if the parameters actually update with the modification I am making. Okay so basically I want the training loop to be as usual but an extra modification is to ensure the weights are scaled by their norm

[EdisonLeeeee]

I am able to train but I am unsure if the parameters actually update with the modification I am making. Okay so basically I want the training loop to be as usual but an extra modification is to ensure the weights are scaled by their norm

No, it should not work as the param is not modified. Currently I have no idea to do so while keeping the gradients during training, sorry :(.

[AdarshMJ]

Okay thank you :)