Weights and biases initialization

[adhikarirsr]

How are GNN layers' weights and biases initialized by default? If I want to initialize my model using Xavier initialization and how would I do so. In general, how sensitive is model's performance to initialization? Is there a rule of thumb for choosing an initialization scheme?

I can initialize as follows:

def init_all(model, init_func, *params, **kwargs):
    for p in model.parameters():
        init_func(p, *params, **kwargs)

model = Net() 
init_all(model, torch.nn.init.normal_, mean=0., std=2)

This does not work with Xavier initialization.

init_all(modelgcn2, torch.nn.init.xavier_normal_)

Error:
ValueError: Fan in and fan out can not be computed for tensor with fewer than 2 dimensions

[rusty1s]

I don't think there is a rule of thumb for this. Most GNN layers make use of xavier initialization, while others make use of standard PyTorch Linear initialization. In my experience, initialization only has effects on smaller datasets. If you want to modify the initialization strategy of a specific operator, you can do so by following the PyTorch guidelines, see here.