LayerNorm question

[dttsou]

Hi,

Thank you for PyG, its an amazing toolkit.

A newbie question, so hopefully easy to answer.

I have a dataset of graphs, which I am using mini-batches to train, and have implemented a network that is quite deep, and would like to add normalisations into each layer.

So, something like (input) --> Norm --> Act --> Conv

Per feature, I'd like to normalise across all nodes in the same graph. In other words, the stddev and mean to be calculated for each feature in each graph, not each feature for the entire batch.

Is it correct to use LayerNorm with the mode "graph" for this?

And, what happens if I use LayerNorm with mode "node"? Will this calculate the stddev and mean for all features in a node and normalise the feature values of the node?

Or am I actually using the wrong normalisation all together?

Many thanks in advance!

[rusty1s]

Yes, LayerNorm(mode="graph") is correct. LayerNorm(mode="node") will normalize each feature vector of each node to have mean=0/std=1.

[dttsou]

Thank you very much for the quick answer!