embeddings question

[rtwod2]

Hi all,

I have some PyTorch geometric graphs, with a varying number of nodes and edges. For nodes, I have 4 features and for edges, I have 1 feature.
I was thinking of creating an embedding layer before my next layer in my architecture, like the code below:

`    self.embedding_x = nn.Embedding(n_input_dim, hidden_dim)
     self.embedding_e = nn.Embedding(e_input_dim, hidden_dim)

`
The problem is that I can't figure out what n_input_dim and e_input_dim should I have.
I was thinking that I should have for node embeddings for example: [#nodes_in_each_batchgraph, hidden_dim], and because these graphs are of a varying number of nodes I thought that n_input_dim=max_num_nodes*batch_size.
But this approach outputs the following error:

site-packages/torch/nn/functional.py", line 2044, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

So I suppose that this is because it tries to find the embeddings in the y-direction (features), not in the x-direction (num of nodes).

Could you please help me with some ideas of what's going on and how should continue?

[wsad1]

Is your goal to use torch.nn.Embedding to find an embedding for each node_id in the graph independent of the features. Not sure if this would be meaningful, unless your nodes are always in some predefined order.

Or do you want to apply a linear transform on the features before passing the graph to a GNN?

[rtwod2]

Yes, I got it. I wrote something, so as to be like a hyper-parameter. The thing is that if I'm doing this, and for example use a Normalization or a Standardization, I have float values, but this is a conflict with my Embeddings which I created writing something like AtomEncoder.
I was thinking of just normalizing only one out of the four node features, which is the one with very big integer values, but if I'm doing this then I couldn't put it inside the Embeddings, which for now is something that I want.
So, maybe I have to think something else for node feature scaling.

[rusty1s]

If you rely on embeddings, there is no need for any normalization.

[rtwod2]

Yes, this is my thinking too. I did made some runs, and it works better now. I have some overfitting, but I suppose is because I don't have many graphs for now. But I'll find a solution for this hopefully soon.

I read that for PyG >= 1.6 it support sparse-matrix multiplication GNNs. But as a general rule of thumb, this holds true for GNNs that do not make use of the central node features x_i or multi-dimensional edge features when computing messages. https://pytorch-geometric.readthedocs.io/en/latest/notes/sparse_tensor.html
I'm thinking of making my transformer sparse and instead of computing all the dot products, to compute only the ones which are connected with edges and of course taking into account the edge features. So, I wanted to ask you what do you mean as multi-dimensional edge features, would you give an example of this?

Thank you for being so patient with my never ending questions!

[rusty1s]

I mean any case in which you have a rich set of features attached to the edges (rather than just a scalar that denotes the weight of an edge). In the attention-case, it should be still possible to apply sparse-matrix multiplication (since you all you do is learning the respective weight).

[rtwod2]

Okayy I got it. So indeed I have my edge features, which are currently two, but my purpose is to have one of them inside the attention, and make changes in your graph transformer you provide. I suppose it's okay.
I' ll update this discussion hopefully this week. Thank you for your time, and your guidance.