Heterogenous message passing

[Angelo92Git]

Could someone please explain the diagram for heterogeneous message passing in the documentation:
https://pytorch-geometric.readthedocs.io/en/latest/tutorial/heterogeneous.html

There are two edge types in this case, ('paper', 'cites', 'paper') and ('author', 'writes', 'paper'). Are there three SAGEConv blocks for the messages paper -> paper, author -> paper, and paper -> author?

[rusty1s]

There exists two node types (paper, author) and three edge types: paper -> paper, author -> paper, paper -> author. That's why there exists two ReLU blocks (one for each node type), and three SAGEConv blocks (one for each edge type).

[vietngth]

Hi, I have a follow-up question: Are there any mathematical formulations describing how this message-passing scheme works?

Also, I managed to setup GAT for hetero learning, so I wonder what the attention mechanism looks like in this case.

[rusty1s]

This closely follows what https://arxiv.org/abs/1703.06103 is doing.

[vietngth]

Can you proof-check my understanding if I extend the same idea of R-GCN to GAT? Assuming that we only have one node type:

1. Aggregation and transformation step (forward pass):

$$H_i^{'(l)} = \sigma(\sum_{r \in \mathcal{R}} \sum_{j \in \mathcal{N}^{(r)}_i} \alpha^{(l, r)} _{ij} \cdot W_r^{(l)} H_j^{(l-1)}), $$ where $\mathcal{R}$ is the relational set, and $\alpha _{ij}$ represents the attention weight between node $i$ and its neighbor $j$ along edge type $r$ at layer $l$. $\mathcal{N}_i^{(r)}$ indicates the neighborhood of node $i$ with relation $r$.

3. Attention mechanism:

$$\alpha _{ij}^{(l, r)} = \text{softmax}(\text{LeakyReLU}(a^{(l, r)\top}[W^{(l)}_r H_i^{(l-1)} || W^{(l)}_r H_j^{(l-1)} ])),$$ where $a^{(l, r)}$ is a learnable attention parameter vector.

4. Multi-head aggregation and transformation step (forward pass):

$$ H_i^{'(l)} = \Big \Vert_{t=1}^{T} \sigma(\sum_{r \in \mathcal{R}} \sum_{j \in \mathcal{N}^{(r)}_i} \alpha^{(l, r, k)} _{ij} \cdot W_r^{(l)} H_j^{(l-1)}), $$

where $\alpha^{(l, r, k)} _{ij}$ denotes $k$-th-head attention coefficient of node $i$ and neighbor $j$ of relation $r$ in layer $l$.

5. Multi-head aggregation and transformation step (final layer - average instead of concatenation):

$$ H_i^{'(L)} = \sigma(\frac{1}{K} \sum^K_{k=1} \sum_{r \in \mathcal{R}} \sum_{j \in \mathcal{N}^{(r)}_i} \alpha^{(l, r, k)} _{ij} \cdot W_r^{(l)} H_j^{(l-1)}), $$

Sorry for invading the thread with a long post, this should be my last question for this! Thank you.

[rusty1s]

Yes, this looks correct. In the end, it just means that attention is computed within the neighborhood of a single edge type.