Could anyone help me to understand the simplification of Monet GMM Kernel expression as compared to the original paper?

[abose17]

Dear altruist, I was going through the Monet implementation in PyG (https://pytorch-geometric.readthedocs.io/en/latest/_modules/torch_geometric/nn/conv/gmm_conv.html#GMMConv) powered by the following titled paper "Geometric deep learning on graphs and manifolds using mixture model cnns" where they used GMM density function as kernel or weighting function. I have two questions, one is regarding the optimization of the equation presented in the paper and another is about the formulation of that optimized equation in the PyG implementation.

1. GMM density function: (1/((2*pi)^(k/2)cov^(1/2)) * exp (-0.5(x-mean)^T * cov^(-1) * (x-mean)) but the paper just presented only exp (-0.5(x-mean)^T * cov^(-1) * (x-mean)) this part to be used as the kernel. So, is the first part (1/((2*pi)^(k/2)cov^(1/2)) of the equation not necessary or omitted? This is not even a constant that the derivation of this would result 1.
2. Even though, I skip the above question, I could not go further after noticing the simplification of the above optimized expression implemented as following.

gaussian = -0.5 * (edge_attr.view(E, 1, D) - self.mu.view(1, K, D)).pow(2)
gaussian = gaussian / (EPS + self.sigma.view(1, K, D).pow(2))
gaussian = torch.exp(gaussian.sum(dim=-1))

I think I am ignorant about the above simplification process. The first line code generates a tensor of shape (E, K, D) and last line makes it (E, K). In the paper, it is mentioned that the mean is (d x 1) matrix and cov is a (d x d) matrix for each kernel. However, in the implementation the cov is (d x 1) matrix which can not be inversed. I did not get why the pow() functions have been used here in both cases, if I correctly follow the optimized expression. Could anyone please give me explanations of how these things are working here?

[rusty1s]

1. I think it is best to reach out for this directly to the authors.
2. Covariants are restricted to have a diagonal form, which lets us represent them as vectors. It's a long time ago since I looked into this, but the pow(2) call should be a direct result of this, i.e. we do not multiply (edge_attr - mu) from left and right anymore.

[abose17]

Matthias, thanks for answering my question. I have known from you that Covariance has been considered as a vector programmatically. It makes some sense to me in case of calculating the numerator part of the expression (x-mean) (x-mean) but I do not understand the calculation of inverse covariance in the denominator part. I did not get how pow(2) function is used to represent that expression even though so far I know pow() just returns a matrix/vector of same shape as the given one with in place powered values. Could you please clarify? Thanks in advance.

[rusty1s]

Yes, the pow(2) stems from the double (x-mean) expression. You are right that it is not necessary for the covariance part. It is needed though if you thing of sigma holding standard deviations rather than variances. I am not entirely sure why the authors implemented it that way, but I assume it leads to more stable training.

[abose17]

Thank you so much Matthias. I highly appreciate your contribution to the Graph ML society.