SVGP Prediction Expression

[AnirbanHFX]

Hi,

I am completely new to Gaussian Process Regression and was looking for some help understanding how the SVGP prediction expression looks like in terms of the parameters obtained from model.named_parameters() and likelihood.named_parameters().

For context, I wish to train an SVGP using GPyTorch and then deploy it manually on a platform that does not support GPyTorch. Therefore I need to understand what the prediction expression looks like.

I followed the example at https://docs.gpytorch.ai/en/stable/examples/04_Variational_and_Approximate_GPs/SVGP_Regression_CUDA.html with the exception that I used my own training and test data.

Here is what my GP model looks like -

class GPModel(gpytorch.models.ApproximateGP):

    def __init__(self, inducing_points):

        variational_distribution = gpytorch.variational.CholeskyVariationalDistribution(inducing_points.size(0))
        variational_strategy = gpytorch.variational.VariationalStrategy(self, inducing_points, variational_distribution, learn_inducing_locations=True)

        super(GPModel, self).__init__(variational_strategy)

        # Define Mean Function kernel
        self.mean_module = gpytorch.means.ConstantMean()

        # Define Covariance Function kernel
        self.covar_module = gpytorch.kernels.ScaleKernel(gpytorch.kernels.RBFKernel())

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)

According to titsias09a, the posterior mean of the SVGP is given by -

I obtained Kmm in this equation from model.covar_module(inducing_points).detach().numpy() and mu from gpr.variational_strategy._variational_distribution.variational_mean.detach().numpy().

I calculated Kxm appropriately using the trained kernel and cross-checked its correctness.

Since the mean of targets in my dataset should be 0, I calculated the posterior mean as described in titsias09a, and also using the following equation (which takes a constant non-zero mean into consideration) -

Note that here c is a constant scalar (obtained from model.mean_module.constant.detach().numpy() and (mu - c) is the element-wise subtraction.

However both these expressions evaluated to something that does not match the predictions from GPyTorch on the same test data. I am not sure whether I am using incorrect expressions, or wrongly interpreting the parameters.

Any help would be appreciated.

Thanks in advance!

[wjmaddox]

This is due to the fact that the variational strategy is in the whitened parameterization, see section 5.1 here although the concept dates back from here.

So you want to compute: K_{xm}K_{mm}^{-1/2} \mu in the zero mean case and c + K_{xm}K_{mm}^{-1/2} (\mu - c) in the non-zero mean case. The inverse matrix square roots should be computed via cholesky.