[Bug] Difference between get_fantasy_model and set_train_data

[Kilidsch]

🐛 Bug

If I understood correctly, using get_fantasy_model or using set_train_data to add data to a model should only be a question of performance, i.e. the resulting model should be the same; but get_fantasy_model is more efficient in updating the model.

In the code snippet below, I get different mean (and variance) even though in the end both models should be conditioned on the same data. Is this a numerical issue? Is my understanding of the methods wrong? Do I just use them incorrectly?

To reproduce

** Code snippet to reproduce **

import math
import torch
import gpytorch

train_x = torch.linspace(0, 1, 1000)
train_y = torch.sin(train_x * (2 * math.pi)) + torch.tan(train_x * math.pi) +torch.randn(train_x.size()) * math.sqrt(0.04)

class ExactGPModel(gpytorch.models.ExactGP):
    def __init__(self, train_x, train_y, likelihood):
        super(ExactGPModel, self).__init__(train_x, train_y, likelihood)
        self.mean_module = gpytorch.means.ConstantMean()
        self.covar_module = gpytorch.kernels.ScaleKernel(gpytorch.kernels.RBFKernel())

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)

# initialize likelihood and model
likelihood = gpytorch.likelihoods.GaussianLikelihood()
model = ExactGPModel(train_x[::20], train_y[::20], likelihood)
model.train()
likelihood.train()
training_iter = 20
# Use the adam optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=0.1)  # Includes GaussianLikelihood parameters

# "Loss" for GPs - the marginal log likelihood
mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, model)

for i in range(training_iter):
    # Zero gradients from previous iteration
    optimizer.zero_grad()
    # Output from model
    output = model(train_x[::20])
    # Calc loss and backprop gradients
    loss = -mll(output, train_y[::20])
    loss.backward()
    optimizer.step()
model.eval()
likelihood.eval()

mask = torch.ones(len(train_x), dtype=bool)
mask[::20] = False
model(train_x)
model_fantasy = model.get_fantasy_model(train_x[mask], train_y[mask])

model.set_train_data(train_x, train_y, strict=False)

test_x = torch.linspace(0, 1, 51)
model(test_x).mean - model_fantasy(test_x).mean

# tensor([-0.0150, -0.0181, -0.0215, -0.0253, -0.0294, -0.0338, -0.0383, -0.0431,
#        -0.0480, -0.0530, -0.0581, -0.0632, -0.0684, -0.0733, -0.0782, -0.0830,
#        -0.0877, -0.0918, -0.0961, -0.1000, -0.1039, -0.1075, -0.1099, -0.1131,
#        -0.1153, -0.1170, -0.1188, -0.1201, -0.1203, -0.1213, -0.1215, -0.1211,
#        -0.1205, -0.1197, -0.1183, -0.1168, -0.1146, -0.1121, -0.1091, -0.1056,
#        -0.1016, -0.0970, -0.0920, -0.0864, -0.0804, -0.0741, -0.0674, -0.0606,
#        -0.0538, -0.0470, -0.0404], grad_fn=<SubBackward0>)

** Stack trace/error message **

// Paste the bad output here!

Expected Behavior

The difference between the means should be zero (same for variance), i.e. the models should make identical predictions.

System information

Please complete the following information:

• GPyTorch Version: 1.15.dev5+g3ad794d3
• PyTorch Version: 2.7.1+cu126
• OS: Ubuntu 22.04

Additional context

Add any other context about the problem here.