Computing confidence region of predictions in Dirichlet GP model

[rcasero]

This is related to #461 , but I think I need more clarification.

In the simple GP regression model, the posterior predictive distribution, predictive mean and confidence bounds are computed as

observed_pred = likelihood(model(test_x))
mu = observed_pred.mean
lower, upper = observed_pred.confidence_region() 

For the Dirichlet GP model, the likelihood is omitted computing instead

test_dist = model(test_x)
pred_means = test_dist.loc

pred_samples = test_dist.sample(torch.Size((256,))).exp()
probabilities = (pred_samples / pred_samples.sum(-2, keepdim=True)).mean(0)

I think that is how you go from Fig. 2 Left (pred_means) to Fig. 2 Right (probabilities`) in the referenced paper Dirichlet-based Gaussian Processes for Large-Scale Calibrated Classification, isn't it?

However, I'm not so clear how to compute the uncertainty for the probabilities (to create the shaded areas in Fig. 2 Right), and I'd be grateful for any insights / directions.

[rcasero]

Replying to myself. Is computing the confidence region as simple as doing the sampling of the posterior

pred_samples = dist_train.sample(torch.Size((256,))).exp()
pred_samples = pred_samples / pred_samples.sum(-2, keepdim=True)

and taking these softmax samples and the same way that mean gives the estimated probabilities, quantile gives the lower and upper bounds of the confidence region?

probabilities = pred_samples.mean(axis=0)
lower_bound = pred_samples.quantile(0.025, axis=0)
upper_bound = pred_samples.quantile(1-0.025, axis=0)

[kayween]

Hi @rcasero, I think your understanding is correct.

For the left panel, the confidence region is computed for the latent function. This is can be computed by

model(test_x).confidence_region()

Under the hood, confidence_region() basically constructs the two standard deviation interval, which corresponds to the 95% confidence interval for Gaussians.

The confidence region in the right panel is more complicated. Here, we want to compute the confidence region for each class probability $p_i$, which is clearly non-Gaussian. It's very reasonable to use quantiles as the confidence region (as you mentioned). This is also what the authors did in their original codebase:
https://github.com/dmilios/dirichletGPC/blob/7e460ca07005a5aed97937d2bf2a8a47b6f8051e/run_dirichletGP_example.py#L115-L127

[rcasero]

@kayween Brilliant, thanks for the pointer to the author's code. OK, so this issue is resolved. Thanks for your help!