Linear Operators and Gaussian Process Regression

[Osburg]

Hey :)

I would like to model data with a linear transformation of GPs (as described here https://link.springer.com/chapter/10.1007/978-3-642-21738-8_20). The GP describes the spatial distribution of a physical quantity, and the linear transformation $L$ defines the measurement process. Each measurement results in multiple ($N_{obs}$) observed values, i.e. $L$ can be formulated as a real ($N_{obs}$ * M)xM matrix, where M is the number of spatial positions at which measurement were done. Naively training an exact GP on this problems leads to infeasibly large ($N_{obs}$ * M)x($N_{obs}$ * M) covariance matrices, but you can use the woodbury trick to reduce the size of matrices to be inverted to M x M.
I implemented my GP training procedure in pytorch from scratch because I did not find a way to make use of the above mentioned simplifications within GPytorch. However, my data has additional structure (e.g. Kronecker structure) which I like to make use of using GPytorch, therefore my questions are:

• Did I miss sth and the above mentioned case (i.e., the data is described by a linear trafo and a GP) is actually natively supported by GPytorch?
• If yes, could you give me a hint what I missed please? :) How can I simultaneously exploit the Kronecker structure AND the aforementioned Woodbury identity simplifications.
• If not: Do you have any tips how I can integrate my woodbury identity simplification for the linear operator + GP, so that it will be compatible with the rest of GPytorch.

Would be very thankful for any help!

Cheers
Aaron :)

[kayween]

Did I miss sth and the above mentioned case (i.e., the data is described by a linear trafo and a GP) is actually natively supported by GPytorch?

AFAIK I don't think GPyTorch has a module implementing linear transfomation of GPs. Note that, however, a linear transformation of GPs is again a GP (with a different kernel like Eq (8) in the paper). Thus, you can implement the transformed GP in GPyTorch.

How can I simultaneously exploit the Kronecker structure AND the aforementioned Woodbury identity simplifications.

There is a way exploting the structure in GPyTorch. And GPyTorch relies on linear operators for this: https://github.com/cornellius-gp/linear_operator

If you wrap your structured covariance matrix as a LinearOperator object, then GPyTorch will automatically exploit the structure for inference (e.g., using the Woodbury identity under the hood). See the linear operator site for more examples on this.

[Osburg]

Hi @kayween, thanks for the quick answer, helps a lot :)