Does Gpytorch use Analytic gradient or Automatic differentiation for training?

[mshisheh]

I am confused about how gpytorch calculates the gradients with respect to the parameters of the model. For instance, let's say I am using ExactGP with Gaussian likelihood, RBF kernel, and constant mean and using MLE (maximum likelihood estimate) for finding the parameters of the model (mean, kernel parameters, and noise). One way to calculate the gradient w.r.t parameters of the model is using analytical gradient which means taking derivative of negative log-likelihood with respect to parameters and finding the equation for each derivation. Another way is to use automatic differentiation provided by PyTorch.

Gpytorch authors have mentioned in their paper with the title of "GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration" that they are using analytical gradient or at least this is what I understood by reading the paper. Am I correct? Also, I couldn't find the code that they have implemented the analytical gradient.

Could anyone help me understand this better, please?

[Balandat]

The "automatic differentiation provided by PyTorch" does compute the analytic gradient (via back-propagation, note that there is no finite differencing or anything like that involved) - it just does so automatically.

[mshisheh]

Thanks Balandat for your quick reply. The attached image shows part the gpytorch paper regarding the training of GP and shows the negative log likelihood and it's derivative. If I understood correctly, the code for derivative equation does not exist because it is calculated by Automatic differentiation. Is that right? Thank you for your attention.

[Balandat]

correct

[mshisheh]

Thank you very much. I appreciate your time.