Fit models in SumMarginalLogLikelihood sequentially by default (#183)

Summary:
Currently, `fit_gpytorch_model` solves a single joint optimization problem to fit ModelListGP (and, more generally, any ModuleList model that has a SumMarginalLogLikelihood).

If there are only a few models then this is just fine. If there are a lot of models, each with a lot of parameters, then the dimension of the optimization problem can become quite large, and the optimizer may have a hard time optimizing the joint problem (as it's not exploiting the independence structure).

This PR changes the default behavior to fit the models sequentially, solving multiple simpler optimization problems instead a single harder one. This can be overridden by passing in `sequential=False` to `fit_gpytorch_model`.

Pull Request resolved: #183

Test Plan: Unit tests

Reviewed By: sdaulton

Differential Revision: D15980508

Pulled By: Balandat

fbshipit-source-id: 5c8be7c8b8db4ceaed6a8d1738527f788097a6ba

Author: Balandat

botorch/fit.py

@@ -9,6 +9,7 @@
 from typing import Any, Callable
 
 from gpytorch.mlls.marginal_log_likelihood import MarginalLogLikelihood
+from gpytorch.mlls.sum_marginal_log_likelihood import SumMarginalLogLikelihood
 
 from .optim.fit import fit_gpytorch_scipy
 
@@ -33,6 +34,11 @@ def fit_gpytorch_model(
         >>> mll = ExactMarginalLogLikelihood(gp.likelihood, gp)
         >>> fit_gpytorch_model(mll)
     """
+    sequential = kwargs.pop("sequential", True)
+    if isinstance(mll, SumMarginalLogLikelihood) and sequential:
+        for mll_ in mll.mlls:
+            fit_gpytorch_model(mll=mll_, optimizer=optimizer, **kwargs)
+        return mll
     mll.train()
     mll, _ = optimizer(mll, track_iterations=False, **kwargs)
     mll.eval()

test/models/test_model_list_gp_regression.py

@@ -64,11 +64,15 @@ def test_ModelListGP(self, cuda=False):
                 self.assertIsInstance(matern_kernel, MaternKernel)
                 self.assertIsInstance(matern_kernel.lengthscale_prior, GammaPrior)
 
-            # test model fitting
+            # test constructing likelihood wrapper
             mll = SumMarginalLogLikelihood(model.likelihood, model)
             for mll_ in mll.mlls:
                 self.assertIsInstance(mll_, ExactMarginalLogLikelihood)
+
+            # test model fitting (sequential)
             mll = fit_gpytorch_model(mll, options={"maxiter": 1})
+            # test model fitting (joint)
+            mll = fit_gpytorch_model(mll, options={"maxiter": 1}, sequential=False)
 
             # test posterior
             test_x = torch.tensor([[0.25], [0.75]], **tkwargs)