Adding Root Updates for Low Rank Updates and Adding Rows/Columns

[wjmaddox]

What would be the best way to incorporate functionality to update root decompositions of symmetric psd tensors such as:
B = A + aa' or adding new rows columns (e.g.

gpytorch/gpytorch/models/exact_prediction_strategies.py

Line 183 in c74b8ed

"""

)?

I have a low rank implementation written as an UpdatedRootLazyTensor but it's suboptimal for general use because it's just a wrapper around a dense lazy tensor with an update function. Would it be best to just implement a add_low_rank to the root and a add_new_rows function analogously?

cc @Balandat @samuelstanton

[Balandat]

So from a user perspective a nice API would be if you had the ability to just add rows/columns to an existing LT and then automatically get the cheap update if you call root_decomposition on that. That might be a bit tricky to do though b/c it would require keeping a reference from the original LT to its root so it would introduce a "dependency" cycle.

[wjmaddox]

So #1499 addresses this by returning a stacked CatLazyTensor that has an overwritten root. Unfortunately, it's going to incur the cost upfront because doing it in lazy fashion is probably just going to eat a lot of "dependency" pointers.