[Docs] OptimizationFunction is-in-place parameter unclear

[dnadlinger]

Currently, the OptimizationFunction docs don't really explain the boolean type parameter (iip).

The only mention is

iip: In-Place vs Out-Of-Place

For more details on this argument, see the ODEFunction documentation.

This is not very illuminiating in the OptimizationFunction case, especially as the cost function there is usually a scalar function, which would be straightforwardly considered to be out-of-place, just going by the signature – yet, as mentioned in #852, the type parameter appears to actually influence (mostly?) the form of gradients, etc. as e.g. generated by AD.

To phrase it a different way: Because OptimizationFunction is a general interface type, I'd expect iip to be meant to describe the objective function I supply, rather than to influence whatever AD/… later make of the code internally to the solvers.

[ChrisRackauckas]

It's not fully completed. There are many functions that can be in-place though, largely the functions for constraints and derivatives. So when using a scalar loss function, it would mostly correspond to the derivative functions. More work needs to be done on all of this though to fully complete it out, and it hasn't been the largest priority since most people use AD to build the derivative functions and there aren't many optimizers that don't want the in-place form. The other part are the constraint functions which can have alternative forms as well.

Though one of the things going on is with multiobjective optimization, which has multiple different forms for the optimization libraries. This is where it is likely to get the main use in the immediate future. Also as we start to build out more Julia-native optimizers which are more complex, such as versions of BFGS to use as GPU kernels, they may want fully out of place forms which then necessitates the need for distinguishing it more. So it's currently a part of the interface, but greatly underutilitized until the solver interfaces catch up.

[dnadlinger]

So would it e.g. be a good idea to add a comment to the effect that for scalar objective function, one can just leave iip at the default even though the objective function is technically of the form f(u, p) and not f(out, u, p), and that it will lead to the more tested path? As you can probably tell by my floundering around trying to ask this question, this seems like a bit of a sharp edge for new users just trying to figure out what the happy path is.

[ChrisRackauckas]

Yes it would be a good idea. Scalar output functions are always out of place, so standard single objectives are always returning the scalar no matter what IIP is set to. We should probably make that clear.