Questions regarding new update

[avik-pal]

• https://github.com/SciML/Optimization.jl/blob/master/lib/OptimizationOptimisers/src/OptimizationOptimisers.jl#L96 -- I would expect iter to be epochs * i. Maybe provide an additional epoch and i fields.
• How does mini-batching interact with LBFGS and other solvers? (Given that the dataiterator is defined only for Optimisers.jl)

[Vaibhavdixit02]

https://github.com/SciML/Optimization.jl/blob/master/lib/OptimizationOptimisers/src/OptimizationOptimisers.jl#L96 -- I would expect iter to be epochs * i. Maybe provide an additional epoch and i fields.

Yeah that's fair

How does mini-batching interact with LBFGS and other solvers? (Given that the dataiterator is defined only for Optimisers.jl)

It's disabled in the current releases, but the interaction would be the same as it's implemented with Optimisers.jl, the reason I have disabled it now is that we need nested iteration of epochs and data, since with exiting packages' implementations we don't get an iterator interface it doesn't make sense, so they behave as fullbatch (which is the correct thing) I will implement a workaround for this but the previous version was strictly worse. Worth mentioning that this gives a good reason to have our implementation of these methods. I will put together a PR for the workaround within this week.

[Vaibhavdixit02]

How does mini-batching interact with LBFGS and other solvers? (Given that the dataiterator is defined only for Optimisers.jl)

I am going to have to punt this unfortunately, I don't want to put in a bandage that I don't find correct. For LBFGS etc full-batch should be used or if absolutely essential the user should iterate over the batches themselves. I will put together an explicit tutorial, but the examples in DiffEqFlux etc will be updated with this in mind already.

The iteration thing is fixed in #830

[avik-pal]

I am going to have to punt this unfortunately, I don't want to put in a bandage that I don't find correct. For LBFGS etc full-batch should be used or if absolutely essential the user should iterate over the batches themselves. I will put together an explicit tutorial, but the examples in DiffEqFlux etc will be updated with this in mind already.

That sounds reasonable