Add a way to forward diff a function that uses linsolve inside

[yolhan83]

Hello, I think there should be a workaround for this mwe

using DifferentiationInterface,LinearSolve
import ForwardDiff

function f!(c,b,A,linsolve)
    A[1,1] = b[1]
    linsolve.A = A
    linsolve.b = b
    c .= solve!(linsolve).u
    return c
end

A = rand(3,3)
b = rand(3)
c = rand(3)
prob = LinearProblem(A, b)
linsolve = init(prob)
jacobian(f!,c,AutoForwardDiff(),b,Cache(A),ConstantOrCache(linsolve))

the reason is doesn't work is pretty clear linsolve.A wants a Matrix{Float64} and we're giving it a Dual, there may be a way using PrealocationTools.jl but I'm not sure how to tell linsolve to use a DiffCache or a Lazy one.

related: https://discourse.julialang.org/t/differentiationinterface-forwarddiff-unable-to-diff-ldiv-any-other-options-to-solve-linear-system-w-o-allocations-and-differentiate/131076/30
Thanks.

[ChrisRackauckas]

ForwardDiff of LinearSolve already works. But this is just a user error.

    linsolve.A = A
    linsolve.b = b

if those aren't dual valued it'll error. You cannot reuse non-AD caches for AD, they are simply not the right memory in size. This needs to use PreallocationTools.jl, specifically LazyBufferCache is the solution to this where you want to LBC a separate linsolve object for the AD world.

[ChrisRackauckas]

Let me know if you need any help writing that.