Zygote Readme example of Jacobian etc

[azev77]

The readme would benefit w/ examples of differentiating general f: Rn -> Rm

f1(x,y)=cos(x)+sin(y)
f2(x,y)=x^2 + y^2
f3(x,y)=log(x) + exp(y)
f(x,y)= [f1(x,y);f2(x,y);f3(x,y)]

gradient(f1, 1, 2)
gradient(f2, 1, 2)
gradient(f3, 1, 2)
gradient(f, 1, 2) # should be a Jacobian, don't know how to get it to work

[AzamatB]

I am pretty sure that Zygote is currently limited to scalar-valued functions only

[azev77]

@AzamatB the following code works:

using ForwardDiff, Zygote

g1(x)=cos(x[1])+sin(x[2])
g2(x)=x[1]^2 + x[2]^2
g3(x)=log(x[1]) + exp(x[2])
gg(x)= [g1(x);g2(x);g3(x)]
x=[1.;2.]
#
Zygote.gradient(g1,x)
Zygote.gradient(g2,x)
Zygote.gradient(g3,x)
Zygote.forward_jacobian(gg,x)[2]
Zygote.hessian(g1,x)
#
ForwardDiff.gradient(g1,x)
ForwardDiff.gradient(g2,x)
ForwardDiff.gradient(g3,x)
ForwardDiff.jacobian(gg,x)
ForwardDiff.hessian(g1,x)

[azev77]

#736

[mcabbott]

xref #98, there ought to be a function jacobian.

Not sure the readme is the place to show how to cobble one together, nor how to call ForwardDiff.

[azev77]

@mcabbott

1. I agree the Readme should not call ForwardDiff.
   The only reason I did above is to compare w/ Zygote.
   PR Update README.md #736 does not call ForwardDiff.
2. I personally wasted a lot of time trying to figure out how to compute a Jacobian of a vector-valued function in Julia. Including such an example in the Readme and/or docs would likely help many beginners.

[mcabbott]

That was a bit cryptic perhaps, but Zygote.forward_jacobian is just a front-end for ForwardDiff:

Zygote.jl/src/lib/forward.jl

Line 22 in 6b44000

function forward_jacobian(f, x, ::Val{N}) where N

I agree that jacobians ought to be mentioned in the manual. But they are secondary to gradient for reverse mode -- the computation for an Rn -> Rm function is just to call the pullback m times, and cat the results. Explaining that somewhere might be worthwhile.

In fact Zygote now does forward mode too, since #503, but not yet in the manual. If m isn't small this will probably be a better way to compute jacobians, and it would make sense to discuss them together. And (hopefully) to define a pair of functions which use pullback / pushforward to get the same result.