more NF examples

[zuhengxu]

as per issue #15

• implement a set of famous NFs illustrating the usage of NormalizingFlows.jl + Bijectors.jl (a tentative list)
  • planar and radial flow
  • Real NVP
  • Neural Spline flow
  • Sparse Hamiltonian flow

[zuhengxu]

Could we add a script to test whether all the example models run, e.g. on buildkite GPU?

@yebai Currently all the examples are only tested on CPUs. Do you have examples of such script (for testing runs) that I can refer to? (I did do include(...jl) manually for all the demos though.)

In terms of GPU, shall we first merge this PR and do GPU examples/test (#49) on a separate PR? the purpose this PR is just to demonstrate how to define and customize flow layers (at least for those popular flows). I think that to properly take advantage of GPU, we might want some interface adjustment.

[yebai]

Currently all the examples are only tested on CPUs. Do you have examples of such a script (for testing runs) that I can refer to? (I did do include(...jl) manually for all the demos though.)

It is probably okay to create a new CI group called NF Examples and include all examples there. If running all examples is time-consuming, we could run only a few iterations and not wait for the convergence. This is mainly to ensure all examples can still run with new releases or PRs.

In terms of GPU, shall we first merge this PR and do GPU examples/test (#49) on a separate PR?

No problem -- it sounds good!

[sunxd3]

General, this is great effort!

Some minor comments, I didn't run the code to see the results.

In the future, probably all of these should be in a documentation page that automatically run and render results. But this PR is fine as it is.
(Worth it to add an issue as TODO?)

[zuhengxu]

Hi! I'm back to finish up this PR. Aside from adding the CI to ensure all examples run, I also played with the batched elbo objective elbo_batch. For flows that supported batched transformations, simply changing the objective from elbo to elbo_batch achieves significant speedups.

For instance, see the running time difference between the two objective on the RealNVP demo. The un-batched elbo:

and the batched elbo:

This batched objective is also important to get GPU working efficiently. In the meantime, I attempted getting GPU working and am having a pretty good understanding on where needs changes. Let's get GPU working properly (finally) in another PR.

[yebai]

Thanks @zuhengxu and @sunxd3!