NEWS.md

Flux Release Notes

v0.12.0

• The Dense layer now supports inputs with multiple batch dimensions.
• Dense and Conv layers no longer perform implicit type conversion.
• Excise datasets in favour of other providers in the julia ecosystem.
• Added option to set bias to false to eliminating bias from being trained.
• Add CTC loss function to Losses module
• Removed kwarg only constructors for convolutional layers.
• Add sparse initialization as described in Deep learning via Hessian-free optimization.
• Moved GPU CI to use buildkite instead of GitLab
• New Parallel layer adds inception module-like building blocks.
• Other new features and bug fixes (see GitHub releases page)

v0.11.2

• Adds the AdaBelief optimiser.
• Other new features and bug fixes (see GitHub releases page)

v0.11

• Moved CUDA compatibility to use CUDA.jl instead of CuArrays.jl
• Add kaiming initialization methods: kaiming_uniform and kaiming_normal
• Use DataLoader with NamedTuples, so that tensors can be accessed by name.
• Error if Dense layers weights and biases are not arrays.
• Add Adaptive Pooling in Flux layers.
• Change to DataLoader's constructor
• Uniform loss interface
• Loss functions now live in the Flux.Losses module
• Optimistic ADAM (OADAM) optimizer for adversarial training.
• Add option for same padding to conv and pooling layers by setting pad=SamePad().
• Added option to set bias to Flux.Zeros to eliminating bias from being trained.
• Added GlobalMaxPool and GlobalMeanPool layers for performing global pooling operations.
• Added ClipValue and ClipNorm in this pr to Flux.Optimise to provide a cleaner API for gradient clipping.
• Added new kwarg-only constructors for the various convolutional layers.
• Documented the convolutional layer constructors accepting weight and bias keyword arguments to supply custom arrays for those fields.
• Testing suite improvements now test for gradients of all layers along with GPU support.
• Functors have now moved to Functors.jl to allow for their use outside of Flux.
• Added helper functions Flux.convfilter and Flux.depthwiseconvfilter to construct weight arrays for convolutions outside of layer constructors so as to not have to depend on the default layers for custom implementations.
• dropout function now has a mandatory active keyword argument. The Dropout struct *whose behavior is left unchanged) is the recommended choice for common usage.
• and many more fixes and additions...

v0.10.1 - v0.10.4

See GitHub's releases.

v0.10.0

• The default AD engine has switched from Tracker to Zygote.jl
  • The dependency on Tracker.jl has been removed.
  • This means Flux now does not depend on using a specialised TrackedArray type, and can be used with normal Array implementations directly.
  • Tracker compatibility is maintained in most common cases, but Zygote will be the preferred AD backend for Flux from now on.
• The CUDNN wrappers have been moved from Flux into CuArrays, to allow for better supporting the CUDA backend, and improve user experience, not to mention making Flux lean.
• *crossentropy functions now work as expected with CuArrays. PR for binarycrossentropy.
• Added clearer docs around training and the Optimiser interface.
• Layer initialisations have been improved with a clearer API on how to extend it for other purposes.
• Better messaging around CUDA availability, with hooks to initialize the GPU as default where possible.
• @treelike has been formalised as a functor, with an effective deprecation.
• testmode! is deprecated in favour of istraining

v0.9.0

• Depthwise convolutional layer API changes from in => mult channel specification to in => out channel specification, and deprecates implicit out constructor.
• New SkipConnection, which can be used to train residual neural network architectures.
• New RADAM optimiser.

v0.8.0

• Dropout now has a dims argument for specifying the unbroadcast dimensions.
• New ConvTranspose layer.
• New Maxout layer
• Datasets are now hash verified on download to avoid corruption.
• We now zero the initial state for RNNs.
• Normalisation can now work on arbitrary dims.
• Many docs and bugfixes thanks to @KristofferC and others.
• NamedTuples now work like Tuples when doing mapleaves.
• New "performance tips" section of the docs.
• The training loop is now more readable and better shows how to use the lower-level APIs.
• New AlphaDropout.
• Data.Iris makes Fisher's Iris dataset available with Iris.labels and Iris.features.
• New InstanceNorm, as popularized by Instance Normalization: The Missing Ingredient for Fast Stylization.
• New GroupNorm, as described in Group Normalization.
• New CrossCor.

AD Changes:

• det, logdet and logabsdet now have adjoints.
• Support for PermuteDimsArray.
• Flux.Tracker is now its own package, in preparation for replacing it with Zygote.

v0.7.0

Despite the heroic efforts of scholars and archeologists, pre-0.7 history is lost to the sands of time.