You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The function `sample` is part of the AbstractMCMC interface. As explained in the [interface guide]({{<metausing-turing-interface>}}), building a sampling method that can be used by `sample` consists in overloading the structs and functions in `AbstractMCMC`. The interface guide also gives a standalone example of their implementation, [`AdvancedMH.jl`]().
38
+
The function `sample` is part of the AbstractMCMC interface. As explained in the interface guide, building a sampling method that can be used by `sample` consists in overloading the structs and functions in `AbstractMCMC`. The interface guide also gives a standalone example of their implementation, [`AdvancedMH.jl`]().
39
39
40
40
Turing sampling methods (most of which are written [here](https://github.com/TuringLang/Turing.jl/tree/main/src/mcmc)) also implement `AbstractMCMC`. Turing defines a particular architecture for `AbstractMCMC` implementations, that enables working with models defined by the `@model` macro, and uses DynamicPPL as a backend. The goal of this page is to describe this architecture, and how you would go about implementing your own sampling method in Turing, using Importance Sampling as an example. I don't go into all the details: for instance, I don't address selectors or parallelism.
Copy file name to clipboardExpand all lines: getting-started/index.qmd
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -93,4 +93,4 @@ A thorough introduction to the field is [*Pattern Recognition and Machine Learni
93
93
:::
94
94
95
95
The next page on [Turing's core functionality]({{<metausing-turing>}}) explains the basic features of the Turing language.
96
-
From there, you can either look at [worked examples of how different models are implemented in Turing]({{<metatutorials-intro>}}), or [specific tips and tricks that can help you get the most out of Turing]({{<metausing-turing-mode-estimation>}}).
96
+
From there, you can either look at [worked examples of how different models are implemented in Turing]({{<metatutorials-intro>}}), or [specific tips and tricks that can help you get the most out of Turing]({{<metausage-performance-tips>}}).
Copy file name to clipboardExpand all lines: tutorials/hidden-markov-models/index.qmd
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -103,7 +103,7 @@ In this case, we use HMC for `m` and `T`, representing the emission and transiti
103
103
The parameter `s` is not a continuous variable.
104
104
It is a vector of **integers**, and thus Hamiltonian methods like HMC and NUTS won't work correctly.
105
105
Gibbs allows us to apply the right tools to the best effect.
106
-
If you are a particularly advanced user interested in higher performance, you may benefit from setting up your Gibbs sampler to use [different automatic differentiation]({{<metausing-turing-autodiff>}}#compositional-sampling-with-differing-ad-modes) backends for each parameter space.
106
+
If you are a particularly advanced user interested in higher performance, you may benefit from setting up your Gibbs sampler to use [different automatic differentiation]({{<metausage-automatic-differentiation>}}#compositional-sampling-with-differing-ad-modes) backends for each parameter space.
Copy file name to clipboardExpand all lines: usage/automatic-differentiation/index.qmd
+83-44Lines changed: 83 additions & 44 deletions
Original file line number
Diff line number
Diff line change
@@ -12,68 +12,81 @@ using Pkg;
12
12
Pkg.instantiate();
13
13
```
14
14
15
-
## Switching AD Modes
15
+
## What is Automatic Differentiation?
16
16
17
-
Turing currently supports four automatic differentiation (AD) backends for sampling: [ForwardDiff](https://github.com/JuliaDiff/ForwardDiff.jl) for forward-mode AD; and [Mooncake](https://github.com/compintell/Mooncake.jl) and [ReverseDiff](https://github.com/JuliaDiff/ReverseDiff.jl) for reverse-mode AD.
18
-
`ForwardDiff` is automatically imported by Turing. To utilize `Mooncake` or `ReverseDiff` for AD, users must explicitly import them with `import Mooncake` or `import ReverseDiff`, alongside the usual `using Turing`.
17
+
Automatic differentiation (AD) is a technique used in Turing.jl to evaluate the gradient of a function at a given set of arguments.
18
+
In the context of Turing.jl, the function being differentiated is the log probability density of a model, and the arguments are the parameters of the model (i.e. the values of the random variables).
19
+
The gradient of the log probability density is used by various algorithms in Turing.jl, such as HMC (including NUTS), mode estimation (which uses gradient-based optimization), and variational inference.
19
20
20
-
As of Turing version v0.30, the global configuration flag for the AD backend has been removed in favour of [`AdTypes.jl`](https://github.com/SciML/ADTypes.jl), allowing users to specify the AD backend for individual samplers independently.
21
-
Users can pass the `adtype` keyword argument to the sampler constructor to select the desired AD backend, with the default being `AutoForwardDiff(; chunksize=0)`.
21
+
The Julia ecosystem has a number of AD libraries.
22
+
You can switch between these using the unified [ADTypes.jl](https://github.com/SciML/ADTypes.jl/) interface, which for a given AD backend, provides types such as `AutoBackend` (see [the documentation](https://docs.sciml.ai/ADTypes/stable/) for more details).
23
+
For example, to use the [Mooncake.jl](https://github.com/compintell/Mooncake.jl) package for AD, you can run the following:
22
24
23
-
For `ForwardDiff`, pass `adtype=AutoForwardDiff(; chunksize)` to the sampler constructor. A `chunksize` of `nothing` permits the chunk size to be automatically determined. For more information regarding the selection of `chunksize`, please refer to [related section of `ForwardDiff`'s documentation](https://juliadiff.org/ForwardDiff.jl/dev/user/advanced/#Configuring-Chunk-Size).
25
+
```{julia}
26
+
# Turing re-exports AutoForwardDiff, AutoReverseDiff, and AutoMooncake.
27
+
# Other ADTypes must be explicitly imported from ADTypes.jl or
28
+
# DifferentiationInterface.jl.
29
+
using Turing
30
+
setprogress!(false)
24
31
25
-
For `ReverseDiff`, pass `adtype=AutoReverseDiff()` to the sampler constructor. An additional keyword argument called `compile` can be provided to `AutoReverseDiff`. It specifies whether to pre-record the tape only once and reuse it later (`compile` is set to `false` by default, which means no pre-recording). This can substantially improve performance, but risks silently incorrect results if not used with care.
32
+
# Note that if you specify a custom AD backend, you must also import it.
33
+
import Mooncake
26
34
27
-
Pre-recorded tapes should only be used if you are absolutely certain that the sequence of operations performed in your code does not change between different executions of your model.
35
+
@model function f()
36
+
x ~ Normal()
37
+
# Rest of your model here
38
+
end
28
39
29
-
Thus, e.g., in the model definition and all implicitly and explicitly called functions in the model, all loops should be of fixed size, and `if`-statements should consistently execute the same branches.
30
-
For instance, `if`-statements with conditions that can be determined at compile time or conditions that depend only on fixed properties of the model, e.g. fixed data.
31
-
However, `if`-statements that depend on the model parameters can take different branches during sampling; hence, the compiled tape might be incorrect.
32
-
Thus you must not use compiled tapes when your model makes decisions based on the model parameters, and you should be careful if you compute functions of parameters that those functions do not have branching which might cause them to execute different code for different values of the parameter.
The previously used interface functions including `ADBackend`, `setadbackend`, `setsafe`, `setchunksize`, and `setrdcache` have been removed.
43
+
By default, if you do not specify a backend, Turing will default to [ForwardDiff.jl](https://github.com/JuliaDiff/ForwardDiff.jl).
44
+
In this case, you do not need to import ForwardDiff, as it is already a dependency of Turing.
35
45
36
-
For `Mooncake`, pass `adtype=AutoMooncake(; config=nothing)` to the sampler constructor.
46
+
## Choosing an AD Backend
37
47
38
-
## Compositional Sampling with Differing AD Modes
48
+
There are two aspects to choosing an AD backend: firstly, what backends are available; and secondly, which backend is best for your model.
39
49
40
-
Turing supports intermixed automatic differentiation methods for different variable spaces. The snippet below shows using `ForwardDiff` to sample the mean (`m`) parameter, and using `ReverseDiff` for the variance (`s`) parameter:
50
+
### Usable AD Backends
41
51
42
-
```{julia}
43
-
using Turing
44
-
using ReverseDiff
52
+
Turing.jl uses the functionality in [DifferentiationInterface.jl](https://github.com/JuliaDiff/DifferentiationInterface.jl) ('DI') to interface with AD libraries in a unified way.
53
+
In principle, any AD library that DI provides an interface for can be used with Turing; you should consult the [DI documentation](https://juliadiff.org/DifferentiationInterface.jl/DifferentiationInterface/stable/) for an up-to-date list of compatible AD libraries.
45
54
46
-
# Define a simple Normal model with unknown mean and variance.
47
-
@model function gdemo(x, y)
48
-
s² ~ InverseGamma(2, 3)
49
-
m ~ Normal(0, sqrt(s²))
50
-
x ~ Normal(m, sqrt(s²))
51
-
return y ~ Normal(m, sqrt(s²))
52
-
end
55
+
Note, however, that not all AD libraries in there are thoroughly tested on Turing models.
56
+
Thus, it is possible that some of them will either error (because they don't know how to differentiate through Turing's code), or maybe even silently give incorrect results (if you are very unlucky).
57
+
Turing is most extensively tested with **ForwardDiff.jl** (the default), **ReverseDiff.jl**, and **Mooncake.jl**.
58
+
We also run a smaller set of tests with Enzyme.jl.
53
59
54
-
# Sample using Gibbs and varying autodiff backends.
Before describing how to choose the best AD backend for your model, we should mention that we also publish a table of benchmarks for various models and AD backends in [the ADTests website](https://turinglang.org/ADTests/).
63
+
These models aim to capture a variety of different features of Turing.jl and Julia in general, so that you can see which AD backends may be compatible with your model.
64
+
Benchmarks are also included, although it should be noted that many of the models in ADTests are small and thus the timings may not be representative of larger, real-life models.
65
+
66
+
If you have suggestions for other models to include, please do let us know by [creating an issue on GitHub](https://github.com/TuringLang/ADTests/issues/new)!
65
67
66
-
Generally, reverse-mode AD, for instance `ReverseDiff`, is faster when sampling from variables of high dimensionality (greater than 20), while forward-mode AD, for instance `ForwardDiff`, is more efficient for lower-dimension variables. This functionality allows those who are performance sensitive to fine tune their automatic differentiation for their specific models.
68
+
### The Best AD Backend for Your Model
67
69
68
-
If the differentiation method is not specified in this way, Turing will default to using whatever the global AD backend is.
69
-
Currently, this defaults to `ForwardDiff`.
70
+
Given the number of possible backends, how do you choose the best one for your model?
70
71
71
-
The most reliable way to ensure you are using the fastest AD that works for your problem is to benchmark them using the functionality in DynamicPPL (see [the API documentation](https://turinglang.org/DynamicPPL.jl/stable/api/#AD-testing-and-benchmarking-utilities)):
72
+
A simple heuristic is to look at the number of parameters in your model.
73
+
The log density of the model, i.e. the function being differentiated, is a function that goes from $\mathbb{R}^n \to \mathbb{R}$, where $n$ is the number of parameters in your model.
74
+
For models with a small number of parameters (say up to ~ 20), forward-mode AD (e.g. ForwardDiff) is generally faster due to a smaller overhead.
75
+
On the other hand, for models with a large number of parameters, reverse-mode AD (e.g. ReverseDiff or Mooncake) is generally faster as it computes the gradients with respect to all parameters in a single pass.
76
+
77
+
The most exact way to ensure you are using the fastest AD that works for your problem is to benchmark them using the functionality in DynamicPPL (see [the API documentation](https://turinglang.org/DynamicPPL.jl/stable/api/#AD-testing-and-benchmarking-utilities)):
72
78
73
79
```{julia}
80
+
using ADTypes
74
81
using DynamicPPL.TestUtils.AD: run_ad, ADResult
75
82
using ForwardDiff, ReverseDiff
76
83
84
+
@model function gdemo(x, y)
85
+
s² ~ InverseGamma(2, 3)
86
+
m ~ Normal(0, sqrt(s²))
87
+
x ~ Normal(m, sqrt(s²))
88
+
return y ~ Normal(m, sqrt(s²))
89
+
end
77
90
model = gdemo(1.5, 2)
78
91
79
92
for adtype in [AutoForwardDiff(), AutoReverseDiff()]
@@ -84,6 +97,32 @@ end
84
97
85
98
In this specific instance, ForwardDiff is clearly faster (due to the small size of the model).
86
99
87
-
We also have a table of benchmarks for various models and AD backends in [the ADTests website](https://turinglang.org/ADTests/).
88
-
These models aim to capture a variety of different Turing.jl features.
89
-
If you have suggestions for things to include, please do let us know by [creating an issue on GitHub](https://github.com/TuringLang/ADTests/issues/new)!
100
+
::: {.callout-note}
101
+
## A note about ReverseDiff's `compile` argument
102
+
103
+
The additional keyword argument `compile=true` for `AutoReverseDiff` specifies whether to pre-record the tape only once and reuse it later.
104
+
By default, this is set to `false`, which means no pre-recording.
105
+
Setting `compile=true` can substantially improve performance, but risks silently incorrect results if not used with care.
106
+
Pre-recorded tapes should only be used if you are absolutely certain that the sequence of operations performed in your code does not change between different executions of your model.
107
+
:::
108
+
109
+
## Compositional Sampling with Differing AD Modes
110
+
111
+
When using Gibbs sampling, Turing also supports mixed automatic differentiation methods for different variable spaces.
112
+
The following snippet shows how one can use `ForwardDiff` to sample the mean (`m`) parameter, and `ReverseDiff` for the variance (`s`) parameter:
113
+
114
+
```{julia}
115
+
using Turing
116
+
using ReverseDiff
117
+
118
+
# Sample using Gibbs and varying autodiff backends.
More generally, you may also consider reparameterising the model to avoid such issues.
105
+
106
+
## ForwardDiff type parameters
107
+
108
+
> MethodError: no method matching Float64(::ForwardDiff.Dual{... The type `Float64` exists, but no method is defined for this combination of argument types when trying to construct it.
When the log probability density of the model is calculated, `a` is sampled from a normal distribution and is thus a Float64; however, when ForwardDiff calculates the gradient of the log density, `a` is a `ForwardDiff.Dual` object.
126
+
However, `x` is _always_ a `Vector{Float64}`, and the call `x[1] = a` attempts to insert a `Dual` object into a `Vector{Float64}`, which is not allowed.
127
+
128
+
::: {.callout-note}
129
+
In more depth: the basic premise of ForwardDiff is that functions have to accept `Real` parameters instead of `Float64` (since `Dual` is a subtype of `Real`).
130
+
Here, the line `x[1] = a` is equivalent to `setindex!(x, a, 1)`, and although the method `setindex!(::Vector{Float64}, ::Real, ...)` does exist, it attempts to convert the `Real` into a `Float64`, which is where it fails.
131
+
:::
132
+
133
+
There are two ways around this.
134
+
135
+
Firstly, you could broaden the type of the container:
This is generally unfavourable because the `Vector{Real}` type contains an abstract type parameter.
148
+
As a result, memory allocation is less efficient (because the compiler does not know the size of each vector's elements).
149
+
Furthermore, the compiler cannot infer the type of `x[1]`, which can lead to type stability issues (to see this in action, run `x = Real[0.0, 1.0]; @code_warntype x[1]` in the Julia REPL).
150
+
151
+
A better solution is to pass a type as a parameter to the model:
152
+
153
+
```{julia}
154
+
@model function forwarddiff_working2(::Type{T}=Float64) where T
0 commit comments