Skip to content

run_ad should test gradients using elementwise rtol, instead of atol #963

@penelopeysm

Description

@penelopeysm

See: TuringLang/ADTests#33 for motivation behind this change.

using DynamicPPL
using DynamicPPL.TestUtils.AD: run_ad
using Distributions
using ADTypes: AutoForwardDiff, AutoMooncake
import ForwardDiff
import Mooncake
using Random: Xoshiro

@model function f(y)
    m ~ Cauchy(0, 10)
    s ~ truncated(Cauchy(0, 5); lower=0)
    h ~ Normal(m, s)
    y ~ Normal(0, exp(h / 2))
end
model = f(1505.62)
xs = DynamicPPL.link(VarInfo(Xoshiro(468), model), model)[:]
julia> fd_res = run_ad(model, AutoForwardDiff(); test=false, params=xs).grad_actual
[ Info: Running AD on f with AutoForwardDiff()
       params : [-80.49419773440104, -1.7776148807021088, -80.65985398063759]
       actual : (-1.2148712356518068e41, [-5.772813893107857, 0.9580727253507426, 1.2148712356518068e41])
3-element Vector{Float64}:
 -5.772813893107857
  0.9580727253507426
  1.2148712356518068e41

julia> mc_res = run_ad(model, AutoMooncake(); test=false, params=xs).grad_actual
[ Info: Running AD on f with AutoMooncake{Nothing}(nothing)
       params : [-80.49419773440104, -1.7776148807021088, -80.65985398063759]
       actual : (-1.2148712356518068e41, [-5.772813893107856, 0.9580727253507425, 1.2148712356518066e41])
3-element Vector{Float64}:
 -5.772813893107856
  0.9580727253507425
  1.2148712356518066e41

julia> fd_res .- mc_res
3-element Vector{Float64}:
 -8.881784197001252e-16
  1.1102230246251565e-16
  1.9342813113834067e25

based on the current check with atol, Mooncake (and indeed all other backends) fail because they have an absolute error on the order of 1e25, when actually the relative error is on the order of 1e-16.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions