Intermittent GC segfault during compilation with SCCNonlinearProblem + custom rule

[ChrisRackauckas-Claude]

Description

Enzyme.gradient through SCCNonlinearProblem (from NonlinearSolve.jl) intermittently segfaults during compilation on Julia 1.10. The crash occurs in gc_mark_obj8 during type inference triggered by Enzyme's thunk generator. When it doesn't crash, the gradient is correct.

Approximately 50% crash rate across fresh Julia processes.

MWE

using Enzyme, NonlinearSolve, SCCNonlinearSolve, SciMLSensitivity

function f1(du, u, p); du[1] = u[1]^2 - p[1]; end
function f2(du, u, p); du[1] = u[1] - p[1] * p[2]; end
explicitfun1!(p, sols) = nothing
function explicitfun2!(p, sols); p[1] = sols[1].u[1]; end

alg = SCCNonlinearSolve.SCCAlg(nlalg = NewtonRaphson())

function loss(p_val)
    p1 = copy(p_val); p2 = copy(p_val)
    sccprob = SciMLBase.SCCNonlinearProblem(
        (NonlinearProblem(f1, [1.0], p1), NonlinearProblem(f2, [1.0], p2)),
        (explicitfun1!, explicitfun2!))
    sum(solve(sccprob, alg).u)
end

# Crashes ~50% of runs with signal 11 (Segmentation fault) in gc_mark_obj8
g = Enzyme.gradient(Enzyme.set_runtime_activity(Enzyme.Reverse), loss, [4.0, 3.0])
println(g)  # ([1.0, 2.0],) when it works

Crash trace

[PID] signal (11.1): Segmentation fault
gc_mark_obj8 at julia/src/gc.c:1855
gc_mark_outrefs at julia/src/gc.c:2634
gc_mark_loop_serial_ at julia/src/gc.c:2690
...
typeinf at julia/compiler/typeinfer.jl:216
const_prop_call at julia/compiler/abstractinterpretation.jl:1211
...
thunk_generator at Enzyme/src/compiler.jl:7055

Notes

• Simple NonlinearProblem (no SCC) does NOT segfault
• The SCC code path involves a @generated function (iteratively_build_sols) with the NonlinearSolveBase Enzyme rule on solve_up
• Requires SciMLSensitivity loaded (provides the Enzyme rule via NonlinearSolveBase extension)
• set_runtime_activity(Reverse) is required; without it, EnzymeRuntimeActivityError
• When it doesn't crash, gradient is correct: [1.0, 2.0]

Versions

• Julia 1.10.11
• Enzyme 0.13.134
• NonlinearSolve 4.16.0
• NonlinearSolveBase 2.18.0
• SCCNonlinearSolve 1.12.0
• SciMLSensitivity 7.99.0 (dev branch with diff_tunables support)

🤖 Generated with Claude Code

Co-Authored-By: Chris Rackauckas accounts@chrisrackauckas.com

[wsmoses]

@ChrisRackauckas-Claude can you reduce the segfault to as small/non complex a problem as possible, ideally with no external dep?

[ChrisRackauckas-Claude]

Attempted to reduce to a SciML-free MWE but could not reproduce the segfault without the actual NonlinearSolve dispatch chain. A pure-Julia version with:

• Custom structs mimicking NonlinearSolution/NonlinearProblem (10+ type params)
• ForwardDiff Jacobian computation inside the solver
• Custom Enzyme augmented_primal/reverse rules
• Same SCC pattern (explicitfun mutation + two solver calls)

...runs 10/10 times without crashing.

The segfault requires the specific interaction between Enzyme's thunk generator, Julia's type inference, and the NonlinearSolve/NonlinearSolveBase/SCCNonlinearSolve dispatch chain (including @generated functions, runtime_generic_augfwd, etc.). The crash occurs in gc_mark_obj8 during const_prop_call → typeinf triggered by Enzyme's compiler.

🤖 Generated with Claude Code

Co-Authored-By: Chris Rackauckas accounts@chrisrackauckas.com

[ChrisRackauckas-Claude]

Fully inlined MWE (no SciML dependencies)

Only depends on Enzyme + LinearAlgebra. Preserves the three structural elements: @generated function with @nexprs/@ntuple, custom EnzymeRules on a function called within it, and mutable parameter arrays mutated between solves.

using Enzyme, LinearAlgebra

# ── Minimal solution type ──────────────────────────────────────────
mutable struct SimpleSol{T}
    u::Vector{T}
end

# ── Newton solver (replaces NonlinearSolve.jl) ────────────────────
function newton_solve(f!::F, u0::Vector{T}, p::Vector{T}) where {F, T}
    u = copy(u0)
    du = similar(u)
    n = length(u)
    J = Matrix{T}(undef, n, n)
    du_pert = similar(du)
    ε = T(1e-8)
    for _ in 1:100
        f!(du, u, p)
        maximum(abs, du) < 1e-10 && break
        for j in 1:n
            old = u[j]
            u[j] = old + ε
            f!(du_pert, u, p)
            J[:, j] .= (du_pert .- du) ./ ε
            u[j] = old
        end
        u .-= J \ du
    end
    return SimpleSol(u)
end

# ── Enzyme custom rule (replaces NonlinearSolveBase EnzymeExt) ────
function Enzyme.EnzymeRules.augmented_primal(
        config, func::Enzyme.Const{typeof(newton_solve)},
        ::Type{RT}, f!, u0, p; kwargs...) where {RT}
    sol = newton_solve(f!.val, u0.val, p.val)
    dsol = Enzyme.make_zero(sol)
    tape = (dsol, f!.val, copy(sol.u), copy(p.val))
    return Enzyme.EnzymeRules.AugmentedReturn(sol, dsol, tape)
end

function Enzyme.EnzymeRules.reverse(
        config, func::Enzyme.Const{typeof(newton_solve)},
        ::Type{RT}, tape, f!, u0, p; kwargs...) where {RT}
    dsol, fval, u_sol, p_val = tape
    ū = dsol.u
    n, np = length(u_sol), length(p_val)
    ε = 1e-8

    du0 = zeros(n); fval(du0, u_sol, p_val)
    du_pert = zeros(n)

    dFdu = zeros(n, n)
    for j in 1:n
        u_p = copy(u_sol); u_p[j] += ε
        fval(du_pert, u_p, p_val)
        dFdu[:, j] .= (du_pert .- du0) ./ ε
    end

    dFdp = zeros(n, np)
    for j in 1:np
        pp = copy(p_val); pp[j] += ε
        fval(du_pert, u_sol, pp)
        dFdp[:, j] .= (du_pert .- du0) ./ ε
    end

    λ = dFdu' \ ū
    dp̄ = -(dFdp' * λ)

    if !(p isa Enzyme.Const)
        if p isa Enzyme.MixedDuplicated
            p.dval[] .+= dp̄
        else
            p.dval .+= dp̄
        end
    end

    Enzyme.make_zero!(dsol.u)
    return (nothing, nothing, nothing)
end

# ── @generated SCC loop (replaces SCCNonlinearSolve.jl) ──────────
function solve_single_scc(f!::F, u0, p, explicitfun!::E, sols) where {F, E}
    explicitfun!(p, sols)
    return newton_solve(f!, u0, p)
end

@generated function iteratively_build_sols(
        fs::Tuple, u0s::Tuple, ps::Tuple, efuns::Tuple, ::Val{N}) where {N}
    quote
        Base.Cartesian.@nexprs $N i -> begin
            sol_i = solve_single_scc(
                fs[i], u0s[i], ps[i], efuns[i],
                Base.Cartesian.@ntuple((i - 1), j -> sol_j))
        end
        return Base.Cartesian.@ntuple $N i -> sol_i
    end
end

# ── Problem functions ─────────────────────────────────────────────
f1!(du, u, p) = (du[1] = u[1]^2 - p[1]; nothing)
f2!(du, u, p) = (du[1] = u[1] - p[1] * p[2]; nothing)
explicitfun1!(p, sols) = nothing
explicitfun2!(p, sols) = (p[1] = sols[1].u[1]; nothing)

# ── Loss & differentiate ─────────────────────────────────────────
function loss(p_val)
    p1 = copy(p_val)
    p2 = copy(p_val)
    sols = iteratively_build_sols(
        (f1!, f2!), ([1.0], [1.0]), (p1, p2),
        (explicitfun1!, explicitfun2!), Val(2))
    return sum(sols[1].u) + sum(sols[2].u)
end

g = Enzyme.gradient(Enzyme.set_runtime_activity(Enzyme.Reverse), loss, [4.0, 3.0])
println(g)  # Expected: ([1.0, 2.0],)

Produces ([1.0000000222044605, 2.0000000888178424],) — correct within finite-difference precision.

🤖 Generated with Claude Code

Co-Authored-By: Chris Rackauckas accounts@chrisrackauckas.com

[ChrisRackauckas-Claude]

Reduced MWE (~55 lines, only Enzyme dependency)

Stripped away Newton solver, LinearAlgebra, implicit function theorem, solution structs with vectors, multi-argument rules, and the SCC coupling logic. The core pattern that remains:

1. @generated function with Base.Cartesian.@nexprs/@ntuple unrolling
2. Custom EnzymeRules.augmented_primal/reverse on a function called within it
3. set_runtime_activity(Reverse) mode

using Enzyme

mutable struct Sol
    x::Float64
end

function mysolve(p::Vector{Float64})
    return Sol(p[1])
end

function Enzyme.EnzymeRules.augmented_primal(
        config, func::Enzyme.Const{typeof(mysolve)},
        ::Type{RT}, p; kwargs...) where {RT}
    sol = mysolve(p.val)
    dsol = Enzyme.make_zero(sol)
    return Enzyme.EnzymeRules.AugmentedReturn(sol, dsol, (dsol,))
end

function Enzyme.EnzymeRules.reverse(
        config, func::Enzyme.Const{typeof(mysolve)},
        ::Type{RT}, tape, p; kwargs...) where {RT}
    dsol = tape[1]
    if !(p isa Enzyme.Const)
        if p isa Enzyme.MixedDuplicated
            p.dval[][1] += dsol.x
        else
            p.dval[1] += dsol.x
        end
    end
    dsol.x = 0.0
    return (nothing,)
end

@generated function unrolled_loop(ps::NTuple{N, Vector{Float64}}, ::Val{N}) where {N}
    quote
        Base.Cartesian.@nexprs $N i -> begin
            sol_i = mysolve(ps[i])
        end
        return Base.Cartesian.@ntuple $N i -> sol_i
    end
end

function loss(p_val)
    p1 = copy(p_val)
    p2 = copy(p_val)
    sols = unrolled_loop((p1, p2), Val(2))
    return sols[1].x + sols[2].x
end

g = Enzyme.gradient(Enzyme.set_runtime_activity(Enzyme.Reverse), loss, [4.0, 3.0])
println(g)  # ([2.0, 0.0],)

Reduction notes

Tested removing each component individually on Julia 1.10.11 + Enzyme 0.13.134:

Variant	Result
Full reduced (above)	✅ ([2.0, 0.0],)
Manual unroll (no @generated)	✅ works
No custom rule (@generated only)	✅ works
N=1 (single component)	✅ works
Immutable struct instead of mutable struct	❌ AugmentedRuleReturnError (Active vs Duplicated)
No set_runtime_activity	❌ AugmentedRuleReturnError (primal not requested)

The reduced MWE does not segfault on current Enzyme — the original crash may require the full SciML type complexity to trigger the GC bug during type inference. The @generated + rule combination is structurally correct but the type pressure from NonlinearSolution, SCCNonlinearProblem, etc. may be what actually overwhelms gc_mark_obj8.

🤖 Generated with Claude Code

Co-Authored-By: Chris Rackauckas accounts@chrisrackauckas.com

[wsmoses]

@ChrisRackauckas-Claude the AugmentedRuleReturnError is an error in your rule, which the message states

[ChrisRackauckas-Claude]

Correction to reduction notes above — the immutable struct and no-set_runtime_activity failures were bugs in my rule definition (not checking needs_primal/needs_shadow config, and missing Active return dispatch). After fixing the rules to properly handle all return annotations, all variants pass:

Variant	Result
Full reduced	✅ ([2.0, 0.0],)
Manual unroll (no @generated)	✅ works
No custom rule (@generated only)	✅ works
N=1 (single component)	✅ works
Immutable struct (Active return)	✅ works
No set_runtime_activity	✅ works

The reduced MWE does not segfault on current Enzyme 0.13.134 / Julia 1.10.11. The original crash likely requires the full SciML type complexity (NonlinearSolution, SCCNonlinearProblem, etc.) to generate enough type inference pressure to trigger the gc_mark_obj8 bug.

🤖 Generated with Claude Code

Co-Authored-By: Chris Rackauckas accounts@chrisrackauckas.com

[ChrisRackauckas-Claude]

True minimal MWE — 4 lines

After progressive reduction, the SCC machinery, @generated functions, and SciMLSensitivity are all unnecessary. The crash reduces to just Enzyme.gradient through a single NonlinearSolve.solve:

using Enzyme, NonlinearSolve

f!(du, u, p) = (du[1] = u[1]^2 - p[1]; nothing)
loss(p) = solve(NonlinearProblem(f!, [1.0], copy(p)), NewtonRaphson()).u[1]

Enzyme.gradient(Enzyme.set_runtime_activity(Enzyme.Reverse), loss, [4.0])

Crashes deterministically (not intermittent) on Julia 1.12.4 with:

Illegal inttoptr
  %13 = inttoptr i64 %.unpack4 to ptr addrspace(10), !dbg !88
signal 6 (-6): Aborted

The crash is in LLVM's GC invariant verifier during Enzyme's post_optimize! → _thunk → thunk_generator path. Enzyme generates IR where a struct field is loaded as i64 and cast to a GC-tracked pointer (addrspace(10)), which Julia's verifier rejects.

Reduction notes

What was removed	Still crashes?
SCCNonlinearSolve (no SCC loop)	✅ crashes
SciMLSensitivity	✅ crashes
set_runtime_activity	✅ crashes
Second solve call	✅ crashes
Everything except construct + field read (no solve)	❌ works fine
Calling solve_up directly (bypass dispatch chain)	❌ different error (runtime, not LLVM)

The crash requires the full solve() dispatch chain (CommonSolve.solve → NonlinearSolveBase.solve → solve_up). Calling solve_up directly doesn't trigger it — the issue is in how Enzyme traces/compiles through the multiple dispatch layers with the deeply-parameterized NonlinearProblem type (8 params wrapping NonlinearFunction with 19 params).

Versions

• Julia 1.12.4
• Enzyme 0.13.134
• NonlinearSolve 4.16.0

Note: the original issue reports gc_mark_obj8 segfault on Julia 1.10 — this may be a different manifestation of the same underlying Enzyme IR generation bug across Julia versions.

🤖 Generated with Claude Code

Co-Authored-By: Chris Rackauckas accounts@chrisrackauckas.com

[ChrisRackauckas-Claude]

Root cause analysis — two bugs

Bug 1: Rule missing DuplicatedNoNeed (NonlinearSolveBase)

The Enzyme rule in NonlinearSolveBaseEnzymeExt.jl only matches:

::Union{Type{Duplicated{RT}}, Type{MixedDuplicated{RT}}}

but Enzyme passes Type{DuplicatedNoNeed{Any}} when the primal isn't needed. This causes a MethodError when calling solve_up directly, and Illegal inttoptr when going through the solve() dispatch chain (Enzyme falls back to native differentiation).

Bug 2: GC segfault in adjoint execution (Enzyme/Julia)

After patching the rule to accept all annotation types (RT <: Enzyme.Annotation + needs_primal/needs_shadow + augmented_rule_return_type), the rule matches correctly but:

• Direct solve_up call: gc_mark_obj8 segfault — the exact crash from the original issue
• Through solve() dispatch: Illegal inttoptr in LLVM GC invariant verifier — likely a separate Enzyme IR generation bug in the dispatch chain

Minimal MWE (4 lines)

using Enzyme, NonlinearSolve
f!(du, u, p) = (du[1] = u[1]^2 - p[1]; nothing)
loss(p) = solve(NonlinearProblem(f!, [1.0], copy(p)), NewtonRaphson()).u[1]
Enzyme.gradient(Enzyme.set_runtime_activity(Enzyme.Reverse), loss, [4.0])

No SCC, no @generated, no SciMLSensitivity needed.

Versions

• Julia 1.12.4, Enzyme 0.13.134, NonlinearSolve 4.16.0

🤖 Generated with Claude Code

Co-Authored-By: Chris Rackauckas accounts@chrisrackauckas.com

[ChrisRackauckas-Claude]

Update: rule fix landed, Enzyme bug remains

The DuplicatedNoNeed rule-matching issue in NonlinearSolveBase was fixed in SciML/NonlinearSolve.jl#885 and released as NonlinearSolveBase v2.19.0.

With the rule fix, two Enzyme bugs remain:

1. Illegal inttoptr through solve() dispatch chain

Illegal inttoptr
  %13 = inttoptr i64 %.unpack3 to ptr addrspace(10), !dbg !87
signal 6 (-6): Aborted

Enzyme generates invalid LLVM IR when differentiating through the solve() → solve_up dispatch chain. Crashes during compilation in post_optimize!, before the rule executes.

2. gc_mark_obj8 segfault when rule fires (with SciMLSensitivity)

signal 11 (1): Segmentation fault
gc_mark_obj8 at julia/src/gc-stock.c:1702

When calling solve_up directly (bypassing the dispatch chain), the rule matches and fires correctly, but the adjoint execution segfaults in GC during type inference. This is the original crash from the issue.

Minimal reproducer (still crashes on Julia 1.12.4 + Enzyme 0.13.134 + NonlinearSolveBase 2.19.0):

using Enzyme, NonlinearSolve
f!(du, u, p) = (du[1] = u[1]^2 - p[1]; nothing)
loss(p) = solve(NonlinearProblem(f!, [1.0], copy(p)), NewtonRaphson()).u[1]
Enzyme.gradient(Enzyme.set_runtime_activity(Enzyme.Reverse), loss, [4.0])

🤖 Generated with Claude Code

Co-Authored-By: Chris Rackauckas accounts@chrisrackauckas.com

[ChrisRackauckas]

I fixed the rule but there's still something here, and finding an MWE seems not possible.