3D FFT `ConcurrencyViolationError`

[islent]

My MWE:

using Distributed
addprocs(8)
@everywhere using Dagger

using FFTW
N = 100
A = rand(ComplexF64,N,N,N);
DA = DArray(A);
fft(DA)

Output:

julia> using Distributed

julia> addprocs(8)
8-element Vector{Int64}:
 2
 3
 4
 5
 6
 7
 8
 9

julia> @everywhere using Dagger

julia>

julia> using FFTW

julia> N = 100
100

julia> A = rand(ComplexF64,N,N,N);

julia> DA = DArray(A);

julia> fft(DA)
ERROR: DTaskFailedException:
  Root Exception Type: ConcurrencyViolationError
  Root Exception:
On worker 6:
On worker 6:
ConcurrencyViolationError("lock must be held")
Stacktrace:
  [1] concurrency_violation
    @ .\condition.jl:8
  [2] assert_havelock
    @ .\condition.jl:25 [inlined]
  [3] assert_havelock
    @ .\condition.jl:48 [inlined]
  [4] assert_havelock
    @ .\condition.jl:72 [inlined]
  [5] _wait2
    @ .\condition.jl:83
  [6] #wait#733
    @ .\condition.jl:127
  [7] wait
    @ .\condition.jl:125 [inlined]
  [8] wait_for_conn
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\cluster.jl:195
  [9] check_worker_state
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\cluster.jl:170
 [10] send_msg_
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\messages.jl:172
 [11] send_msg
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\messages.jl:122 [inlined]
 [12] #remotecall_fetch#159
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\remotecall.jl:460
 [13] remotecall_fetch
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\remotecall.jl:454
 [14] remotecall_fetch
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\remotecall.jl:492 [inlined]
 [15] #259
    @ E:\.julia\packages\Dagger\Jl5aj\src\memory-spaces.jl:44
 [16] #invokelatest#2
    @ .\essentials.jl:1055 [inlined]
 [17] invokelatest
    @ .\essentials.jl:1052 [inlined]
 [18] #153
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\remotecall.jl:425 [inlined]
 [19] run_work_thunk
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\process_messages.jl:70
 [20] run_work_thunk
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\process_messages.jl:79
 [21] #100
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\process_messages.jl:88
Stacktrace:
  [1] wait_ref
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\remotecall.jl:573 [inlined]
  [2] call_on_owner
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\remotecall.jl:563 [inlined]
  [3] wait
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\remotecall.jl:586 [inlined]
  [4] #remotecall_wait#163
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\remotecall.jl:495
  [5] remotecall_wait
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\remotecall.jl:495 [inlined]
  [6] remotecall_wait
    @ D:\Programs\julia-1.11.6\share\julia\stdlib\v1.11\Distributed\src\remotecall.jl:519 [inlined]
  [7] move!
    @ E:\.julia\packages\Dagger\Jl5aj\src\memory-spaces.jl:40
  [8] #invokelatest#2
    @ .\essentials.jl:1055 [inlined]
  [9] invokelatest
    @ .\essentials.jl:1052 [inlined]
 [10] #63
    @ E:\.julia\packages\Dagger\Jl5aj\src\threadproc.jl:22
Stacktrace:
  [1] wait
    @ .\task.jl:370
  [2] fetch
    @ .\task.jl:390 [inlined]
  [3] #execute!#62
    @ E:\.julia\packages\Dagger\Jl5aj\src\threadproc.jl:28
  [4] execute!
    @ E:\.julia\packages\Dagger\Jl5aj\src\threadproc.jl:13
  [5] #200
    @ E:\.julia\packages\Dagger\Jl5aj\src\sch\Sch.jl:1523 [inlined]
  [6] #112
    @ E:\.julia\packages\Dagger\Jl5aj\src\options.jl:208 [inlined]
  [7] with
    @ .\scopedvalues.jl:269
  [8] with_options
    @ E:\.julia\packages\Dagger\Jl5aj\src\options.jl:207
  [9] macro expansion
    @ E:\.julia\packages\Dagger\Jl5aj\src\sch\Sch.jl:1521 [inlined]
 [10] do_task
    @ E:\.julia\packages\Dagger\Jl5aj\src\utils\reuse.jl:12
 [11] DoTaskSpec
    @ E:\.julia\packages\Dagger\Jl5aj\src\sch\Sch.jl:1183
 [12] #invokelatest#2
    @ .\essentials.jl:1055 [inlined]
 [13] invokelatest
    @ .\essentials.jl:1052 [inlined]
 [14] reusable_task_loop
    @ E:\.julia\packages\Dagger\Jl5aj\src\utils\reuse.jl:598
 [15] #43
    @ E:\.julia\packages\Dagger\Jl5aj\src\utils\reuse.jl:560
  This Task:  DTask(id=30179, move!(5 inputs...))
Stacktrace:
  [1] fetch(t::Dagger.ThunkFuture; proc::OSProc, raw::Bool)
    @ Dagger E:\.julia\packages\Dagger\Jl5aj\src\dtask.jl:19
  [2] fetch
    @ E:\.julia\packages\Dagger\Jl5aj\src\dtask.jl:14 [inlined]
  [3] #fetch#129
    @ E:\.julia\packages\Dagger\Jl5aj\src\dtask.jl:72 [inlined]
  [4] fetch
    @ E:\.julia\packages\Dagger\Jl5aj\src\dtask.jl:68 [inlined]
  [5] wait_all(f::Function; check_errors::Bool)
    @ Dagger E:\.julia\packages\Dagger\Jl5aj\src\queue.jl:97
  [6] wait_all
    @ E:\.julia\packages\Dagger\Jl5aj\src\queue.jl:92 [inlined]
  [7] #spawn_datadeps#354
    @ E:\.julia\packages\Dagger\Jl5aj\src\datadeps.jl:1038 [inlined]
  [8] spawn_datadeps
    @ E:\.julia\packages\Dagger\Jl5aj\src\datadeps.jl:1030 [inlined]
  [9] copyto!(B::DArray{ComplexF64, 3, Blocks{3}, typeof(cat)}, A::DArray{ComplexF64, 3, Blocks{3}, typeof(cat)})
    @ Dagger E:\.julia\packages\Dagger\Jl5aj\src\array\copy.jl:36
 [10] _fft!(output::DArray{…}, input::DArray{…}, dims::Tuple{…}; decomp::AbstractFFTsExt.Pencil)
    @ AbstractFFTsExt E:\.julia\packages\Dagger\Jl5aj\ext\AbstractFFTsExt.jl:127
 [11] fft(DA::DArray{ComplexF64, 3, Blocks{3}, typeof(cat)}, dims::Tuple{Int64, Int64, Int64}; decomp::AbstractFFTsExt.Pencil)
    @ AbstractFFTsExt E:\.julia\packages\Dagger\Jl5aj\ext\AbstractFFTsExt.jl:62
 [12] fft
    @ E:\.julia\packages\Dagger\Jl5aj\ext\AbstractFFTsExt.jl:59 [inlined]
 [13] fft(DA::DArray{ComplexF64, 3, Blocks{3}, typeof(cat)})
    @ AbstractFFTsExt E:\.julia\packages\Dagger\Jl5aj\ext\AbstractFFTsExt.jl:59
 [14] top-level scope
    @ REPL[8]:1
Some type information was truncated. Use `show(err)` to see complete types.

My coding environments:

julia> versioninfo()
Julia Version 1.11.6
Commit 9615af0f26 (2025-07-09 12:58 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 16 × Intel(R) Xeon(R) W-10885M CPU @ 2.40GHz
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, skylake)
Threads: 8 default, 0 interactive, 4 GC (on 16 virtual cores)
Environment:
  JULIA_DEPOT_PATH = E:\.julia
  JULIA_NUM_THREADS = 8

(@v1.11) pkg> st Dagger
Status `E:\.julia\environments\v1.11\Project.toml`
  [d58978e5] Dagger v0.19.0

(@v1.11) pkg> st FFTW
Status `E:\.julia\environments\v1.11\Project.toml`
  [7a1cc6ca] FFTW v1.9.0

[jpsamaroo]

It seems like you're running Julia with multiple threads while also using Distributed.jl - this is a broken configuration for Distributed, and causes issues like you're seeing here. You can either run without Julia threads, run without workers (just using threads, which is the most performant option usually), or switch to DistributedNext.jl, which has proper multithreading support: Dagger.set_distributed_package!("DistributedNext").

[islent]

Thanks!

I find setting FFTW.set_num_threads(1) on all workers can avoid the error.

But yes, DistributedNext is better and faster than Distributed.