Quarto render stalls when using Distributed

[alecloudenback]

Here's a small example Quarto doc which has a Distributed example. Running this code by itself (Julia command line or REPL) produces the desired output. The Quarto document/website also renders if using the Jupyter engine.

---
engine: Julia
---
# Introduction

This is a book created from markdown and executable code.



```{Julia}
using Distributed
Distributed.addprocs(4)
```

When running with the first three lines deleted (i.e. the Jupyter engine):

dist_mwe quarto preview
Preparing to preview

Watching files for changes
Browse at http://localhost:3374/
GET: /

Starting julia-1.11 kernel...Done

Executing 'intro.quarto_ipynb'
  Cell 1/1: ''...Done

pandoc 
  to: html
  output-file: intro.html
  standalone: true
  toc: true
  number-sections: true
  title-prefix: dist_mwe
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png
  number-offset:
    - 1
  variables: {}
  
metadata
  crossref:
    chapters: true
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  bibliography:
    - references.bib
  theme:
    - cosmo
    - brand
  hide-description: true
  title: "[1]{.chapter-number}\_ [Introduction]{.chapter-title}"
  
GET: /intro.html

When running with the Julia engine in the front matter:

➜  dist_mwe quarto preview
Preparing to preview
[1/1] intro.qmd
Starting julia control server process. This might take a while...
Julia server process started.
Running [1/1] at line 10:  using Distributed

Julia Version 1.11.4
Commit 8561cc3d68d (2025-03-10 11:36 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  CPU: 14 × Apple M4 Max
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, apple-m1)
Threads: 1 default, 0 interactive, 1 GC (on 10 virtual cores)
Environment:
  JULIA_PROJECT = @.
  JULIA_LOAD_PATH = @:@stdlib

Also tested on Julia 1.10.9 with same effect (doesn't work with QuartoNotebookRunner, but same code will run with the IJulia kernel via Quarto, as well as in the REPL).

[MichaelHatherly]

Thanks for the report @alecloudenback. What is the OS, and Julia versions (is it a 1.11.*?) that you're able to replicate this on?

Do you know what particular expression it is hanging on? Would you be able to split that let up into individual code blocks so we can more easily see which one it stops on. I assume it would be the run_distributed_simulation() at the very end, but would be good to have confirmation on that.

[alecloudenback]

I updated the OP to narrow the MWE to the addprocs line. I've only tested on Mac (full version info also added to the OP).

[alecloudenback]

After updating dependencies, including Julia to 1.11.5, Quarto to 1.7.31, and QuartoNotebookRunner to v0.17.0 I no longer have this issue in my real repository or the MWE.

[MichaelHatherly]

Ah that's good to hear @alecloudenback. If it happens again just go ahead and reopen this one.

[alecloudenback]

I'm definitely still experiencing this with Julia 1.12.5 and Quarto 1.19.16.

I tried to go back to the version combo below, but I do get the issue still and am not able to recreate the working environment (ie still broken when installing 1.11.5 with juliaup and Quarto 1.7.31 and manually installing QNR to 0.17.0)

After updating dependencies, including Julia to 1.11.5, Quarto to 1.7.31, and QuartoNotebookRunner to v0.17.0 I no longer have this issue in my real repository or the MWE.

[MichaelHatherly]

Hi @alecloudenback, there's been some refactoring of the worker system that's currently on main, would you be able to test with current main and see whether it's still happening there?

https://github.com/PumasAI/QuartoNotebookRunner.jl/blob/main/revise/quarto.jl#L2, that env var is the trick to pointing your quarto at a non-default QNR package folder.

[alecloudenback]

Thanks @MichaelHatherly. The two line MWE above,

using Distributed
Distributed.addprocs(4)

Now works with QNR#main but my real example no longer hangs but has an error like the following. Here's a new MWE

# Introduction

```{Julia}
using Distributed
addprocs(1)
@everywhere f() = 1
fetch(@spawnat 2 f())
```

Has the following error:

➜  dist_mwe QUARTO_JULIA_PROJECT=/Users/alecloudenback/.julia/environments/QNR quarto render
[1/2] index.qmd
[2/2] intro.qmd
Starting julia control server process. This might take a while...
Julia server process started.
Running [1/1] at line 4:  using Distributed
ERROR: Julia server returned error after receiving "run" command:

Failed to run notebook: /Users/alecloudenback/prog/dist_mwe/intro.qmd

The underlying Julia error was:

EvaluationError: Encountered 1 error during evaluation

Error 1 of 1
@ /Users/alecloudenback/prog/dist_mwe/intro.qmd:4
Distributed.RemoteException(2, CapturedException(UndefVarError(:Notebook, 0x000000000000970b, Main), Any[(deserialize_module at Serialization.jl:1034, 1), (handle_deserialize at Serialization.jl:933, 1), (deserialize at Serialization.jl:851, 1), (deserialize_datatype at Serialization.jl:1543, 1), (handle_deserialize at Serialization.jl:904, 1), (deserialize at Serialization.jl:851, 1), (handle_deserialize at Serialization.jl:911, 1), (deserialize at Serialization.jl:851 [inlined], 1), (deserialize_msg at messages.jl:87, 1), (message_handler_loop at process_messages.jl:176, 1), (process_tcp_streams at process_messages.jl:133, 1), (#process_messages##0 at process_messages.jl:121, 1)]))
Stacktrace:
 [1] remotecall_fetch(f::Function, w::Distributed.Worker, args::Distributed.RRID; kwargs::@Kwargs{})
   @ Distributed ~/.julia/juliaup/julia-1.12.5+0.aarch64.apple.darwin14/Julia-1.12.app/Contents/Resources/julia/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:465
 [2] remotecall_fetch(f::Function, w::Distributed.Worker, args::Distributed.RRID)
   @ Distributed ~/.julia/juliaup/julia-1.12.5+0.aarch64.apple.darwin14/Julia-1.12.app/Contents/Resources/julia/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:454
 [3] remotecall_fetch
   @ ~/.julia/juliaup/julia-1.12.5+0.aarch64.apple.darwin14/Julia-1.12.app/Contents/Resources/julia/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:492 [inlined]
 [4] call_on_owner
   @ ~/.julia/juliaup/julia-1.12.5+0.aarch64.apple.darwin14/Julia-1.12.app/Contents/Resources/julia/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:565 [inlined]
 [5] fetch(r::Distributed.Future)
   @ Distributed ~/.julia/juliaup/julia-1.12.5+0.aarch64.apple.darwin14/Julia-1.12.app/Contents/Resources/julia/share/julia/stdlib/v1.12/Distributed/src/remotecall.jl:619
 [6] top-level scope
   @ ~/prog/dist_mwe/intro.qmd:0


Stack trace:
    at writeJuliaCommand (file:///Applications/quarto/share/extension-subtrees/julia-engine/_extensions/julia-engine/julia-engine.js:1068:13)
    at eventLoopTick (ext:core/01_core.js:179:7)
    at async executeJulia (file:///Applications/quarto/share/extension-subtrees/julia-engine/_extensions/julia-engine/julia-engine.js:955:20)
    at async Object.execute (file:///Applications/quarto/share/extension-subtrees/julia-engine/_extensions/julia-engine/julia-engine.js:594:20)
    at async renderExecute (file:///Applications/quarto/bin/quarto.js:129865:25)
    at async renderFileInternal (file:///Applications/quarto/bin/quarto.js:130118:35)
    at async renderFiles (file:///Applications/quarto/bin/quarto.js:129914:9)
    at async renderProject (file:///Applications/quarto/bin/quarto.js:130479:23)
    at async _Command.actionHandler (file:///Applications/quarto/bin/quarto.js:135810:24)
    at async _Command.execute (file:///Applications/quarto/bin/quarto.js:95902:7)

[MichaelHatherly]

Ah yeah, that's that annoying serialization of thunks issue, I think. Might be able to get around it by using remotecall directly:

using Distributed
addprocs(1)
@everywhere f() = 1
fetch(remotecall(Core.eval, 2, Main, :(f())))

Happens because notebook code runs in an isolated Notebook module, and Distributed uses Main for stuff, pretty sure remember seeing this problem in Pluto as well, which uses a similar module isolation mechanism as here.