How to use Julia engine when qmd uses packages?

[sbacelar]

Description

I am trying to use Julia with the Julia engine QuartoNotebookRunner.jl.
I have installed Quarto pre-release build 1.5.37 for mac. I have been able to run the simple example on the terminal and rendering a pdf.

but not the one in the same web page (https://quarto.org/docs/computations/julia.html) where there is a need for the Plots package. I added the Plots package to my work directory and tried to do render document.qmd --to pdf and I got this error insisting that I should install Plots:

shell> quarto render document.qmd --to pdf
Running [1/1] at line 16:  using Plots
ERROR: Julia server returned error after receiving "run" command:
Failed to run notebook: /Users/sergiobacelar/Dropbox/JULIA/sbquarto_2/document.qmd
ERROR: EvaluationError: Encountered 1 error during evaluation

Error 1 of 1
@ /Users/sergiobacelar/Dropbox/JULIA/sbquarto_2/document.qmd:16
ArgumentError: Package Plots not found in current path.
- Run `import Pkg; Pkg.add("Plots")` to install the Plots package.```

[cscheid]

@jkrumbiegel Would you mind helping out here?

[cscheid]

@jkrumbiegel Thanks, this is very helpful. I think that adding step-by-step instructions to our docs explaining how to install (say) Plots in one such environment would be a big improvement, and wouldn't take a lot of additional space.

[cderv]

Thanks a lot @jkrumbiegel ! This clarifies a lot - we should definitely add more about this in our doc.

[jkrumbiegel]

I have written up something in quarto-dev/quarto-web#1102

[sbacelar]

I usually use Quarto with R. Now, I am trying to use it with Julia, and I am having some problems. I only tried to run the example on the web page [https://quarto.org/docs/computations/julia.html] using the Jupyter and Julia engines and Quarto version 1.5.37. Let me say first that I had no problems running it on Ubuntu (I had only to install rsvg-convert) and partially on WSL2. But I had problems running the same example on MacOS (GFK and Plots not found) and Windows 11 (I couldn't install rsvg-convert with Chocolatey, and I also had problems with Plots. I am including the error messages I received in each case:

MacOS 11.7.10 jupyter engine: quarto render --to pdf

Outputs pdf but inserts the following after the figure:

GKS: dlopen(/Users/sergiobacelar/.julia/artifacts/86a2153e203fb861d380988509048a2ffa8e3a9f/li
Referenced from: /Users/sergiobacelar/.julia/artifacts/86a2153e203fb861d380988509048a2ffa8e
Reason: Incompatible library version: cairoplugin.so requires version 11603.0.0 or later, b

MacOS 11.7.10 julia engine: quarto render --to pdf

(sbquarto_2) pkg> st
Status ~/Dropbox/JULIA/sbquarto_2/Project.toml
[91a5bcdd] Plots v1.40.4

shell> quarto render document.qmd --to pdf
Running [1/1] at line 16: using Plots
ERROR: Julia server returned error after receiving "run" command:
Failed to run notebook: /Users/sergiobacelar/Dropbox/JULIA/sbquarto_2/document.qmd
ERROR: EvaluationError: Encountered 1 error during evaluation

Error 1 of 1
@ /Users/sergiobacelar/Dropbox/JULIA/sbquarto_2/document.qmd:16
ArgumentError: Package Plots not found in current path.

• Run import Pkg; Pkg.add("Plots") to install the Plots package.

Windows 11 julia engine: quarto preview --to pdf

Julia server process started.
Running [1/1] at line 16: using Plots
ERROR: Julia server returned error after receiving "run" command:
Failed to run notebook: D:\Dropbox\JULIA\sbquarto_2\document.qmd
ERROR: EvaluationError: Encountered 1 error during evaluation

Error 1 of 1
@ D:\Dropbox\JULIA\sbquarto_2\document.qmd:16
Error showing value of type Plots.Plot{Plots.GRBackend}
only png or svg allowed. got: :pdf
Stacktrace:
[1] error(s::String)
@ Base .\error.jl:35
[2] _show(io::IOContext{IOBuffer}, ::MIME{Symbol("text/html")}, plt::Plots.Plot{Plots.GRBackend})
@ Plots C:\Users\sbacelar.julia\packages\Plots\ju9dp\src\output.jl:203
[3] #invokelatest#2
@ .\essentials.jl:892 [inlined]
[4] invokelatest
@ .\essentials.jl:889 [inlined]
[5] show(io::IOContext{IOBuffer}, m::MIME{Symbol("text/html")}, plt::Plots.Plot{Plots.GRBackend})
@ Plots C:\Users\sbacelar.julia\packages\Plots\ju9dp\src\output.jl:232
[6] show(io::IOContext{IOBuffer}, m::String, x::Plots.Plot{Plots.GRBackend})
@ Base.Multimedia .\multimedia.jl:123

ERROR: Internal julia server error

Stack trace:
at writeJuliaCommand (file:///C:/Users/sbacelar/AppData/Local/Programs/Quarto/bin/quarto.js:41672:19)
at eventLoopTick (ext:core/01_core.js:153:7)
at async executeJulia (file:///C:/Users/sbacelar/AppData/Local/Programs/Quarto/bin/quarto.js:41581:22)
at async Object.execute (file:///C:/Users/sbacelar/AppData/Local/Programs/Quarto/bin/quarto.js:41332:20)
at async renderExecute (file:///C:/Users/sbacelar/AppData/Local/Programs/Quarto/bin/quarto.js:77870:27)
at async renderFileInternal (file:///C:/Users/sbacelar/AppData/Local/Programs/Quarto/bin/quarto.js:78038:43)
at async renderFiles (file:///C:/Users/sbacelar/AppData/Local/Programs/Quarto/bin/quarto.js:77906:17)
at async render (file:///C:/Users/sbacelar/AppData/Local/Programs/Quarto/bin/quarto.js:82734:21)
at async Command.actionHandler (file:///C:/Users/sbacelar/AppData/Local/Programs/Quarto/bin/quarto.js:82882:32)
at async Command.execute (file:///C:/Users/sbacelar/AppData/Local/Programs/Quarto/bin/quarto.js:8015:13)

Windows 11 jupyter engine: quarto preview --to pdf

FATAL (C:/Users/sbacelar/AppData/Local/Programs/Quarto/share/filters/main.lua:7144) An error occurred:
Could not convert a SVG to a PDF for output. Please ensure that rsvg-convert is available on the path.
Error running filter C:/Users/sbacelar/AppData/Local/Programs/Quarto/share/filters/main.lua:
...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:2617: attempt to call a nil value (global 'crash_with_stack_trace')
stack traceback:
...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:2196: in function 'fail'
...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:7144: in upvalue 'call_rsvg_convert'
...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:7168: in function <...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:7150>
(...tail calls...)
...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:2925: in function <...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:2919>
[C]: in ?
[C]: in method 'walk'
...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:535: in function <...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:524>
(...tail calls...)
...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:1312: in local 'callback'
...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:1330: in upvalue 'run_emulated_filter_chain'
...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:1366: in function <...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:1363>
stack traceback:
...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:535: in function <...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:524>
(...tail calls...)
...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:1312: in local 'callback'
...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:1330: in upvalue 'run_emulated_filter_chain'
...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:1366: in function <...lar/AppData/Local/Programs/Quarto/share/filters/main.lua:1363>

WSL2 with Julia engine: quarto preview --to pdf

ERROR: Julia server returned error after receiving "run" command:
Failed to run notebook: /home/sbacelar/JULIA/sbquarto/document2.qmd
ERROR: EvaluationError: Encountered 1 error during evaluation

Error 1 of 1
@ /home/sbacelar/JULIA/sbquarto/document2.qmd:16
Error showing value of type Plots.Plot{Plots.GRBackend}
only png or svg allowed. got: :pdf
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:35
[2] _show(io::IOContext{IOBuffer}, ::MIME{Symbol("text/html")}, plt::Plots.Plot{Plots.GRBackend})
@ Plots ~/.julia/packages/Plots/ju9dp/src/output.jl:203
[3] #invokelatest#2
@ ./essentials.jl:892 [inlined]
[4] invokelatest
@ ./essentials.jl:889 [inlined]
[5] show(io::IOContext{IOBuffer}, m::MIME{Symbol("text/html")}, plt::Plots.Plot{Plots.GRBackend})
@ Plots ~/.julia/packages/Plots/ju9dp/src/output.jl:232
[6] show(io::IOContext{IOBuffer}, m::String, x::Plots.Plot{Plots.GRBackend})
@ Base.Multimedia ./multimedia.jl:123

ERROR: Internal julia server error

Stack trace:
at writeJuliaCommand (file:///opt/quarto/bin/quarto.js:41672:19)
at eventLoopTick (ext:core/01_core.js:153:7)
at async executeJulia (file:///opt/quarto/bin/quarto.js:41581:22)
at async Object.execute (file:///opt/quarto/bin/quarto.js:41332:20)
at async renderExecute (file:///opt/quarto/bin/quarto.js:77870:27)
at async renderFileInternal (file:///opt/quarto/bin/quarto.js:78038:43)
at async renderFiles (file:///opt/quarto/bin/quarto.js:77906:17)
at async render (file:///opt/quarto/bin/quarto.js:82734:21)
at async renderForPreview (file:///opt/quarto/bin/quarto.js:83760:26)
at async render (file:///opt/quarto/bin/quarto.js:83643:29)

[cscheid]

@sbacelar please create a new discussion with this, you're commenting deep inside a separate thread.

[dmi3kno]

Do I understand correctly that Quarto uses its own "private" environment only to run QuartoNotebookRunner.jl? Is it absolutely certain that it will never require anything to be installed in that private environment directly?

Second question: is there a reason why this setting is not default in Quarto?

engine: julia
julia:
  exeflags: ["--project=."]

[cderv]

Since this discussion, we tried to update the doc at https://quarto.org/docs/computations/julia.html#using-the-julia-engine to make it more clear.

Do I understand correctly that Quarto uses its own "private" environment only to run QuartoNotebookRunner.jl? Is it absolutely certain that it will never require anything to be installed in that private environment directly?

Yes. AFAIU this is use to start the Julia server to run the Julia cells.

Second question: is there a reason why this setting is not default in Quarto?

engine: julia
julia:
  exeflags: ["--project=."]

The default used is --project=@. which seems to be more "clever". From our doc:

This makes Julia search for an environment (a Project.toml or JuliaProject.toml file) starting in the directory where the quarto notebook is stored and walking up the directory tree from there.

From Julia's doc: https://docs.julialang.org/en/v1/manual/environment-variables/index.html#JULIA_PROJECT

If the variable is set to @. (note the trailing dot) then Julia tries to find a project directory that contains Project.toml or JuliaProject.toml file from the current directory and its parents. See also the chapter on Code Loading.

Setting --project=. only consider the current directory IIUC.

Is the current default not working ? It should cover the same usage case than --project=.

Direct Julia engine support is quite new, so any feedback welcome really to improve it !

Hopefully the new doc helps understand;