Struggles with environment variables

[bbolker]

Following on from #217:

• I can get one of the suggested solutions to work (specify env variables YAML frontmatter) but not the other (via _environment file); it would be nice to make _environment work, as the environment variables I need to define are specific to my environment and so shouldn't be used by others (another use case would be if they were API secrets ...)
• I can't find the solution documented anywhere except in the (closed) issue, it would be nice if it were more visible.

A bit more detail:

• I want to use RCall and my R installation is a non-standard place, so I need to set **LD_LIBRARY_PATH=/usr/local/lib/R/lib
• if this env variable isn't set/seen from the running environment, RCall fails.

This works fine:

---
title: "Julia env test"
engine: julia
julia:
  env: [ "LD_LIBRARY_PATH=/usr/local/lib/R/lib" ]
---

```{julia}
ld_library_path = get(ENV, "LD_LIBRARY_PATH", "missing")
println("LD_LIBRARY_PATH = $ld_library_path")
```

if I remove the env: stuff in the YAML, then LD_LIBRARY_PATH is printed as "missing", even if I have an _environment file containing the line LD_LIBRARY_PATH=/usr/local/lib/R/lib (and even after issuing quarto call engine close env.qmd to make sure I was restarting the runner). As mentioned above, subsequent attempts to load/use RCall will fail (for me) if the env variable is not set.

I looked through

• https://github.com/PumasAI/QuartoNotebookRunner.jl/blob/main/README.md
• https://quarto.org/docs/computations/julia.html
• https://quarto.org/docs/projects/environment.html
• https://docs.julialang.org/en/v1/manual/environment-variables/ (refers to Julia-specific environment variables)

but couldn't find any relevant information (I may have missed it), either about the fact that QuartoNotebookRunner inherits environment variables differently, or describing the YAML solution that works.

Quarto version 1.7.32, Julia 1.11.6, Ubuntu 24.04.3 LTS, R (unstable) (2025-08-10 r88559). As far as I can tell I'm using version 0.17.3 of QuartoNotebookRunner (by digging down into my .julia/packages/QuartoNotebookRunner directory: I don't know a better way to figure this out ...

Results of quarto check:

Quarto 1.7.32
[✓] Checking environment information...
      Quarto cache location: /home/bolker/.cache/quarto
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.6.3: OK
      Dart Sass version 1.85.1: OK
      Deno version 1.46.3: OK
      Typst version 0.13.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.7.32
      Path: /opt/quarto/bin

[✓] Checking tools....................OK
      TinyTeX: (not installed)
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Using: Installation From Path
      Path: /usr/local/texlive/2024/bin/x86_64-linux
      Version: 2024

[✓] Checking Chrome Headless....................OK
      Using: Chrome found on system
      Path: /usr/bin/chromium-browser
      Source: PATH

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.12.3
      Path: /usr/bin/python3
      Jupyter: 5.3.2
      Kernels: python3

[✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........OK
      Version: 4.6.0
      Path: /usr/local/lib/R
      LibPaths:
        - /usr/lib/R/site-library
        - /usr/local/lib/R/library
      knitr: 1.50
      rmarkdown: 2.29

[✓] Checking Knitr engine render......OK

[jkrumbiegel]

The environment variable mechanism needs an overhaul. What currently happens is that when the control server is started through the first quarto render ..., it inherits the env variables from quarto. Quarto itself collects the env variables to set from all the different supported places. So the control server gets them but only when it starts, so it wouldn't pick up on any changes.

The server then starts worker processes which actually render notebooks but each of those gets its completely separate set of env variables which are set here:

QuartoNotebookRunner.jl/src/server.jl

Line 33 in 8ba293e

cd(() -> Malt.Worker(; exe, exeflags = _exeflags, env), dirname(path))

The env variables we set are determined here

QuartoNotebookRunner.jl/src/server.jl

Lines 131 to 175 in 8ba293e

	function _exeflags_and_env(options)
	env_exeflags =
	JSON3.read(get(ENV, "QUARTONOTEBOOKRUNNER_EXEFLAGS", "[]"), Vector{String})
	options_exeflags = map(String, options["format"]["metadata"]["julia"]["exeflags"])
	# We want to be able to override exeflags that are defined via environment variable,
	# but leave the remaining flags intact (for example override number of threads but leave sysimage).
	# We can do this by adding the options exeflags after the env exeflags.
	# Julia will ignore earlier uses of the same flag.
	exeflags = [env_exeflags; options_exeflags]
	env = map(String, options["format"]["metadata"]["julia"]["env"])
	# Use `--project=@.` if neither `JULIA_PROJECT=...` nor `--project=...` are specified
	if !any(startswith("JULIA_PROJECT="), env) && !any(startswith("--project="), exeflags)
	# Set it via the env variable since this is the "weakest" form of
	# setting it, which allows for an implicit `--project=` provided by a
	# custom linked `juliaup` channel to override it, e.g `julia
	# +CustomChannel` that is a link to `julia +channel --project=@global`.
	pushfirst!(env, "JULIA_PROJECT=@.")
	end
	# if exeflags already contains '--color=no', the 'no' will prevail
	pushfirst!(exeflags, "--color=yes")
	# Ensure that coverage settings are passed to the worker so that worker
	# code is tracked correctly during tests.
	# Based on https://github.com/JuliaLang/julia/blob/eed18bdf706b7aab15b12f3ba0588e8fafcd4930/base/util.jl#L216-L229.
	opts = Base.JLOptions()
	if opts.code_coverage != 0
	# Forward the code-coverage flag only if applicable (if the filename
	# is pid-dependent)
	coverage_file =
	(opts.output_code_coverage != C_NULL) ?
	unsafe_string(opts.output_code_coverage) : ""
	if isempty(coverage_file) || occursin("%p", coverage_file)
	if opts.code_coverage == 1
	push!(exeflags, "--code-coverage=user")
	elseif opts.code_coverage == 2
	push!(exeflags, "--code-coverage=all")
	elseif opts.code_coverage == 3
	push!(exeflags, "--code-coverage=@$(unsafe_string(opts.tracked_path))")
	end
	isempty(coverage_file) || push!(exeflags, "--code-coverage=$coverage_file")
	end
	end
	return exeflags, env
	end

The workers specifically should not inherit env variables unless I'm missing something, that behavior stems from the original Malt implementation.

The quarto devs themselves have determined that proliferating env variables by mutating them in the quarto process is not good: quarto-dev/quarto-cli#12621

So in the future, QuartoNotebookRunner will receive a set of env variables that quarto collates and will take care to have each worker process run with those. That work started here #306