Set seed prior to rendering

[richarddmorey]

Description

I opened this request over on quarto-r, but was directed here from https://github.com/quarto-dev/quarto-r/issues/90.

When generating many documents from a single source file, it is extremely useful to be able to set the random seed prior to rendering. That way, you can take a single document and render it many times with different, but reproducible, results. For instance, I use this to generate random assignments in a teaching context.

With Rmarkdown, I can set a random seed prior to rendering, and I'll get the same document every time. Consider the following Rmd file (which I'm calling test_seed_rmd.Rmd):

---
title: "Untitled"
output: html_document
---

```{r}
rnorm(1)
```

If I use withr::with_seed to compile it:

withr::with_seed(
  seed = 123,
  {
    rmarkdown::render('test_seed_rmd.Rmd')
  }
)

then I get the output ## [1] -0.5604756 every time.

Because of the way Quarto works (using the CLI), this seems to be more difficult (also, because it is designed with support for multiple languages in mind, and each has their own seed - I imagine that makes thinking about RNGs hard). I cannot, for instance, do this:

withr::with_seed(
  seed = 123,
  {
    quarto::quarto_render('test_seed_qmd.qmd')
  }
)

and expect a reproducible document, because the rendering is passed off to a CLI program.

@cderv brainstormed some ideas here (https://github.com/quarto-dev/quarto-r/issues/90#issuecomment-1905839682) and I was also thinking environment variables might work (e.g. setting an environment variable as a seed that is then picked up within the document)? (That's not ideal, though, because it requires extra code in the document itself).

Is there any way of making this work?

[mcanouil]

Hum, seems very tricky as even setting the seed for knitr is known to have issues.
For Quarto, such feature would also have to be implemented for Jupyter and not only knitr.

This would be indeed a nice enhancement.

[cscheid]

For Quarto, such feature would also have to be implemented for Jupyter and not only knitr.

Let me stop everyone here: this is the root of the issue. withr::with_seed is R-specific. Quarto can't do this for every Jupyter kernel that uses a different programming language.

That's just not a feature we can write.

[mcanouil]

I think the R part is mostly an example rather than the solution for everything, but I do not know how simple it is to control the random number generator in Jupyter.

[cderv]

The global issue for Quarto in a language-agnostic way is whether it will be possible to execute some arbitrary code for engine execution before any other rendering happens for the engine.

This is indeed really tricky because

• Quarto also runs some code in the engine language (python, Julia, or R), so should it be run before or after this?
  An easy way for the user right is to add a first cell with the desired process so that the following cells consider this.
• This goes beyond reproducibility as a document would not be self-contained in their execution. And this raises the question to me about should this be an engine feature or really a quarto feature.

That is my understanding of this topic.

[cscheid]

how simple it is to control the random number generator in Jupyter.

It's impossible - different Jupyter kernels run different programming languages with different PRNG APIs

[richarddmorey]

For Quarto, such feature would also have to be implemented for Jupyter and not only knitr.

Let me stop everyone here: this is the root of the issue. withr::with_seed is R-specific. Quarto can't do this for every Jupyter kernel that uses a different programming language.

That's just not a feature we can write.

The question is not "how do I use withr::with_seed with quarto, but "how can I set the random seed (or the random seeds) prior to compilation?"

Re: @cderv: "This goes beyond reproducibility as a document would not be self-contained in their execution." - I think with RNGs the documents are never "self-contained" in a sense, in that the RNG state is not determined by the document itself (unless explicitly set in the document), right? So the ask is a way of determining the initial RNG state(s) besides "whatever the engines initializes them to".

[cscheid]

So the ask is a way of determining the initial RNG state(s) besides "whatever the engines initializes them to".

Unfortunately that's impossible even in principle. Different programming languages use different PRNGs and different APIs. Quarto allows you to run any programming language you want. This is not only through the Jupyter engine: reticulate runs Python, and although I haven't tested it, I would be very surprised with withr::with_seed does anything with reticulate, which is arguably still R code (albeit calling Python)

This is unfortunate but an inevitable consequence of Quarto being a broader system: we cannot determine the environment accurately enough ahead of execution for such a feature to be safe and useful.

Personally speaking, I think a better design is for your analysis code to check an environment variable like "USE_FIXED_SEED" and then take the value from there if set. In that way, its behavior is described independently of how you chose to call the rendering process.

[richarddmorey]

Personally speaking, I think a better design is for your analysis code to check an environment variable like "USE_FIXED_SEED" and then take the value from there if set. In that way, its behavior is described independently of how you chose to call the rendering process.

In my use case, the document isn't necessarily "mine" - that is, it was (or may be) written by someone else and I want a reproducible way of rendering it without changing the document itself. This doesn't seem an unreasonable desire in terms of reproducibility (that is, to be able to control the rendering environment).

[mcanouil]

control the rendering environment

You will very likely never be able to do that unless you use technology such as Docker container.

[cscheid]

This doesn't seem an unreasonable desire

It's an understandable desire, but it's not possible in the context of where Quarto needs to operate. Again, I understand why you want to do it. But let me put it plainly: it's not that it's hard for us to implement, or that it's not worth the effort. It's that it's not possible to do it in Quarto.