Anyway to freeze pre-render step?

[NFA]

Hi all,

As a part of attempting to do reproducible research, I am testing out quarto for my scientific article I am working on as part of my PhD studies. I have a project consisting of two documents: article.qmd and supplemental.qmd. I edit these in parallel, as I go. In the _quarto.yml I specify different pre-render targets that simply are R scripts that load raw data, produces figures (saved to a figure directory), produce certain numerics and statistics (saved to a .rds file) that I later reference in the documents. Regardless of if I do quarto render, for the entire project or render individual documents with quarto render article.qmd or quarto render supplemental.qmd the pre-render target is run. This adds unnecessary time to the process, especially since nothing has changed. As part of a next step I will incorporate another pre-render script that executes a Stan model, then each render of the quarto project will take close to 30 minutes on my work laptop.

I noticed that I can use manage execution of documents using the freeze option, but it seems to not work for pre-render targets. An option to do something similar to freeze: auto for pre-render targets would be very useful. Currently I just comment it out when I focus on the textual part.

Is there anyway to freeze pre-render or get a similar functionality? Or is commenting out the pre-render line the only workaround for now?

Thanks for a neat project!

[cscheid]

Have you considered using the knitr cache?

[NFA]

No problem at all!

I could do that, and it would certainly work. However, as you say it's a bit of a hack - I would rather probably just continue to comment out pre-render scripts in _quarto.yml , and uncomment them when I want them run. I was hoping that perhaps there was some elegant option I could just enable, very much like the freeze option for the documents themselves.

I think perhaps this is more about that I am slightly going against what certain aspects of the design philosophy behind quarto is. That is considering I am not using the code chunks to generate figures or calculate things but rather just load and insert pre-calculated results into the text, or the pre-made figures.

My rationale for doing it my way is that I don't want to duplicate unnecessary code or statistical line(s) of argument into multiple documents (right now those are the article itself and the supplemental materials, but perhaps later some presentation slides). Therefore, I decided to use the pre-render and have things like:

   pre-render: 
      - scripts/cst_analysis.R # Generates two figures for the article, two figures for supplemental materials, and one table for supplemental materials
      - scripts/srf_analysis.R # one figure for article, table for supplemental
      - scripts/psd_analysis.R # runs a mixture model in Stan, and from results one figure for article and 6 for supplemental

I want to keep the full analysis of the data together in one place, some figures will be highlighted in the article, and some are for the supplemental materials. Many of the scripts combine my Design of Experiments matrix with input and output data and perform various calculations and are by nature rather large. I find it quite advantageous, my scripts are well documented using comments and I have message() calls every now and then to output to console (when quarto render is called) the current figure or statistical test being performed.

[mcanouil]

In this case I would setup a targets workflow instead of several R scripts.
Note that with targets you have a way better cache system than the one from knitr, for instance, you don't have to save figures/tables since you can load base plots, ggplot2 or basically any R object from any step of your workflow.

Or source the R scripts inside a code cell to have freeze/caching working.

[NFA]

Hi, thanks for the reply. I assume you are referring to the https://docs.ropensci.org/targets/ package. I assume I would need to set up a another layer on top of quarto with a targets makefile structure specifying the dependency graph with the quarto document(s) as the final output. I see how that can help, even more so, if the dependency hierarchy is more complex than what I have. No time to learn another tool right now, need to get the paper done - but I will keep it in mind for my next publication.

I do make documents (previosly .Rmd, now .qmd) with R or Python code inside code chunks, but they are mostly for internal use as a basis for discussion or validation together with research colleagues on visualizations, computations and statistical analyses. The document that is slated for publication I want to keep "cleaner" if that makes sense, but still with that paper trail and connection to the analyses performed. For example, the first R chunk is a setup chunk (label: setup), where I load the every number I present (R^2. p-values, F-statistics, means, ranges, etc), and formatted accordingly (significant digits, decimal numbers, power representation). Subsequently I just output that using inline R code, r cst$model$parameter_b_interval.

[mcanouil]

Use include shortcode (possibly conditional) or knitr child with your R scripts.
This way in your "main" document you don't "see" it.

---
Some yaml header
---

Some text

{{< include _pre-requisite.qmd >}}

Some text with `r something_from_pre_requisite`.

Where the file is something like

```{r}
#| include: false
library(package)
source("analysis.R")
```

This way, and as said previously, you'll be able to use knitr caching.

[NFA]

Aha! That worked quite nicely. You have my thanks.

Maybe not the most elegant solution as either quarto or pandoc makes use of \r after processing a chunk, overwriting the console text I added. So my user-faced process documentation is lost during the build process, but I can live with that. I will still have that targets package on my radar for upcoming works.

Thanks for all the help!

[thomascwells]

I haven't implemented this yet (still just learning quarto), but would this be a good use-case for project profiles?

I think something like this might work:

_quarto.yml (as you currently have it but without the pre-render)

project:
  render:
    - article.qmd
    - supplemental.qmd
...
...

_quarto-include-prerender.yml (just these two lines)

project:
  pre-render: my_prerender.R

Then at the command line:

# without prerender
quarto render

# with prerender
quarto render --profile include-prerender