Render a single qmd using multiple parameter sets

[joxborrow]

Is there a way to render a single qmd multiple times using different parameters as part of a quarto project?

[mcanouil]

See https://quarto.org/docs/computations/parameters.html and/or https://quarto.org/docs/projects/profiles.html.

[zachsiegel-capsida]

@mcanouil I see many versions of "render a .qmd notebook many times with different parameters to produce different output files in a Python loop" online.

This is a worse, DIY version of Quarto projects, that fails to achieve many of the nice-to-haves of Quarto projects (like automatic navbar population based on contents of the project directory and convenient command-line options for freezing/caching output, etc.).

I'm not sure whether this is what @joxborrow is after, but I personally would love a way to execute the same .qmd file with multiple parameters, to produce multiple output files.

For me, the ideal solution would be an extension to the Quarto projects render option that allows the same .qmd file to be rendered multiple times with different parameters, to different output files, e.g.:

project:
  render:
    - template: analysis_1.qmd
      params: study_1.yml
      output: study_1-analysis_1.html
    - template: analysis_1.qmd
      params: study_2.yml
      output: study_2-analysis_1.html
...

There are many frameworks that support static site generation like Quarto, and there is precedent for an interface to dynamically specify generating output targets like this. In NextJS, this would be getStaticPaths (pages router) or generateStaticParams (app router).

Currently, there are a few ways I can imagine using the Quarto "project" system to achieve this:

• a pre-render script that makes multiple copies of the same base .qmd files.
• a pre-render script that makes many .qmd files that consist of a single {{< include _template.qmd >}} tag. This is similar to generating a bunch of .yml files, but dynamically generating a bunch of .yml files is trivial from any programming environment, whereas dynamically generating .qmd files is definitely "code-gen" and is not as trivial.

Any response would be greatly appreciated! Maybe the render configuration already supports using the same .qmd file with multiple output-file options and there is just a lack of documentation!

Thank you! Quarto is great and I'm thrilled to get to use it.

[zachsiegel-capsida]

My point was that:

• I suggested a way to render the same .qmd "template" into multiple output files (each with a different .yml config), e.g. study_1-analysis_1.html and study_2-analysis-1.html via study_1.yml and study_2.yml.
• This would make it ill-defined which output a link such as [analysis_1](analysis_1.qmd) should point to.

I made the counterpoint to my suggestion to figure out whether my suggestion was deeply violating some design decisions in Quarto.

[mcanouil]

This is already possible.
You can use profiles, you can pass yaml metadata or params to the CLI, etc.

Please read the documentation, try it, then if it does not work for you I strongly suggest to come back with a GitHub repository showing the structure of your project and what you want as outputs.

[zachsiegel-capsida]

I strongly suggest you read the documentation and try to use it before making wrong assumptions.

"read the damn manual" is a fair point! I will mention that I did try out Quarto quite a bit and wrote a Python script that uses subprocess to dynamically build a project. I absolutely pored over the documentation about Quarto projects first to figure out whether this already exists!

Profiles can be exclusive, inclusive, can be merged.

This is new to me! I will try to find out more about this!

[zachsiegel-capsida]

Please read the documentation, try it, then if it does not work for you I strongly suggest to come back with a GitHub repository showing the structure of your project and what you want as outputs.

If I can't make it happen, I will absolutely make a minimal example of an input-output pair that I want to achieve and post a new issue referencing this one. If there's a way to do it that I still haven't figured out, having the dialogue there (with the example repo linked) will be more productive for future viewers.

Thank you so much @mcanouil !

[mcanouil]

Not sure to understand what you did to build a project but seems more complicated than it probably needs to be.

Quarto has commands to create projects and has extension mechanisms to create project templates.

At least go through the get started and other relevant guides for your use.

[zachsiegel-capsida]

If you set navbar and sidebar contents use glob pattern and possibly different pattern in profiles, how's that not working?

Surely I'm missing something but my understanding of profiles is that using multiple profiles will build multiple versions of your entire project into different directories. But the files in one of those directories would have been built without knowledge of the other directories, and thus the auto-generated navbars in the static files in those built directories (via globs or whatever else) would not be able to link to each other.

Any additional comments are still so appreciated, learning a lot from this dialogue @mcanouil

[f2il-kieranmace]

@zachsiegel-capsida did you figure this out? I have the similar (and simpler) question: #7805

[f2il-kieranmace]

or @joxborrow ?

[f2il-kieranmace]

I think I answered how to do it, by using the include technique. See accepted answer in #7805

[zachsiegel-capsida]

Hey @f2il-kieranmace I don't love your solution since it requires dynamically generating .qmd files. To me, dynamically generating a standard serialization format (like .json or .yml) is preferable over .qmd code-gen. Quarto supports your use-case in this way using profiles as @mcanouil suggested.

Instead of generating .qmd files, you could generate the following .yml files:

_quarto-USA.yml:

title: "USA"
params:
  country: USA

_quarto-UK.yml:

title: "UK"
params:
  country: UK

and then use quarto render analysis.qmd --profile US && quarto render analysis.qmd --profile UK to achieve what you want. To set the output directory or filename you may have to add something like output-file: US.html or output-dir US to your config files.

I have some issues with this method that I will put in a separate message.

[zachsiegel-capsida]

@mcanouil Should I open separate issues for these is this a good place to discuss them?

1. There is no (documented) way to specify input parameters in the yaml that sets other parameters for Python (/Jupyter) execution.

The only way to specify parameters to quarto render is with the command line -P alpha:0.2 argument or with the --execute-params params.yml argument. The parameters must live in a separate .yml file (params.yml) to your other render options.

For R, the params key can be specified in _quarto.yml or _quarto-XXXX.yml profile files, but for Python there is nothing equivalent. There is not even a way to pass the desired value of --execute-params via yaml!

My ideal solution would be able to specify params via a params: key in any yaml file included in a quarto render command (even if rendering uses Jupyter). I also think it should be possible to specify an execute-params: key in those yaml files in case someone really wants to store their variables in a separate .yml or .json file.

Commentary on whether I'm missing something on this point would be greatly appreciated.

2. I believe there should be a way to call quarto render sequentially, with multiple profiles. The command quarto render --profile a,b merges profiles a and b, which is a great feature. But sometimes it is desirable to execute quarto render analyses --profile experiment_1 && quarto render analysis --profile experiment_2, i.e. render the same project multiple times (to produce multiple sets of output files, possibly in different directories using the output-dir: variable in each profile).

Ideally for me, this would be possible via the command line or via project yaml.

Like, maybe in _quarto.yml it could be possible to have an entry like this:

profile:
  default-render-all-profiles: true
  group: 
    - [basic, advanced]

and then quarto render would render the project with each profile activated, one after another. Perhaps specifying groups: wouldn't even be necessary - all _quarto-XXXX.yml files in the current directory could be used.

Commentary on whether I'm missing something on this point would be greatly appreciated.

@mcanouil I really appreciated your previous comments, additional feedback would be very appreciated!

[mcanouil]

Here is a modified version I can share: https://github.com/mcanouil/generate-quarto-invoices

[EmiliaJarochowska]

Wow, thank you for this, this indeed works fine! A much better solution with generating metadata into each file, rather than what I tried to do by passing parameters from a parameters.yaml file. I really appreciate that you modified this version to share it, thank you 💐

[mcanouil]

Well you can do something similar/complementary using parameters it it's mostly for computation.
If you don't need computation then metadata is the way to go.

[zachsiegel-capsida]

@mcanouil I've also been passing "parameters" as root-level yaml keys, but I worry this is an anti-pattern as it could conflict with other keys Quarto uses. I much prefer the R params key for this, and I wish it was available in Python.

[mcanouil]

There is no limits with metadata.
Put everything under whatever-name-that-is-unlikely-to-exist.
Params are no metadata, it's a storage for computational parameters given to knitr.

Metadata and params are two very different things.

[EmiliaJarochowska]

Thank you @zachsiegel-capsida for keeping on asking these questions :-) and @mcanouil and @f2il-kieranmace for the solutions you've suggested. I followed this discussion carefully and I second @zachsiegel-capsida's questions. I couldn't make his suggested solution work nor translate the documentation into a different solution so I hope you will have some patience 🙂 I use quarto a lot and enthusiastically but solving this problem also required me to reach to Python, which I don't use that often.

My use case follows the more general situation outlined by @zachsiegel-capsida. I want to generate reports for batches of output files, which are dynamically generated in series of simulations, so it is important to be able to render the same document for a new batch of files, each with a unique name, without a lot of manual work. Basically I am trying to use Quarto as a lab notebook to publish summaries of each simulation run. Summaries are all based on one template with the same plots and calculations that I am trying to run multiple times with parameters read from a yml file. I am aware of R solutions (e.g. here) but decided that it is important to use quarto without requiring R Studio because the potential community in my discipline is very Python-dominated.

Based on the suggestions in this thread and this brief one by @mcanouil I reached the following situation:

• download a batch into a data folder outside of the repository
• generate a parameters.yaml from the list of files saved in data
• generate copies of the template Quarto document, to be rendered one by one
• iterate through the files listed in parameters.yaml to execute quarto render for each file in rendered_files
  All this is done in Python. Also I cannot make it work and I am not sure why. Running the same command from CLI gives a rendered output but running it with Pythons's subprocess doesn't.

1. Is there a way of doing it using a quarto project with profiles? I do not see how to apply it here but it seems to be strongly argued for by @mcanouil
2. Is there any other way of running the same quarto template multiple times with parameters read from a yaml file when the list of files to be generated is dynamic?
3. If none of the above works, has anyone been successful in automating this task using Python's subprocess and have a suggestion why the output might not be generated?

I hope someone will be able to answer any of these questions. In case it helps, the code is here: https://github.com/MindTheGap-ERC/LMA_utils/tree/automate_execution (just a prototype for now)

[EmiliaJarochowska]

Ok so I have to apologize to @mcanouil, I managed to set up a script that generates a list of files to be rendered in a Quarto project and that solution works. It is an adaptation of the solution by @f2il-kieranmace so thank you all for sharing this here.

[gmmps]

Hi! I´m new to this. I wonder if you know if there is a step by step.

I have my notebook Untitled4.ipynb. I´m trying to render the same file with different combinations of params as is possible with R and Rstudio, but I can´t find the way to do it with python and the render on the command line.

---

title: "Probando reportes con varios parámetros"
format:
html:
code-fold: true

---

#| tags: [parameters]

years = 2022
provinces= 'Provincia A'
stations= 'Invierno'

#| echo: false
from IPython.display import display, Markdown

print(years, provinces, stations)

display(Markdown(
f"""
Este es un reporte correspondiente al Año {years} en la {provinces} en {stations}.
"""
)).

The params I can use are these:

years = [2022, 2023, 2024]
provinces = ['Provincia A', 'Provincia B', 'Provincia C', 'Provincia D']
seasons = ['Invierno', 'Primavera', 'Verano', 'Otoño']

I can´t find the way to use --execute params. I have a .txt file with the possible combinations, such as :
(2022, 'Provincia C', 'Otoño')
(2022, 'Provincia D', 'Invierno')
(2022, 'Provincia D', 'Primavera')
(2022, 'Provincia D', 'Verano')
(2022, 'Provincia D', 'Otoño')
(2023, 'Provincia A', 'Invierno')
(2023, 'Provincia A', 'Primavera').

and I also want the output to have for example {years}{provinces}{stations}.html as is possible in R.

When I tried to change params manually :quarto render Untitled4.ipynb -P provinces: 'Provincia B' -P years:2023 -P stations:'Primavera' --output test_1.html the error output was: pandoc: Provincia B: withBinaryFile: does not exist (No such file or directory) and when I deleted -P provinces: 'Provincia B' it ran, but with the parameters set in the notebook (it didn´t take into account the changes).

Thanks!!!

[mcanouil]

@gmmps Could you format your message using code blocks as currently it's a bit hard to read.

You could share a small self-contained "working" (reproducible) example to work with, i.e., a complete Quarto document (as below) or a Git repository.

You can share a Quarto document using the following syntax, i.e., using more backticks than you have in your document (usually four ````).

````qmd
---
title: "Reproducible Quarto Document"
format: html
engine: knitr
---

This is a reproducible Quarto document using `format: html`.
It is written in Markdown and contains embedded R code.
When you run the code, it will produce a plot.

```{r}
plot(cars)
```

![A placeholder image](https://placehold.co/600x400.png)

The end.
````

[fernandoroa]

Another possible solution: https://stackoverflow.com/questions/76158024/include-parameters-in-header-and-footer-in-quarto-rmarkdown-pdf#answer-76405809