Workaround to control the execution order of qmd files in a project?

[allefeld]

Description

There have been a number of issues/discussions about this (my #6944, @eculler's #10463, @maxdrohde's #10490). Official support is planned with 'Epic: Execution Planner' (#6518), but it is still a while until that will be finished. The purpose of this post is to discuss what workarounds can be used in the meantime.

On Linux, the current behavior seems to be that files are processed in the order in which they occur in the directory (i.e. without any sorting). How that order is determined is a matter of the underlying filesystem; again on Linux using ext4, it seems to depend on the order of hashes of filenames. On other OSes / filesystems it may be alphabetical, or by time. Therefore in principle one could fiddle around with filenames & dates until the order is right – but that is of course not practical.

In previous Quarto versions (1.4?), a workaround was to list all files explicitly in _quarto.yml under project / render, but that behavior has changed in the meantime (see #10463, #10490), so the workaround no longer works.

So what else can be done to control the execution order without built-in Quarto functionality?

[allefeld]

@cscheid wrote: "The robust solution is to use a pre-render script to produce all necessary inputs before the files are rendered." But if the inputs are produced by code in qmd files, that would mean calling quarto render in a pre-render script, which seems like asking for trouble.

My current approach is to not have a pre-render script called from Quarto, but a render script calling Quarto, for all qmd files separately in order. So instead of

quarto render

I do

quarto render <file1.qmd>
quarto render <file2.qmd>
…

That seems to be working okay in the cases I tried, but I'm not sure whether it can be relied on to always work. In particular, I'm not sure whether quarto render does something beyond what quarto render <file> for all files does.

Here is a toy project testing this: inc-vs-glo.zip.
The included render.sh does both kinds of rendering (global and incremental) and stores the result in separate subdirectories. Apart from minor differences the two are equivalent.

[allefeld]

No hurry, I justed wanted to have this cleanly in one place. Thanks!

[eculler]

Thanks @allefeld. My workaround for this has been to commit my .ipython directory to GitHub, so that I can manually run notebooks and save the intermediate results, and then everything runs in any order on GitHub Actions. I suppose I could also have a script that runs all the notebooks in order and then renders my website. But it still would be nice to be able to control the rendering order!

I did also try sorting the QUARTO_INPUT_FILES environment variable in a pre-render script, which didn't work for me. However, that may have been because I was re-setting the environment variable with os.environ, rather than running a shell command to export it to the larger context. So, that may be a solution that avoids running quarto render in the script (but I don't have a working example). I've not found it easy to control quarto environment variables in code, because lots of processes get spawned and it's tricky to make them universal. And what I originally assumed was the case, that quarto itself doesn't actually use the environment variable to determine run order, and it's really just a copy, might still be true.

[allefeld]

Thanks @eculler. I'm not sure I understand your first paragraph. How does configuration in .ipython/ affect execution order? Or are you saying you give Quarto ipynb files instead of qmd files, which are already executed in order, and therefore Quarto doesn't execute anything?

Second, interesting idea. But I don't think it can work – as far as I know, exporting a variable makes it accessible to subprocesses, but not to parent processes?

[eculler]

The .ipython doesn't affect the run order. The reason I needed a consistent run order was because I was transferring items from one notebook to another with storemagic. That method lets me cache the intermediate values by running individual notebooks so the run order during render doesn't matter.

[allefeld]

Thanks, I see.