How to efficiently break up a computationally heavy article into separate notebooks?

[DayalStrub]

Description

Context

I am trying to use (the amazing!) quarto to write an article that has a lot of slow computations. I would like to break the article up into notebooks and use the caching option with the _freeze folder and files, but can't find a solution to "embed" (or similar) both outputs and variables (for inline use), or to split the article text and code into separate parts.

Manuscript approach

The documentation for manuscripts shows how to

• embed the output of a notebook cell into the index.qmd file, but I cannot use this approach to pass variables from a secondary notebook, say analysis.qmd, to the index.qmd for inline printing, e.g. if analysis.qmd defines n_obs and i want to write {python} n_obs inline in index.qmd.
• include the whole secondary notebook into the index.qmd notebook, which is nice because one can have text and computations side by side, but still split the article into multiple notebooks. However, this doesn't work with _freeze, as it essentially copy-pastes secondary notebook into index.qmd and everything is rerun at every quarto render.

Book approach

This is something I haven't tried yet, but I was wondering whether the book project type might be an option here (if one is happy without the manuscript website, etc.). My thinking is that one could split up the article into sections, and use the book chapters config option, and then some custom pdf documentclass format to avoid the new pages per chapter in books, and also make it render like an article. [EDIT: see comment below with link to mock up in github.]

[Side note: it might be nice to split books/articles at different levels depending on how large a chapter/section is, i.e. have the option to split into chapters, but also (sub-)sections if needed. I don't see from the docs that this is possible.]

Question

• is there a suggested/best approach to this?
• are there any examples online/on GitHub of computationally heavy articles, and people splitting them up?

[mcanouil]

I am not sure what you want to do exactly.
Splitting in multiple files won't reduce the "computationally heavy" aspect, even using cache/freeze.

In the R/knitr ecosystem, I would personnaly use targets for computations, and include the results in the Quarto document.

In Jupyter, I don't know if there is a tool like the targets R package

[Side note: it might be nice to split books/articles at different levels depending on how large a chapter/section is, i.e. have the option to split into chapters, but also (sub-)sections if needed. I don't see from the docs that this is possible.]

Use Quarto profile to change what parts/chapters are to be rendered.

Could you share a small setup of what you have right now? Preferably as a Git repository.

[DayalStrub]

sorry for the late reply! wanted to put together a simple example, but took a while.

Splitting in multiple files won't reduce the "computationally heavy" aspect, even using cache/freeze.

I thought that was the whole point of _freeze/, e.g. when set to auto then the notebooks are only re-rendered when the source changes, so "computationally heavy" notebooks that aren't being worked on won't be re-rendered.

i do really like your idea of a workflow orchestrator, like targets. there's a few in python. will have a think and see if there is something well suited/similar to targets. [though wouldn't really solve my issue, rather make my workflow better.]

i have put a tiny example repo together, using the manuscript template. i have put the introduction section into _introduction.qmd and am then includeing it in index.qmd. this is supposed to be an example of "splitting out computations", but keeping code and text together (which feels better).

i have also used inline code, and am printing a variable in the text, see this line. unlike embeding figures or tables, i'm not able to pass variables between notebooks. that means that even if i was happy to split computations and text, as done in notebooks/data-screening.ipynb from which a figure is embedded, i wouldn't be able to call the variable (avg_years_between_eruptions) from _introduction.qmd if the text (and inline code) was in index.qmd. note: this is the only "blocker" to splitting out the code (if one accepts to not have code and text together).

i'm not too thrilled by using include in this way, as it makes for very slow renderings. also, freeze doesn't seem to work well with include.

i might be completely wrong/missing something, but what i think would be great would be if manuscript projects had the concept of "sections" a bit like chapters for book projects, and then being able to set them in _quarto.yaml. so in this example, it might look like

...
sections: # cf chapters
    - index.qmd
    - _introduction.qmd
    - ...
    - references.qmd
...

but assuming i'm not the first to want to write a "big" article, wanting to split things, and wanting to use inline code. so was keen to hear whether others had smart set-ups i hadn't thought of.

hope this makes a bit more sense.

[DayalStrub]

have played about with using the book project and tweaking to make it look more manuscript/article like. see github repo. not perfect, but shows the workflow of splitting a manuscript into sections, and might be a good solution for me to work on paper with collaborators (especially as preview and _freeze seem to be working nicely). i will have to tinker with latex or migrate to manuscript once it's ready to publish.