One or more notebooks embedded in page

[matthew-brett]

This is just to tell you about a use-case that we have, that might be of interest.

Summary

We want to separate the idea of a 'page' and a 'notebook', so that a page can contain one or more notebooks.

Detail

We (@stefanv and I) are writing a new edition of a book on probability and statistics.

For the book format, it makes most sense to us to have chapters, and pages, that are relatively long. These chapters will have several examples. Often we want to have separate notebooks for the students to interact with for each example.

This in turn means we would like to change from the standard "Download / run this page as a notebook" idiom, that works really well in JupyterBook - to an idiom where, at any point in a page, there is a 'Download / run this part of the page as a notebook'.

At the moment we are doing that with an awful hack, where we use macros like r begin_notebook('first_example') and r end_notebook() to drop comments into the built .Rmd page, and then post-process to extract the notebooks. But this is just to wonder out loud whether there is a better way to do this - and whether this is an interesting way of thinking about the relationship of pages to notebooks.

[jjallaire]

We are contemplating creating an {{< include >}} shortcode that would operate on full documents -- with this I think you could create the notebooks and then use e.g. {{< include notebook.ipynb >}} to stitch them together into a chapter.

[matthew-brett]

Yes - that would be one way of doing it - but we really like the fact that the notebooks are integrated into the text, as we are working on them. I mean, that it would make the workflow less fluid, to have to stop editing in one page, then go to another to continue the train of thought, of editing, then come back to the original.

[jjallaire]

Another thing you could do which still requires post-processing but would be much cleaner is to use a combination of:

1. The keep-ipynb option to preserve the ipynb notebook generated from the qmd
2. Cell metadata (any custom yaml cell options are preserved as cell metadata) to group together cells.

Then at least you'd be post-processing an ipynb in JSON format as opposed to HTML.

[matthew-brett]

Please forgive the longer explanation as to why .Rmd and post-processing works pretty well for us, with the possibility that a) we've misunderstood something, and we could do better and b) in case it's useful for thinking about this use-case - that we think is going to become more common in interactive textbooks.

The approach that we took is to write all our pages, and therefore, all our notebok content, in .Rmd format - for a few reasons:

• This works very well for integration with Jupyter via Jupytext. In fact, as a heavy Python and Jupyter user, I nearly always do substantial editing of Jupyter notebooks in .Rmd format, and usually do not save the generated .ipynb files in version control. That's what I'd recommend as standard practice for working with Jupyter notebooks, should anyone ask.
• The standard .ipynb format cannot support the `r do_something()` or `r some_var` extensions that we use all the time. Myst has some equivalents, but they are considerably less friendly.
• In our particular case, the book will have parallel R and Python editions, so it's convenient to start in .Rmd format for the R notebooks.

For the post-processing, we use Jupytext to parse the generated GFM into notebook cells, then work on that, before writing to .Rmd for R or .ipynb for Python - see https://github.com/resampling-stats/resampling-with/blob/master/scripts/build_notebooks.py.

[jjallaire]

Okay, that makes perfect sense to me. I would say that your workflows seems quite well thought through and well suited to your requirements.

[matthew-brett]

On further reflection, we switched to a workflow that generates the notebooks via a Pandoc filter. Our pages have markup like:

::: {#notebook name="my_notebook"}

This will be a notebook (as well as text in the page).

```{r}
print('A message')
```

:::

The filter writes out the parsed div as a notebook in RMarkdown or Jupyter .ipynb format, according to a flag in the metadata, and inserts links to download the notebook and to interact with it, via Binder.

[jjallaire]

That seems like very good approach! one question: is #notebook meant to be .notebook (i.e. can you support more than one notebook in a chapter/page?).

[matthew-brett]

Aha! Thanks - I hadn't registered that we could have only one Div with that name. Will change to .notebook.

[matthew-brett]

I refactored the filter out into a pip-installable module at noteout.

[jjallaire]

That's fantastic!

I noticed you needed to order the quarto filters first -- I am not sure what the default should be here (user filters before or after quarto, I can see both sides). Do you have an opinion about that?

[matthew-brett]

I only did that because I wanted the option to get cross-references resolved in the notebook markdown - but I haven't tried to post-process those yet, and so I don't claim any great insight as to which to prefer in general.

[mcanouil]

I believe the "Embedding Jupyter Notebook Cells" (https://quarto.org/docs/authoring/notebook-embed.html) resolves this.