render with parameters change source ipynb file

[kompre]

Description

I'm playing with parameters and have noticed an somewhat inconsistent behavior when using a ipynb file instead of a qmd one.

• when rendering a qmd file with parameters, the source qmd file does not change
• when rendering an ipynb file with parameters (and the --execute options), the source ipynb file change, because there it is added the cell with the injected parameters.

Here a simple example:

1. the basic qmd file:

---
title: "fun_with_parameters"
---

## Fun with parameters

```{python}
#| tags: 
#|   - parameters

a = 1
b = 1
```


the parameters are:

- `{python} f"a : {a}"`
- `{python} f"b : {b}"`

2. convert this file into ipynb and change the filename (so it does not get overwritten when rendering the qmd file)
3. execute the shell command quarto render -P a:23 -P b:34 --execute

both document will render with the given parameter values, but:

• source qmd file is unchanged
• source ipynb is modified with the injection, and following execution, of the parameters

I'm using vscode and the change in the ipynb source file could be "sneaky", meaning that if I have the file open in the editor because I'm likely working on it with some unsaved changes, then the editor will not update to the most updated file, but preserve the current view, and warns me of the version mismatch when I'm trying to save again.

If instead I have open the injected version of the ipynb, in order to restore to the previous state I would have to delete the injected code cell, then rerun the notebook.

I understand that this may be the intended behavior, but I would prefer for the source file to remain unchanged, otherwise I could have specified directly in the code cell those parameters and compute the notebook with less back and forth.

Could this maybe an option to have an intermediate file where the changes are applied for rendering? I think something similar already happen for the inline python code, that is evaluated for the output document, but the code is not changed to the evaluated value in the source file.

Why I'm doing this

I'm doing my authoring mostly with ipynb because I've lots of code cell, so using a jupyter notebook is useful for fast preview while still editing. Among these code cell, I'm generating lot of equations (sympy), but the format that is quarto friendly and let me to label the equation is not fully compatible with KaTeX, i.e. the latex engine that render equations in the jupyter notebook (really is the \label command that gives the most trouble). So being able to use parameters at rendering time would let me set a default parameter in my code so that equation are rendered correctly by KaTeX during editing in the notebook, and then set another value when rendering with quarto so that I can still cross-reference the equation

[cscheid]

(and the --execute options)

The execute option in a .ipynb will change the ipynb in a number of other ways that are consistent with the Jupyter model. Specifically, we store the results of the execution in that .ipynb. It would be inconsistent (and impossible to later use jupyter lab or jupyter notebook) if we were to not include the parameters as well.

This behavior is designed to mimic the workflows that Jupyter users are accustomed to, and is working as intended.

[cscheid]

Minimal example:

$ cat test.ipynb
{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "391cf7f7",
   "metadata": {},
   "source": [
    "## Hello\n",
    "\n",
    "This is some text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "223fc8cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"And this is some code\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
$ quarto render test.ipynb --execute

Starting python3 kernel...Done

Executing 'test.ipynb'
  Cell 1/1: ''...Done

pandoc
  to: html
  output-file: test.html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png

metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  title: Hello

Output created: test.html
$ cat test.ipynb
{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "391cf7f7",
   "metadata": {},
   "source": [
    "## Hello\n",
    "\n",
    "This is some text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "223fc8cd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "And this is some code\n"
     ]
    }
   ],
   "source": [
    "print(\"And this is some code\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}

Note the new outputs entry:

   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "And this is some code\n"
     ]
    }
   ],

[cscheid]

If you need to maintain the source ipynb, you could consider quarto convert to turn your .ipynb into a .qmd, and then work from there.

[kompre]

If I'm not mistaken, the --execute option is necessary when rendering a ipynb with parameters because otherwise the -P option by itself would do nothing to modify the output.

this would nothing:

quarto render file.ipynb -P a:23 -P b:34

parameters would not be injected into the file.ipynb and therefore the notebook would not be executed with the given parameters, rendering the current state of the notebook.

Maybe it would be nice to have a --execute-copy that would create an intermediate file where the changes happens. For example in the _manuscript folder I spy some *.out.ipynb that are copies of the source notebooks with the inline codes executed and rendered. Those copies then are "published" with the manuscript, not the source file itself.

If you need to maintain the source ipynb, you could consider quarto convert to turn your .ipynb into a .qmd, and then work from there.

I guess I could setup a pre-render step to convert the ipynb to qmd and then render just the qmd with paramaters (maybe a book type, so I can specify which files needs to be rendered). It's a little bit more overhead, but could work; needs to play with that.