How to Pass File Paths as Parameters for Batch Processing?

[hasibagen]

Description

Issue Description:

I would like to ask how to pass the file path as a parameter to achieve batch processing of large amounts of data. This is my current code:

quarto render test_par.ipynb --to docx -P sample_path="/home/_2024_05_10_15_44_16.csv"
pandoc

The code seems to run successfully without any errors, but it appears that the parameter is not being passed correctly.

to: docx
output-file: test_par.docx
default-image-extension: png
ascii: false

metadata
title: test_par
mainfont: STZhongsong
monofont: STZhongsong
papersize: a4

Output created: test_par.docx

Here is the relevant portion of my Quarto file and the reproducible demo file:

---
title: "test_par"
mainfont: "STZhongsong"
monofont: "STZhongsong"
format:  docx
params:
  sample_path: 
execute:
  enabled: true
  echo: false
  warning: false
  eval: true
ascii: false
fig-dpi: 300
papersize: a4
---

#| tags: [parameters]
print(sample_path)

test_par.zip

[hasibagen]

I have referred to
#9107
and added
execute:
enabled: true

[hasibagen]

Is this completely impossible?

It seems that this post mentioned that jupter does not support parameter passing?
#5941

But I may prefer to use jupter at present.

[mcanouil]

Please make sure your code is properly formatted and is correct.
You are missing the code cell, you have a YAML key (params) without a value.

Also be sure to read the documentation: https://quarto.org/docs/computations/parameters.html#jupyter

The options papersize or the font options does not apply to docx format, please be sure to read the guide and reference pages of the documentation:

• https://quarto.org/docs/reference/formats/docx.html
• https://quarto.org/docs/output-formats/ms-word.html

---

You can share a self-contained "working" (reproducible) Quarto document using the following syntax, i.e., using more backticks than you have in your document (usually four ````).
See https://quarto.org/bug-reports.html#small-is-beautiful-aim-for-a-single-document-with-10-lines.

If you have multiple files (and if it is absolutely required to have multiple files), please share as a Git repository.

R	Python
````qmd --- title: "Reproducible Quarto Document" format: html engine: knitr --- This is a reproducible Quarto document. {{< lipsum 1 >}} ```{r} x <- c(1, 2, 3, 4, 5) y <- c(1, 4, 9, 16, 25) plot(x, y) ``` {#fig-placeholder} {{< lipsum 1 >}} The end after @fig-placeholder. ````	````qmd --- title: "Reproducible Quarto Document" format: html engine: jupyter --- This is a reproducible Quarto document. {{< lipsum 1 >}} ```{python} import matplotlib.pyplot as plt x = [1, 2, 3, 4, 5] y = [1, 4, 9, 16, 25] plt.plot(x, y) plt.show() ``` {#fig-placeholder} {{< lipsum 1 >}} The end after @fig-placeholder. ````

Additionally and if not already given, please share the output of quarto check within a code blocks (i.e., using three backticks ```txt), see https://quarto.org/bug-reports.html#check.

[hasibagen]

I also find a bug:
if the code is:

---
title: "Reproducible Quarto Document"
format:  docx
params:
  datapath: null
execute:
  enable : true
  echo: false
  warning: false
  eval: true
jupyter: python3
---

```{python}
#| tags: [parameters]
#
```


```{python}
#| label: import
#| tags: [parameters]
print(datapath)
```
# markdown

the result is

quarto render bug_rep.qmd -P datapath:1

Starting python3 kernel...Done

Executing 'bug_rep.quarto_ipynb'
  Cell 1/3: ''.........Done
  Cell 2/3: ''.........Done
  Cell 3/3: 'import'...Done

pandoc
  to: docx
  output-file: bug_rep.docx
  default-image-extension: png

metadata
  title: Reproducible Quarto Document
  jupyter: python3

Output created: bug_rep.docx

but if the code is

---
title: "Reproducible Quarto Document"
format:  docx
params:
  datapath: null
execute:
  enable : true
  echo: false
  warning: false
  eval: true
jupyter: python3
---

```{python}
#| label: import
#| tags: [parameters]
print(datapath)
```

# markdown

it will report:

quarto render bug_rep.qmd -P datapath:1

Executing 'bug_rep.quarto_ipynb'
  Cell 1/2: 'import'...ERROR:

An error occurred while executing the following cell:
------------------
print(datapath)
------------------


---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[4], line 1
----> 1 print(datapath)

NameError: name 'datapath' is not defined

[mcanouil]

Following the documentation:

---
title: "Reproducible Quarto Document"
format: html
engine: jupyter
execute:
  enabled: true
---

```{python}
#| tags: [parameters]
datapath = None
```

```{python}
print(datapath)
```

quarto render index.qmd --execute-param "datapath:weird" --execute 

[hasibagen]

Sorry but, did you look at the documentation regarding parameters and Jupyter? From the example you shared, it does not seem to be the case.

• https://quarto.org/docs/computations/parameters.html#jupyter

params is for knitr, not Jupyter as described in the link above, thus the fact that it does not work.

I read this document carefully, but maybe I didn't understand the meaning.

This document seems to say
For Jupyter, Quarto uses the same syntax for defining parameters as Papermill. To parameterize a document, designate a cell with the tag parameters and provide appropriate default values:

#| tags: [parameters]

alpha = 0.1
ratio = 0.1

quarto render document.ipynb -P alpha:0.2 -P ratio:0.3

So I thought it was also possible to directly transfer parameters to ipynb. I tried

---
title: "Reproducible Quarto Document"
format: docx
execute:
enable : true
echo: false
warning: false
eval: true
jupyter: python3
---
```{python}
#| label: import
#| tags: [parameters]
print(datapath)
```
quarto render document.ipynb -P datapath:0.1

version

[hasibagen]

---
title: "Reproducible Quarto Document"
format: html
engine: jupyter
execute:
  enabled: true
---

```{python}
#| tags: [parameters]
datapath = None

print(datapath)

Does this mean that if I want to use a parameter, I need to initialize it beforehand?
Because code like this will also result in an error.

---
title: "Reproducible Quarto Document"
format: html
engine: jupyter
execute:
  enabled: true
---

```{python}
#| tags: [parameters]
print(datapath)
```

quarto render index.qmd --execute-param "datapath:weird" --execute

Executing 'index.quarto_ipynb'
  Cell 1/2: ''...ERROR:

An error occurred while executing the following cell:
------------------
print(datapath)
------------------


---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[4], line 1
----> 1 print(datapath)

NameError: name 'datapath' is not defined

[mcanouil]

You are still not following the documentation. #| tags: [parameters] is the code cell where you initialise the parameters, not use them.
Look at my example or the documentation as a screenshot below:

[mcanouil]

As long as you don't follow the syntax/documentation, yes it will lead to error as expected.

[hasibagen]

initialise the parameters

Now I understand what you mean, I understand that I need to initialise the parameters before using them.

[hasibagen]

Following the documentation:

---
title: "Reproducible Quarto Document"
format: html
engine: jupyter
execute:
  enabled: true
---

```{python}
#| tags: [parameters]
datapath = None

print(datapath)

quarto render index.qmd --execute-param "datapath:weird" --execute

I would like to ask if it is possible for you to also give a minimal demo of ipynb? Because I seem to have the same problem as in this post
#9107

I have now fully implemented it using qmd, but I still have problems with ipynb.

Thank you very much for your help.

[mcanouil]

Use quarto convert on your file it should give you the correct Jupyter Notebook.

[hasibagen]

Another small problem that I encountered before is that it seems that the init cell will be automatically repeated once, and there cannot be a label name. If there is one, an error will be reported.

---
title: "Reproducible Quarto Document"
format: html
engine: jupyter
execute:
  enabled: true
---

```{python}
#| label: prep
#| tags: [parameters]
datapath = None
```

```{python}
#| tags: [parameters]
print(datapath)
```
quarto render index.qmd -P datapath:4

Starting python3 kernel...Done

Executing 'index.quarto_ipynb'
  Cell 1/3: 'prep'...Done
  Cell 2/3: 'prep'...Done
  Cell 3/3: ''.......Done

ERROR: Cell label names must be unique (found duplicate '#prep')

Stack trace:
    at file:///opt/quarto/bin/quarto.js:30322:23
    at jupyterToMarkdown (file:///opt/quarto/bin/quarto.js:32297:9)
    at eventLoopTick (ext:core/01_core.js:153:7)
    at async Object.execute (file:///opt/quarto/bin/quarto.js:40543:25)
    at async renderExecute (file:///opt/quarto/bin/quarto.js:77997:27)
    at async renderFileInternal (file:///opt/quarto/bin/quarto.js:78165:43)
    at async renderFiles (file:///opt/quarto/bin/quarto.js:78033:17)
    at async render (file:///opt/quarto/bin/quarto.js:82893:21)
    at async Command.actionHandler (file:///opt/quarto/bin/quarto.js:83041:32)
    at async Command.execute (file:///opt/quarto/bin/quarto.js:8017:13)

[mcanouil]

Why do you have #| tags: [parameters] in a code cell that does not have parameters to be initialised?

[mcanouil]

FYI, I can reproduce on the latest development version (I'll open an issue, if I don't find the other one).

---
format: html
engine: jupyter
---

```{python}
#| label: test
#| tags: [parameters]
datapath = None
```

Trigger the crash by sending a new parameter value

quarto render index.qmd --execute-param datapath:"weird" --execute

Executing 'index.quarto_ipynb'
  Cell 1/2: 'test'...Done
  Cell 2/2: 'test'...Done

ERROR: Cell label names must be unique (found duplicate '#test')

Stack trace:
    at file:///Users/mcanouil/Projects/quarto/quarto-cli/src/core/jupyter/labels.ts:53:15
    at jupyterToMarkdown (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/core/jupyter/jupyter.ts:758:5)
    at eventLoopTick (ext:core/01_core.js:153:7)
    at async Object.execute (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/execute/jupyter/jupyter.ts:378:20)
    at async renderExecute (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-files.ts:232:25)
    at async renderFileInternal (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-files.ts:592:35)
    at async renderFiles (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-files.ts:325:9)
    at async render (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/render-shared.ts:102:18)
    at async Command.actionHandler (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/command/render/cmd.ts:248:26)
    at async Command.execute (file:///Users/mcanouil/Projects/quarto/quarto-cli/src/vendor/deno.land/x/[email protected]/command/command.ts:1948:7)

Keeping the default parameter value

quarto render index.qmd --execute

Executing 'index.quarto_ipynb'
  Cell 1/1: 'test'...Done

pandoc 
  to: html
  output-file: index.html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png
  
metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  
Output created: index.html

[mcanouil]

@cscheid Do you recall the issue/discussion about the duplicated code cell? Here it is slightly different as this duplication associated with the label creates a crash.

[cscheid]

I don't. Please just create a new one.

[mcanouil]

• Parameters and jupyter engine crashes when label is set #10097