quarto with poetry: The papermill package is required for processing --execute-params

[bryanwhiting]

I have the papermill package installed in my poetry environment, but it's not working

poetry run quarto render \
    quarto/_template.qmd \
    -P json_file_path:"xxx" \
    -P data_path:"xxx" \
    --to html
ERROR: 

The papermill package is required for processing --execute-params

[bryanwhiting]

Answer: i got this working by installing papermill in my root directory (by deactivating the conda shell I was in or something).

E.g.,:

deactivate
pip3 install papermill

and then execute the command above again and it worked.

Probably just a conflict of environments.

[cderv]

TL;DR: quarto render file.qmd is rendered using a different Python version than quarto render file.ipynb, of course in this case _environment does not apply.

I am not sure I followed everything here but one thing about quarto render

• If you do on a .qmd using Jupyter engine, quarto render will trigger the notebook execution
• If you do on an existing .ipynb, Quarto won't by default execute the cells.

This is mentioned at https://quarto.org/docs/projects/code-execution.html#notebooks

Note that when rendering an .ipynb Quarto will not execute the cells within the notebook by default (the presumption being that you have already executed them while editing the notebook). If you want to execute the cells you can pass the --execute flag to render

This could be a difference if you don't know about this when comparing what happens.

Sharing just in case it could explain

[EmiliaJarochowska]

Thank you and apologies I wasn't clear. I try to explain better. I double-checked and this is still the case.

1. quarto render file.qmd:

• generates file.ipynb as a transitional step in rendering
• stops because a wrong Python installation is used and it lacks dependencies

2. quarto render file.ipynb --execute works: it uses Python version in the virtual environment, as indicated by including which python in the notebook (although I am not sure that executing the shell command necessarily shows which Python was used to execute the notebook).

However, this only occurred when quarto was using Conda.jl Python over the one specified in the venv. Now that I removed the .julia directory, this problem is null as quarto uses Python selected by poetry.

I'll work on the issue as you suggested but I am still trying to find out why my project is identified as a Julia project, both by Quarto and VS Code. I check all directories in the path to the project for stray Julia files but there weren't any so still no clue.

[mcanouil]

What do you mean identified by VSCode as a "Julia project"? There is no such thing in VSCode for Quarto document.

Could it be you did some modification of your VSCode setup?

Did you try Quarto in a Terminal/PowerShell outside of VSCode?

[EmiliaJarochowska]

Oh yes, sorry, please see above for details:

for some reason this entire repo is detected as a Julia environment, by VS Code as well as by Quarto

I always ran quarto from the terminal, but I just noted that both Quarto and VS Code seem to see it as a Julia project, which probably explains why Conda.jl Python was used over poetry settings, as explained by @cderv:

I believe Quarto will default to using Julia bundle python with Conda.jl when julia is used

So principally Quarto behaved correctly, although that would be hard to piece together from documentation. I only continued this dicussion because of the curious fact that different Python seems to have been used to execute .qmd and .ipynb.

My feeling is that Python path of the virtual environment should have priority over other factors (such as Julia being used), at least from the perspective of replicability, I'll make some experiments with a mixed Quarto Julia & Python project for the issue.

[cderv]

My feeling is that Python path of the virtual environment should have priority over other factors (such as Julia being used), at least from the perspective of replicability, I'll make some experiments with a mixed Quarto Julia & Python project for the issue.

Just as a reminder in this thread is now rather long. Did this happens when the virtual env is activated before running quarto render ?

What happens with poetry run quarto render ... ? (https://python-poetry.org/docs/cli/#run)

If you can come with a reproducible example to set up ( maybe using docker, CI space, or even codespaces, DevPod, GitPod, or any other), it would help debug and understand how Quarto does select the python to use and the engine to use. Reminder: Quarto does auto binding, but you can configure the engine to use (https://quarto.org/docs/computations/execution-options.html#engine-binding)

[atwam]

Coming here late, having had the same issue.
Your fix, combined with the discussion in #5833 helped me solve the problem:

• Having poetry installed, with a proper venv where jupyter is installed
• Running jupyter kernelspec list to see my python3 kernel, and then reading the corresponding json /home/.../.venv/share/jupyter/kernels/python3/kernel.json, I got the following content:

{
 "argv": [
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3 (ipykernel)",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}

I think the root of the issue is in the python line: when I launch such a kernel, it will usually pick-up the venv activated version of python. But running quarto render will have it get the executable for the kernel (see exec.ts, which in our case is python.
While that executable will be run by quarto (see jupyter-kernel.ts), it ends up calling Deno.run, which under the hood calls rust's std::process::Command. That should take into account the PATH variable (according to its documentation), but it looks like PATH was modified somehow, or it find python which does not map to the current shell's PATH.

Anyway, changing the kernel.json above to have the full path to the local python shell seems to work. A proper quarto fix would be to query the proper full path of python, by using which python, or allowing the jupyter python executable to be set in the options.

[cderv]

I am not sure I followed all the specific problem here, but know that quarto 1.5 will correctly check for any activated venv to use this python version.

So you should try your original way using latest quarto 1.5 prerelease and verify your activated venv in shell is picked up.

[cscheid]

Anyway, changing the kernel.json above to have the full path to the local python shell seems to work. A proper quarto fix would be to query the proper full path of python, by using which python, or allowing the jupyter python executable to be set in the options.

That's not a proper fix. Quarto uses jupyter. Jupyter reports that kernel, and we are following what the kernel says. The fix is, unfortunately, for Jupyter+Python package managers to properly manage their kernel.json files.

[hasibagen]

I would like to ask how to successfully pass path parameters to quarto's ipynb file. This is my current code.

quarto render test_par.ipynb --to docx -P sample_path="/home/_2024_05_10_15_44_16.csv"
pandoc
to: docx
output-file: test_par.docx
default-image-extension: png
ascii: false

metadata
title: test_par
mainfont: STZhongsong
monofont: STZhongsong
papersize: a4

Output created: test_par.docx

---
title: "test_par"
mainfont: "STZhongsong"
monofont: "STZhongsong"
format: docx
params:
sample_path:
execute:
echo: false
warning: false
eval: true
ascii: false
fig-dpi: 300
papersize: a4
---
#| tags: [parameters]
sample_path = "/home/2024_04_16_16_50_34.csv"
print(sample_path)

It doesn't seem to report an error, but there are no parameters passed in either, because the generated path is still the path I wrote in ipynb

test_par.zip

[mcanouil]

@hasibagen Could you properly format your post using code blocks for code and terminal outputs? Thanks.
If your code contains code blocks, you need to enclose it using more backticks, i.e., usually four ````.

Also, since your question is a bit different, please open a new separate discussion. You can link to this one if appropriate.
Thank you!

[hasibagen]

@hasibagen Could you properly format your post using code blocks for code and terminal outputs? Thanks. If your code contains code blocks, you need to enclose it using more backticks, i.e., usually four ````.

Also, since your question is a bit different, please open a new separate discussion. You can link to this one if appropriate. Thank you!

Thank you very much for your suggestion, I have modified the code and raised a new discussion.

#10081 (comment)