How might I debug quarto's failure to find package 'papermill'?

[dcnorris]

Description

I did notice #4040, but my problem does not resolve so easily.
As you can see, I have indeed installed papermill, and it is on the PATH.
But quarto fails to find it when passing a parameter to a Julia code block.

• Same error message happens with papermill installed under either /home/david/.local/ or /usr/local/.
• I have deleted .julia-cache and tried --no-execute-daemon, to no avail.

How does quarto search for such packages, and how might I proceed to investigate this error?

david@forschung:~/DTAT$ quarto render article.qmd --execute-debug -P draft:false
- Connecting to kernel

Starting julia-1.9 kernel...2023-06-06 12:33:56,727 - root - Level 25 - starting notebook server subprocess
2023-06-06 12:33:56,727 - root - Level 25 - starting notebook server subprocess
2023-06-06 12:33:56,892 - root - Level 25 - running notebook server subprocess
2023-06-06 12:33:56,893 - root - Level 25 - creating notebook server (tcp: /run/user/1000/jt/ecf2ef01077e06de6edc)
2023-06-06 12:33:56,896 - root - Level 25 - notebook server bound to port 42763
- Kernel connection successful
2023-06-06 12:33:57,060 - root - Level 25 - handling server request
- Sending execute command to kernel
- Execute command sent, reading response
2023-06-06 12:33:57,062 - root - Level 25 - executing notebook
2023-06-06 12:33:57,062 - root - Level 25 - inside notebook_execute
2023-06-06 12:33:57,103 - root - Level 25 - read notebook
- Error response received
ERROR: 

The papermill package is required for processing --execute-params
david@forschung:~/DTAT$ which papermill
/usr/local/bin/papermill
david@forschung:~/DTAT$ pip show papermill
Name: papermill
Version: 2.4.0
Summary: Parametrize and run Jupyter and nteract Notebooks
Home-page: https://github.com/nteract/papermill
Author: nteract contributors
Author-email: [email protected]
License: BSD
Location: /usr/local/lib/python3.9/dist-packages
Requires: tqdm, nbformat, pyyaml, ansiwrap, click, requests, nbclient, tenacity, entrypoints
Required-by: 
david@forschung:~/DTAT$ 

[cderv]

Can you run `quarto check jupyter``to see which kernel are found ?

Then you can look at jupyter kernelspec list to see the kernel registered.

You could then open the kernel json file to check which environment will indeed be used (e.g jupyter\kernels\julia-1.9\kernel.json)

Which is the jupyter kernel you are using with Quarto ? It seems you are using with Julia, so you need to check which Jupyter Julia will use.

Having which papermill working in terminal may not be enough if at the end tje Python using with Julia does not have it available.

Hope it helps give hints on what to look for

[dcnorris]

Python (and thus Julia?) tooling/organization baffles me, but this all seems to link up reasonably to unique executables:

david@forschung:~$ quarto check jupyter

[✓] Checking Python 3 installation....OK
      Version: 3.9.2
      Path: /usr/bin/python3
      Jupyter: 4.11.1
      Kernels: julia-1.8, julia-1.9, python3

[✓] Checking Jupyter engine render....OK

david@forschung:~$ jupyter kernelspec list
Available kernels:
  julia-1.8    /home/david/.local/share/jupyter/kernels/julia-1.8
  julia-1.9    /home/david/.local/share/jupyter/kernels/julia-1.9
  python3      /home/david/.local/share/jupyter/kernels/python3
david@forschung:~$ which julia
/home/david/julia-1.9.0/bin/julia
david@forschung:~$ cd .local/bin
david@forschung:~/.local/bin$ ll | grep julia
lrwxrwxrwx 1 david david  33 May 31 17:07 julia -> /home/david/julia-1.9.0/bin/julia
lrwxrwxrwx 1 david david  33 May 31 17:07 julia-1.9 -> /home/david/julia-1.9.0/bin/julia
lrwxrwxrwx 1 david david  33 May 31 17:07 julia-1.9.0 -> /home/david/julia-1.9.0/bin/julia
david@forschung:~/.local/bin$ cd ../share/jupyter/kernels/julia-1.9/
david@forschung:~/.local/share/jupyter/kernels/julia-1.9$ cat kernel.json 
{
  "display_name": "Julia 1.9.0",
  "argv": [
    "/home/david/julia-1.9.0/bin/julia",
    "-i",
    "--color=yes",
    "--project=@.",
    "/home/david/.julia/packages/IJulia/6TIq1/src/kernel.jl",
    "{connection_file}"
  ],
  "language": "julia",
  "env": {},
  "interrupt_mode": "signal"
}
david@forschung:~/.local/share/jupyter/kernels/julia-1.9$ cd ../python3/
david@forschung:~/.local/share/jupyter/kernels/python3$ cat kernel.json 
{
 "argv": [
  "/usr/bin/python3",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3 (ipykernel)",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}

When you say "which environment will be used", does this refer to "env": {}, line of Julia kernel.json?

[cderv]

When you say "which environment will be used", does this refer to "env": {}, line of Julia kernel.json?

No I mean which python environment Julia will use.

When using Julia, you need IJulia (https://quarto.org/docs/computations/julia.html#ijulia). When installing IJulia, Julia will need to use / find a Python installation See more at https://julialang.github.io/IJulia.jl/stable/manual/installation/#Installing-IJulia

Either it will use your system Python, but depending on how and when you did the installation, it could use its own through Conda.jl
As explained in IJulia doc above ENV["JUPYTER"] can control the jupyter to use if not found.

You can open Julia 1.9 and then run

> using IJulia
> IJulia.Jupyter

to check which Jupyter is used I guess.

If this is Conda.jl used, then I guess you need to configure the Python environment from Julia itself. See more here: https://github.com/JuliaPy/Conda.jl#basic-functionality

This is quite complicated world and not directly related to Quarto. I discovered all this with configuration problem, and I am not 100% the above applies and true in all cases. Though I know that I add to reinstall and rebuild IJulia with setting ENV["JUPYTER"] so that the main python installation was picked up. I add to do that with Julia 1.9 too so maybe something changed in Julia world too...

Can you look into the above ?

[dcnorris]

FYI using IJulia needs lowercase 'u', and IJulia.Jupyter code you suggested did not work. The IJulia API offers no means for such introspection AFAICT.

However, following the last bullet under 'General Troubleshooting Tips' on the IJulia Troubleshooting page I did ENV["JUPYTER"]=""; Pkg.build("IJulia") to rebuild IJulia to use Julia's own Conda-based Jupyter, and then did using Conda; Conda.add("papermill") to ensure that installation has a papermill it can find. This did the trick, it seems; the error message has been eliminated.

---

Edit: The parameter passing works, too.

[cderv]

FYI using IJulia needs lowercase 'u',

Thanks I edited !

and IJulia.Jupyter code you suggested did not work.

This probably works when you set ENV["JUPYTER"] to a value. I wasn't sure, thanks for confirming!

This did the trick, it seems; the error message has been eliminated.

Awesome ! Glad it help you !