how to set the python kernel for quarto in Nix context

[biscotty666]

Description

I'm having trouble with quarto running python code. Specifically, it is not finding the correct kernel. I can import my environment modules in a qmd document in rstudio without a problem, yet when I run quarto preview, it says that it cannot find the modules. So it is using a different kernel.

The twist here is that I use nix, and so I don't use a typical venv environment, and there is no /usr/bin/python on the system. Rstudio sees the kernel, and things work as expected, but for some reason quarto does not.

Any help would be much appreciated. In general, I will be using pre-rendered ipynb documents, but it would be nice to get this working.

[mcanouil]

Please read the documentation before asking questions. Thank you.

https://quarto.org/docs/computations/python.html#kernel-selection

Additionally, there are other discussions about kernel discovery issue.

Finally, the troubleshooting guide: https://quarto.org/docs/troubleshooting/#advanced

---

The twist here is that I use nix,

The Quarto CLI packaged by Nix doesn't use the proper dependencies. There is no guarantees it works properly and as designed/expected by the Quarto team.
Unless proven the issue is Quarto CLI and not the wrongly packaged version provided for Nix users, there is no support.

[biscotty666]

quarto check

[✓] Checking Python 3 installation....OK
      Version: 3.12.9
      Path: /nix/store/d03530jzrd6i26593w5v82iv6gbra8qn-python3-3.12.9-env/bin/python3.12
      Jupyter: 5.7.2
      Kernels: haskell, python3

which is the wrong kernel, as I knew. This is what I need to change. As to those other sections, I'm not using jupyter, and they all seem jupyter-specific. Point three is not a terminal command, at least not one the works for me, and everything in the advanced troubleshooting seems to assume I do, or at least I cannot translate it to a non-jupyter context. I'm not trying to be difficult. If I just knew how quarto selects it's kernel.

[mcanouil]

What do you mean "wrong kernel"?
The output shows you the different kernels that Quarto sees. It sees two kernels and only two.

Meaning there is an issue with your setup, if there are more.
If it's in an environment, you need to activate the environment first.
Did you install your kernel in the activated environment? nix/store/d03530jzrd6i26593w5v82iv6gbra8qn-python3-3.12.9-env/bin/python3.12

I think you are mistaken. You are using Jupyter with a specific kernel.
Jupyter is not Jupyter Lab, it's a tool to work with Jupyter Notebook.
There is no way you can use a Jupyter kernel without using Jupyter.

[biscotty666]

❯ which python
/nix/store/2f2ymldccm86nqxlyhv76f5kjfy1h265-python3-3.12.9-env/bin/python

is the right kernel, the one that quarto sees is the wrong one.

The nix store contains many python versions, one for each development shell I have on my system. That's the way it is set up. It's not like FHS.

To elaborate, through environment variables, each dev shell selects the correct kernel. Perhaps I could set a Jupyter environment variable, though...hmmm

[mcanouil]

I don't know how your environments are activated but I believe the issue is there.

You could use QUARTO_PYTHON environment variable to force Quarto to use whatever version you want.
This is also documented on the same page by the way: https://quarto.org/docs/computations/python.html#installation

And check your PATH. I bet you have multiple paths to Python binary in it (which is bad).

Quarto will select a version of Python using the Python Launcher on Windows or system PATH on MacOS and Linux. You can override the version of Python used by Quarto by setting the QUARTO_PYTHON environment variable.

[biscotty666]

QUARTO_PYTHON

was indeed the answer I needed. Thank you.

[cderv]

@biscotty666 I believe Quarto will look for python3 binary. So this one
/nix/store/2f2ymldccm86nqxlyhv76f5kjfy1h265-python3-3.12.9-env/bin/python won't be found by default by Quarton and this one is /nix/store/d03530jzrd6i26593w5v82iv6gbra8qn-python3-3.12.9-env/bin/python3.12 possibly because your python3 found in PATH points to it. Check which python3

QUARTO_PYTHON is a good solution if you want manual configuration. If you want automatic detection

• Either use venv for Quarto Project
• Or be sure to get the right python3 binary found in PATH.

So for your current situation, I would check your PATH and which python versions are available.

[biscotty666]

I appreciate you concern, which is certainly important on most Linux systems. Nix is different, though, and setting the environment variable is the best solution for my system, the one I was looking for initially.

I don't actually have a "base Python" on my system. If I open a terminal and type which python, it says not found. I don't have R or rstudio installed either. Each project has it's own version of all the packages necessary, they live in /nix/store/... and are available when working in the project's directories, but not outside of them. This is why venvs and docker-style containers are not necessary on a nix-enabled system. When entering the shell (loaded automatically when entering the project's directory), a whole mess of variables are set to point to the right targets in /nix/store. In this case, all I needed to do was to add the environment variable in the file that defines the development shell (environment variables are unset when leaving the directories). But they all live in nix/store, and my hunch is that Quarto just chose the first one it found.

[cderv]

Oh good to know ! Thanks for explaining !

[biscotty666]

I guess this problem isn't quite solved. I am explicitly setting $QUARTO_PYTHON to the python kernel I want:

❯ echo $QUARTO_PYTHON
/nix/store/6dr540l6c865w9y2bhx2hczbkgb3mzvl-python3-3.12.10-env/bin/python

Which matches the kernel I'm using

❯ which python
/nix/store/6dr540l6c865w9y2bhx2hczbkgb3mzvl-python3-3.12.10-env/bin/python

I am trying to run the sample code from the Quarto dashboard documentation https://quarto.org/docs/dashboards/,

I can execute import plotly.express as px in the qmd document using R Studio, and I can import it in a repl,. I can knit a Rmd file with the same import, it works fine as well. Basically, everything works as expected with the exception of quarto render, which gives:

Starting python3 kernel...Done

Executing 'gapminder.quarto_ipynb'
  Cell 1/4: ''...ERROR:

An error occurred while executing the following cell:
------------------
import plotly.express as px
df = px.data.gapminder()
------------------


---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[1], line 1
----> 1 import plotly.express as px
      2 df = px.data.gapminder()

ModuleNotFoundError: No module named 'plotly'

So it seems that the render command is not using the same python kernel as everything else. Is there another environment variable I need to set?

I ran quarto check and it said:

[✓] Checking Python 3 installation....OK
      Version: 3.12.10
      Path: /nix/store/6yj63zgm59syif21pzd2qxv7kwh2igq0-python3-3.12.10-env/bin/python3.12
      Jupyter: 5.7.2
      Kernels: haskell, python3

Which is the wrong Python, yet

❯ echo $QUARTO_PYTHON
/nix/store/6dr540l6c865w9y2bhx2hczbkgb3mzvl-python3-3.12.10-env/bin/python

[cderv]

Can you tell me what flag to use?

Look at the help: It is --log

❯ quarto check --help

Usage:   quarto check [target]
Version: 99.9.9

Description:

  Verify correct functioning of Quarto installation.

  Check specific functionality with argument install, jupyter, knitr, or all.

Options:

  -h, --help              - Show this help.
  --output      <path>    - Output as JSON to a file
  --no-strict             - When set, will not fail if dependency versions don't match precisely
  --log         <file>    - Path to log file
  --log-level   <level>   - Log level (info, warning, error, critical)
  --log-format  <format>  - Log format (plain, json-stream)
  --quiet                 - Suppress console output.
  --profile               - Active project profile(s)

I'll add mention of this in https://quarto.org/docs/troubleshooting/#verbose-mode (with QUARTO_LOG being also an option to set the path of the log)

[allumik]

Sorry for the delay, for some reason building nixpkgs failed for me so I wrote a overlay for the preFixup build phase of the quarto package. This fixed the problem for forced system python binaries for the quarto executable from within conda-shell, but I am interested if these changes also work for @biscotty666. If it does, I will make a pull request for https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/quarto/default.nix and the discussion should be moved there.

The current overlay in my home-manager package list overriding the --set with --prefix for Python and R binaries:

with pkgs;
let

# other stuff

  quartoRenvForCustom = rWrapper.override {
    packages = [ rPackages.rmarkdown ];
  };
  # 'python312' is assumed to be your specific Python environment (e.g., pkgs.python312)
  quartoPythonEnvForCustom = python312.withPackages (ps: with ps;
    [ jupyter ipython ]
  );

  # --- Define your custom Quarto package with an overridden preFixup phase ---
  quarto_custom = quarto.overrideAttrs (oldAttrs: {
    # Now, replace the preFixup script.
    preFixup = ''
      wrapProgram $out/bin/quarto \
        --set QUARTO_DENO ${lib.getExe deno} \
        --set QUARTO_PANDOC ${lib.getExe pandoc} \
        --set QUARTO_ESBUILD ${lib.getExe esbuild} \
        --set QUARTO_DART_SASS ${lib.getExe dart-sass} \
        --set QUARTO_TYPST ${lib.getExe typst} \
        ${lib.optionalString (rWrapper != null) "--prefix QUARTO_R : ${quartoRenvForCustom}/bin/R"} \
        ${lib.optionalString (python312 != null) "--prefix QUARTO_PYTHON : ${quartoPythonEnvForCustom}/bin/python3"}
    '';
  });

# other user packages defined here, eventually imported in home.nix 

[allumik]

Note that for me, this still required the custom location for QUARTO_PYTHON env var defined before quarto, but now it isn't overwritten by the system binary anymore.

[allumik]

Opened NixOS/nixpkgs#416671, maybe future discussion should be moved there as this issue seems to originate from NixOS side.

[biscotty666]

I'm sorry I haven't had time to test this. I'm slammed right now with other things, but as soon as I can, I will. Thanks