Using virtual environments with quarto

[Walser52]

Description

I'm having a hard time figuring this out so please be gentle :). Essentially I am having trouble using virtual environments.

1. I created a website project in VSCode. Did a quarto check jupyter. Got:

[✓] Checking Python 3 installation....OK
      Version: 3.9.13 (Conda)
      Path: /home/fes33/anaconda3/bin/python
      Jupyter: 4.11.1
      Kernels: julia-1.9, python3

2. Created a venv by following the documentation i.e.
   python3 -m venv env
   source env/bin/activate
   python3 -m pip install jupyter matplotlib pandas

3. Terminal prompt gets an (env) label. Doing quarto check jupyter again changes the path

[✓] Checking Python 3 installation....OK
      Version: 3.9.13
      Path: /home/fes33/Website/env/bin/python3
      Jupyter: 5.5.0
      Kernels: python3, julia-1.9

[✓] Checking Jupyter engine render....OK

4. Going to index.qmd, I do a simple:

---
title: "Website"
---

```{python}
import pandas
```

5. This lapses the venv i.e. the (env) before prompt vanishes and I get the message:

ModuleNotFoundError: No module named 'pandas'

There is an unactivated Python environment in env. Did you forget to activate it?

My Quarto version is 1.3.433

Would appreciate any help on how to diagnose it. Ultimately I want to bind a virtual environment seamlessly to a project.

[mcanouil]

Thanks for using Quarto.
Could you edit the title of your post?

There is an unactivated Python environment in env. Did you forget to activate it?

Are you sure it is activated? (how familiar are you with Python and virtual environment?)

My Quarto version is 4.3.3

This is not a Quarto version.

What's the output of quarto check inside your project?
Are you using an IDE and which?
Did you run quarto in the same terminal where the virtual environment was activated?

[Walser52]

I think it does get activated because my prompt changes:

Secondly when I do a quarto check jupyter I get the path that corresponds to ./env/bin/python3.

BUT now when I press preview, I get the error that pandas isn't available and there is an unactivated python environment in env. Moreover the prompt goes back to:

And quarto check now yields:

[✓] Checking Python 3 installation....OK
      Version: 3.10.12
      Path: /bin/python3
      Jupyter: 5.5.0
      Kernels: julia-1.9, python3

[✓] Checking Jupyter engine render....OK

This /bin/python3 is different from the anaconda python that quarto check returned originally. I've also tried setting the environment variable QUARTO_PYTHON but doesn't seem to work.

Sorry: the quarto version is 1.3.433. Long day. And I'm using VS Code. And yes all this is happening in the same terminal in VS Code.

[mcanouil]

It might be some settings of yours in VSCode such as settings Python path globally for some extension, etc..
What if you use only terminal to execute quarto instead of VSCode?

Please provide the entirety of quarto check.

QUARTO_PYTHON did you set it globally or in one terminal session? if set in a session, by default it will only affect this session, meaning this does not change anything elsewhere including using the Preview button.
You could try to set in in _environment at the root of your project as described in the documentation (https://quarto.org/docs/projects/environment.html).

[adivekar-utexas]

FYI, I fixed the issue of (env) (base) by referring to this thread: microsoft/vscode-python#20885 (comment)

Command to run:

conda config --set auto_activate_base False

[mcanouil]

Thanks for sharing!

[danieltomasz]

Hi @Walser52
Quarto can specify jupyter kernel
https://quarto.org/docs/computations/python.html
for example

---
title: "matplotlib demo"
format:
  html:
    code-fold: true
jupyter: python3
---

in your case you created an virtual env env, you might try to explicitly set jupyter: env, I usually specify explicitly the jupyter kernel, that is less error prone
But what I understand you didnt create an jupyter kernel within env virtual environment,, I am not developer, but I was always convinced that quarto doesnt care about virtual envs, only the kernels you created (the default one is python3 when you install jupyter I think),is the same as with jupyter notebook, in the doc page is more over this topic https://quarto.org/docs/computations/python.html#kernel-selection

The Jupyter kernel used by Quarto is determined using the jupyter metadata option. For example, to use the Xeus Python kernel, do this:

as per multiple kernels https://ipython.readthedocs.io/en/stable/install/kernel_install.html#kernels-for-different-environments

If you want to have multiple IPython kernels for different virtualenvs or conda environments, you will need to specify unique names for the kernelspecs.

in your MVE try python3 -m ipykernel install --user --name env from within your env, the add metadata option in yaml

[mcanouil]

@danieltomasz this isn't explicit in contrary you implied with your initial example that "env" was a kernel name or a path which in both cases is wrong. This was just a side note as this might confuse future readers.

[danieltomasz]

@mcanouil this was explicit, my whole point was about that, but maybe I put it in wrong order, I noticed later that @Walser52 didn't explictly set up/named the kernel in his virtual env (caled env which is confusing, but I didnt first choose this convention),. I didnt edit my whole response. I only at the end on my response asked him again for python3 -m ipykernel install --user --name env and put this into his quarto notebook metadata (first referening to 2 relevant pages)

[mcanouil]

FYI the confusion part is:

in your case you created an virtual env env, you might try to explicitly set jupyter: env, I usually specify explicitly the jupyter kernel, that is less error prone

I was always convinced that quarto doesnt care about virtual envs

It does, it looks for that if it's activated first.
it just have fallback methods.

In a directory with an activated virtual environment:

> jupyter kernelspec list
Available kernels:
  julia-1.9    /Users/mcanouil/Library/Jupyter/kernels/julia-1.9
  python3      /Users/mcanouil/Projects/misc/quarto-playground/.venv/share/jupyter/kernels/python3

In another place:

> jupyter kernelspec list
Available kernels:
  python3      /opt/homebrew/lib/python3.11/site-packages/ipykernel/resources
  julia-1.9    /Users/mcanouil/Library/Jupyter/kernels/julia-1.9

In both cases you'll set jupyter: python3 with python3 the name of the kernel known by Jupyter. The path is taken from this.

[adivekar-utexas]

Okay for anyone still facing this issue, try:

1. Disable the auto-conda activate on creating a new terminal: conda config --set auto_activate_base False
2. From a terminal (any terminal) activate your ./env folder: conda activate ./env
3. Run pip install ipykernel in case you don't have it in your conda env.
4. Create a new global Jupyter kernel which links to ./env, but give it a descriptive name: python3 -m ipykernel install --user --name quarto_my_env
5. In _quarto.yml, set the key: jupyter: quarto_my_env

After this, my VSCode finally is able to render the Quarto book as expected.

[adivekar-utexas]

Another debugging trick (if you're still facing issues) is to create a Quarto cell to run:

import os, sys 
print(sys.path)
print(os.getcwd())

This will print the path and current folder while executing the quarto file.

[cderv]

[✓] Checking Python 3 installation....OK
Version: 3.9.13 (Conda)

I am noticing your are using Conda here. Why not follow the conda version of the virtualenv documentation ?
https://quarto.org/docs/projects/virtual-environments.html#using-conda

I wonder if we could haven't a specific implementation in Quarto that conflicts with using both conda and not conda. 🤔

To check what python version is really used in the chunk of you document, you could render a document with your env activated and the following content so that you can see which python path is used really in a chunk. This leverage how venv works

---
title: "Website"
---

```{python}
import sys
sys.prefix
```

It should be the path of you venv env inside your project. When I do try to reproduce your step, this is what I get

cd /tmp/
mkdir test-quarto
cd test-quarto/
python3 -m venv env
source env/bin/activate
python3 -m pip install jupyter matplotlib pandas

Then I created the test.qmd above and got after quarto render this path /tmp/test-quarto/env for sys.prefix which I believe is the right Python.

So I believe Quarto does what it is supposed to do... unless there is a edge case because of some mixed configuration. hence my question about Conda in the first place.

Also, please confirm this is all happening inside Terminal only - if some of your feedback are happening in VSCODE only for example, this could be different as the IDE itself could modify the python interpreter used.

Hope it helps you help us. Thanks.

[Walser52]

I saw the conda env part. Frankly I didn't even know you could do that in conda till I saw the documentation. However I've been moving away from conda since it sometimes takes ages to solve the environment.

All this happened in the VS Code terminal. If it happens again I'll try the R Studio or bash.

Thank you, you've been most kind. For all the little headaches, using Quarto has been a most enjoyable and educating experience.