Using (multiple) virtual environments with R and Python

[ccamara]

Description

Note: This is related, but not identical to #1548, #497 and I have read this page: https://quarto.org/docs/projects/virtual-environments.html

TLDR: I want to create a blog where I will be using R AND Python and for some posts, I would want to use specific virtual environments combined with freeze or cache. renvaddresses some needs, conda addresses others, but I can't address them all.

Detailed context
I want to create a blog where I will be using R AND Python and I want to use virtual environment(s) for dependencies as well as freezing to reduce rendering times. I know I can use different virtual environments, such as renv or conda. I have used them in the past with quarto, but in projects where I either used R OR Python respectively.

I have several specific issues I'd like to address:

1. I need to use a virtual environment for both programming languages
2. I am creating a blog, this will contain posts that will eventually be outdated. I do not want to check if the code from an old post is working after updating versions nor I want to update the code if it fails.
3. Some packages may be used in a single blog post only, so I wouldn't want to keep them in the project's virtual environment

In relation to 1:

I know conda can also install some r packages (amongst binaries, for example, quarto!), but I'm afraid that to my knowledge, not every R package is available on conda and sometimes not every version is there, too.

I recently discovered that I can use renv for python packages too (details here: https://rstudio.github.io/renv/articles/python.html and https://rstudio.github.io/renv/articles/python.html), which would be a solution for that, and probably a better fit than conda. Regretfully, it doesn't address items 2 and 3 (see below).

In relation to 2 and 3:

I was considering using virtual environments within the blog post (like suggested here https://stackoverflow.com/questions/74451630/multiple-python-venv-in-a-single-quarto-website) and enable freeze.

The (quite manual -and therefore, perfectible) workflow would be:

1. locate the environment within the post's folder
2. activate the post's environment
3. render the post, so the rendered version is cached in _freeze/ folder
4. activate the project's environment
5. render the blog

As far as I know, this would not work with renv, because there can only be one renv per project. That means that it would only work if I use conda environments, but as seen before, may not be good for some packages.

So, my question is: is there any way/workflow to address the 3 outlined issues above?

[mcanouil]

Few clarifications:

• renv is not a virtual environment or an environment in the same sense as Conda or VirtualEnv.
• if you want to use R and Python in the same Quarto document, it will go through one engine and only one:
  • via knitr (and renv setup which can be setup/linked to a Conda/VirtualEnv to be used using reticulate.
  • via Jupyter and an activated Conda/VirtualEnv in which you have a library such as rpy2.

Far as I know, there is nothing forbidding you to setup renv and/or Conda/VirtualEnv on a per Quarto document (blog post) basis¹, i.e., within the directory in which there is the qmd file with the code to be evaluated/executed.
Did you try that? As it should work.

Footnotes

1. maybe a constraint that comes with your workflow and/or IDE. ↩

[ccamara]

Hi thanks for your quick reply, @mcanouil

I'm not sure about the internal differences between renvand conda or VirtualEnv. I am refering to it as virtual environment in the sense that it handles dependencies and in the same way it is mentioned here: https://quarto.org/docs/projects/virtual-environments.html . Therefore, I'm afraid I may be missing something that you deem to be important and I fail to see. Could you elaborate?

Thanks for explaining how the different engines work with different virtual environment solutions and how do they deal with "the other" programming language (via reticulate or rpy2).

As for having multiple renv within the same project: I didn't know that was even possible. How am I supposed to make it work? As far as I know, RStudio configures renv as per project basis. You can either activate or deactivate renv completely, without the ability of selecting a particular one. I couldn't find any mention to it on https://rstudio.github.io/renv . Also, according to https://quarto.org/docs/projects/virtual-environments.html#renv Rstudio would be able to detect the presence of a renv in the root folder, but there's no mention to other setups (I understand that using renv would not work in other IDES?). Could you please detail how should I do that?

[cderv]

As for having multiple renv within the same project: I didn't know that was even possible. How am I supposed to make it work?

This is not something really common to have one library per post. Usually we have project library. Though renv has a specific feature with embed() and use(): https://rstudio.github.io/renv/reference/embed.html

Theoretically, you embed some package information (equivalent to lockfile) inside a chunk in the document. And when the script is render, it will

1. Create a temporary library path.
2. Install the requested packages and their recursive dependencies into that library.
3. Activate the library, so it's used for the rest of the script.

So it seems this would allow to have one package environment per document.

I never really use it in such blog project. Though there was discussion about renv and blogdown in the past. This could be interesting maybe:

• Use each blog post's renv environment when rendering rstudio/blogdown#469

this will contain posts that will eventually be outdated. I do not want to check if the code from an old post is working after updating versions nor I want to update the code if it fails.

This is what freeze feature in Quarto is about. You could basically render once the .qmd using your current packages available, save and commit the freeze directory, and this would allow you post to re-render to HTML even with no package or R process. So no need to deal with renv per post for this kind of need. You could work with a global library and no matter its state, freeze will insure you will be able to render your blog.

Just sharing thoughts and ideas. Hope it helps!