venv on top of uenv

Author: boeschf

docs/guides/storage.md

@@ -203,9 +203,13 @@ The first step is to create the virtual environment using the usual workflow.
     # and other useful tools
     uenv start prgenv-gnu/24.11:v1 --view=default
 
+    # unset PYTHONPATH and set PYTHONUSERBASE to avoid conflicts
+    unset PYTHONPATH
+    export PYTHONUSERBASE=/user-environment/env/default
+
     # create and activate a new relocatable venv using uv
     # in this case we explicitly select python 3.12
-    uv venv -p 3.12 --relocatable --link-mode=copy /dev/shm/sqfs-demo/.venv
+    uv venv -p /user-environment/env/default/bin/python --system-site-packages --relocatable --link-mode=copy /dev/shm/sqfs-demo/.venv
     # You can also point to the uenv python with `uv venv -p $(which python) ...`
     # which, among other things, enables user portability of the venv
     cd /dev/shm/sqfs-demo
@@ -231,12 +235,16 @@ The first step is to create the virtual environment using the usual workflow.
     # and other useful tools
     uenv start prgenv-gnu/24.11:v1 --view=default
 
+    # unset PYTHONPATH and set PYTHONUSERBASE to avoid conflicts
+    unset PYTHONPATH
+    export PYTHONUSERBASE=/user-environment/env/default
+
     # for the example create a working path on SCRATCH
     mkdir $SCRATCH/sqfs-demo
     cd $SCRATCH/sqfs-demo
 
     # create and activate the empty venv
-    python -m venv ./.venv
+    python -m venv --system-site-packages ./.venv
     source ./.venv/bin/activate
 
     # install software in the virtual environment

docs/software/ml/pytorch.md

@@ -810,12 +810,15 @@ $ exit # (6)!
     Python virtual environments can be slow on the parallel Lustre file system due to the amount of small files and potentially many processes accessing it.
     If this becomes a bottleneck, consider [squashing the venv][ref-guides-storage-venv] into its own memory-mapped, read-only file system to enhance scalability and reduce load times.
 
-??? bug "Python packages from uenv shadowing those in a virtual environment"
-    When using uenv with a virtual environment on top, the site-packages under `/user-environment` currently take precedence over those in the activated virtual environment. This is due to the uenv paths being included in the `PYTHONPATH` environment variable. As a consequence, despite installing a different version of a package in the virtual environment from what is available in the uenv, the uenv version will still be imported at runtime. A possible workaround is to prepend the virtual environment's site-packages to `PYTHONPATH` whenever activating the virtual environment.
+??? bug "Python packages from uenv view shadowing those in a virtual environment"
+    Some uenv views set the `PYTHONPATH` environment variable and/or do not set the `PYTHONUSERBASE` environment variable.
+    This can lead to unexpected behavior when using Python virtual environments on top of the uenv, as the packages installed in the uenv view may take precedence over those in the virtual environment.
+    A possible workaround is to unset the `PYTHONPATH` and set the `PYTHONUSERBASE` environment variables, as described in the [Python virtual environments with uenv guide][ref-guides-storage-venv]:
     ```bash
-    export PYTHONPATH="$(python -c 'import site; print(site.getsitepackages()[0])'):$PYTHONPATH"
+    unset PYTHONPATH
+    export PYTHONUSERBASE=/user-environment/env/default
     ```
-    It is recommended to apply this workaround if you are constrained by a Python package version installed in the uenv that you need to change for your application.
+    It is recommended to apply this workaround if you are constrained by a Python package version installed in the uenv view that you need to change for your application.
 
 !!! note
     Keep in mind that

docs/software/uenv/index.md

@@ -334,6 +334,10 @@ Views are loaded using the `--view` flag for `uenv start` (also for `uenv run` a
     $ uenv start --view=prgenv-gnu:default,editors:ed prgenv-gnu/24.11:v1,editors
     ```
 
+!!! info
+    Python virtual environments can be created on top of a uenv view.
+    However, to ensure that the Python interpreter and packages from the uenv view are used, the `PYTHONPATH` and `PYTHONUSERBASE` environment variables must be set correctly, see our guide on [Python virtual environments with uenv][ref-guides-storage-venv].
+
 #### Modules
 
 Most uenv provide the modules, that can be accessed using the `module` command.