Skip to content

Commit eb89002

Browse files
committed
Update venv-squasfs documentation with support for relocatable venvs with uv
1 parent 88c0f1c commit eb89002

File tree

1 file changed

+19
-2
lines changed

1 file changed

+19
-2
lines changed

docs/guides/storage.md

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,6 @@ This file can be mounted as a read-only [Squashfs](https://en.wikipedia.org/wiki
3333
#### Step 1: create the virtual environment
3434

3535
The first step is to create the virtual environment using the usual workflow.
36-
This might be slow, because we are not optimising this stage for file system performance.
3736

3837
```bash
3938
# for the example create a working path on SCRATCH
@@ -67,6 +66,24 @@ pip install torch torchvision torchaudio \
6766

6867
In our "simple" pytorch example, I counted **22806 inodes**!
6968

69+
##### Alternative virtual environment creation using uv
70+
71+
The installation process described above is not optimized for file system performance and will still be slow on Lustre filesystems. An alternative way to create the virtual environment is to use the [uv](https://docs.astral.sh/uv/) tool, which supports _relocatable_ virtual environments and asynchronous package downloads for better installation times. This way, the installation process is much shorter and the resulting squashfs image can be shared across projects, as the virtual environment can be safely used from any location.
72+
73+
```bash
74+
# activate the uenv as before
75+
uenv start prgenv-gnu/24.11:v1 --view=default
76+
77+
# create and activate a new relocatable venv using uv
78+
uv venv --relocatable --link-mode=copy /dev/shm/sqfs-demo/.venv
79+
cd /dev/shm/sqfs-demo
80+
source .venv/bin/activate
81+
82+
# install software in the virtual environment using uv
83+
uv pip install --link-mode=copy torch torchvision torchaudio \
84+
--index-url https://download.pytorch.org/whl/cu126
85+
```
86+
7087
#### Step 2: make a squashfs image of the virtual environment
7188

7289
The next step is to create a single squashfs file that contains the whole `$SCRATCH/sqfs-demo/.pyenv` path.
@@ -117,7 +134,7 @@ Note that the original virtual environment is still installed in `$SCRATCH/sqfs-
117134
A benefit of this approach is that the squashfs file can be copied to a location that is not subject to the Scratch cleaning policy.
118135

119136
!!! warning
120-
Virtual environment are usually not relocatable as they contain symlinks to absolute locations inside the virtual environment. Therefore, you need to mount the image in the exact same location where you created the virtual environment.
137+
Virtual environments are not relocatable by default as they contain symlinks to absolute locations inside the virtual environment. This means that the squashfs file must be mounted in the exact same location where the virtual environment was created, unless it contains a virtual environment specifically created using a tool with support for relocatable virtual environments (e.g. `uv venv --relocatable` as mentioned in step 1), in which case it can be mounted in any location.
121138

122139
#### Step 4: (optional) regenerate the virtual environment
123140

0 commit comments

Comments
 (0)