Skip to content

Commit 3f12e4b

Browse files
committed
Enhance explanations and add bytecode precompilation step
1 parent eb89002 commit 3f12e4b

File tree

1 file changed

+6
-3
lines changed

1 file changed

+6
-3
lines changed

docs/guides/storage.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,20 +68,24 @@ pip install torch torchvision torchaudio \
6868

6969
##### Alternative virtual environment creation using uv
7070

71-
The installation process described above is not optimized for file system performance and will still be slow on Lustre filesystems. An alternative way to create the virtual environment is to use the [uv](https://docs.astral.sh/uv/) tool, which supports _relocatable_ virtual environments and asynchronous package downloads for better installation times. This way, the installation process is much shorter and the resulting squashfs image can be shared across projects, as the virtual environment can be safely used from any location.
71+
The installation process described above is not optimized for file system performance and will still be slow on Lustre filesystems. An alternative way to create the virtual environment is to use the [uv](https://docs.astral.sh/uv/) tool, which supports _relocatable_ virtual environments and asynchronous package downloads. The main benefit of a relocatable virtual environment is that it does not need to be created in the final path from where it will be used. This allows the use of shared memory to speed up the creation and initialization of the virtual environment and, since the virtual environment can be used from any location, the resulting squashfs image can be safely shared across projects.
7272

7373
```bash
7474
# activate the uenv as before
7575
uenv start prgenv-gnu/24.11:v1 --view=default
7676

7777
# create and activate a new relocatable venv using uv
78-
uv venv --relocatable --link-mode=copy /dev/shm/sqfs-demo/.venv
78+
# in this case we explicitly select python 3.12
79+
uv venv -p 3.12 --relocatable --link-mode=copy /dev/shm/sqfs-demo/.venv
7980
cd /dev/shm/sqfs-demo
8081
source .venv/bin/activate
8182

8283
# install software in the virtual environment using uv
8384
uv pip install --link-mode=copy torch torchvision torchaudio \
8485
--index-url https://download.pytorch.org/whl/cu126
86+
# optionally, to reduce the import times, precompile all
87+
# python modules to bytecode before creating the squashfs image
88+
python -m compileall .venv/lib/python3.12/site-packages
8589
```
8690

8791
#### Step 2: make a squashfs image of the virtual environment
@@ -145,4 +149,3 @@ If you need to modify the virtual environment, run the original uenv without the
145149

146150
!!! hint
147151
If you save the updated copy in a different file, you can now "roll back" to the old version of the environment by mounting the old image.
148-

0 commit comments

Comments
 (0)