Speed of pixi run --frozen

[kcajf]

As far as I understand, with --frozen these two commands should be very lightweight:

pixi shell-hook --environment=default --frozen
pixi run --environment=default --frozen ...

i.e. they should "just" be verifying that the contents of .pixi match pixi.lock (and shouldn't be going out to the internet).

They both take somewhere around 1 second though, which makes me wonder why. What checks do they need to do? Is there a way to make this check very fast, if neither pixi.lock nor the contents of .pixi have changed? by e.g. comparing mtimes or some hash?

[kcajf]

ah, I think this is related to #1541

[baszalmstra]

that should be quite fast indeed! Do you have both files so we can benchmark them?

[kcajf]

Here's a slightly slimmed-down pixi.toml that removes a few private channel libraries (some of which are pypi):

[project]
name = "asdf"
version = "0.1.0"
description = "asdf"
authors = ["asdf"]
channels = [
"conda-forge"
]
platforms = ["linux-64"]

[tasks]

[system-requirements]
linux = "4.18"

[dependencies]
"ruamel.yaml" = "0.18.6.*"
atomicwrites = "1.4.1.*"
attrdict = "2.0.1.*"
boto3 = "1.34.55.*"
bottleneck = "1.3.8.*"
catboost = "1.2.3.*"
click = "8.1.7.*"
cvxopt = "1.3.2.*"
dash = "2.6.1.*"
dnspython = "*"
gunicorn = "*"
holoviews = "*"
joblib = "1.3.2.*"
libblas = { version = "*", build = "*openblas" }
lightgbm = "4.3.0.*"
matplotlib = "3.8.3.*"
msgspec = "0.18.*"
networkx = "2.8.*"
numba = "0.59.0.*"
numpy = "1.26.3.*"
optuna = "3.5.0.*"
pandas = "1.5.3.*"
panel = "1.3.8.*"
plotly = "5.19.0.*"
pyarrow = "13.0.*"
pycurl = "7.45.1.*"
pytest = "8.0.2.*"
python = "3.11.*"
python-gssapi = "1.8.3.*"
pyyaml = "6.0.1.*"
requests = "2.31.0.*"
requests-kerberos = "0.12.*"
scikit-learn = "1.4.0.*"
scipy = "1.12.0.*"
slack_sdk = "3.27.1.*"
sortednp = "0.4.1.*"
tabulate = "0.9.*"
websocket-client = "1.8.*"
zstandard = "0.22.0.*"
zstd = "1.5.5.*"
pygraphviz = ">=1.13,<2"
cloudpickle = ">=3.0.0,<3.1"

time pixi shell-hook --environment=default --frozen
pixi shell-hook --environment=default --frozen  0.42s user 0.90s system 351% cpu 0.374 total

this now takes ~370ms, down from 1s before. Still quite a lot. It seems to roughly scale with number of packages / file size - if you comment some out then do pixi install it drops proportionally.

[kcajf]

Actually I've just timed the full env, and it's closer to 2.6 seconds

[baszalmstra]

I've been trying to debug this further but have not been able to pinpoint why this is so slow for you.

With the --frozen option:

• Pixi loads the lock-file,
• Reads the json files from the /conda-meta/ folder and compares that to the lock-file.
• If that still matches, simply runs the command in the environment.

On my machine, even with very large environments (1+MB lock files) this takes at most 400ms.

I assume you have decent hardware? Not running on a network drive?

Could you run with -vvv to see if there is any strange output there?

[kcajf]

Thanks for looking @baszalmstra . I'm on a fast machine and the pixi environment is on local SSD storage.

[kcajf@loywkcajf project]$ pixi -vvv shell-hook --environment=default --frozen
DEBUG pixi::config: Loading config from /etc/pixi/config.toml
DEBUG pixi::config: Failed to load system config: /etc/pixi/config.toml (error: failed to read config from '/etc/pixi/config.toml')
DEBUG pixi::config: Loading config from /home/kcajf/.config/pixi/config.toml
 INFO pixi::config: Loaded config from: /home/kcajf/.config/pixi/config.toml
DEBUG pixi::config: Loading config from /home/kcajf/.config/pixi/config.toml
 INFO pixi::config: Loaded config from: /home/kcajf/.config/pixi/config.toml
DEBUG pixi::config: Loading config from /home/kcajf/.pixi/config.toml
DEBUG pixi::config: Failed to load global config: /home/kcajf/.pixi/config.toml (error: failed to read config from '/home/kcajf/.pixi/config.toml')
DEBUG pixi::config: Loading config from /data/home/kcajf/code/project/.pixi/config.toml
DEBUG pixi::config: Failed to load local config: /data/home/kcajf/code/project/.pixi/config.toml (error: failed to read config from '/data/home/kcajf/code/project/.pixi/config.toml')
 INFO pixi::environment: verifying prefix location is unchanged, with prefix file: /data/home/kcajf/code/project/.pixi/envs/default/conda-meta/pixi_env_prefix
 INFO pixi::lock_file::update: skipping check if lock-file is up-to-date
DEBUG pixi::environment: Wrote environment file to: "/data/home/kcajf/code/project/.pixi/envs/default/conda-meta/pixi"
 INFO pixi::environment: Creating prefix file at: /data/home/kcajf/code/project/.pixi/envs/default/conda-meta/pixi_env_prefix
 INFO pixi::environment: No update needed for the prefix file.
 INFO pixi::environment: Checking if history file exists: /data/home/kcajf/code/project/.pixi/envs/default/conda-meta/history
 INFO pixi::lock_file::resolve::uv_resolution_context: uv keyring provider is disabled
DEBUG uv_client::base_client: Using request timeout of 30s
DEBUG pixi::install_pypi: [Install] Using Python Interpreter: Interpreter { platform: Platform { os: Manylinux { major: 2, minor: 28 }, arch: X86_64 }, markers: MarkerEnvironment { inner: MarkerEnvironmentInner { implementation_name: "cpython", implementation_version: StringVersion { string: "3.11.7", version: "3.11.7" }, os_name: "posix", platform_machine: "x86_64", platform_python_implementation: "CPython", platform_release: "4.18.0-348.20.1.el8_5.x86_64", platform_system: "Linux", platform_version: "#1 SMP Thu Mar 10 20:59:28 UTC 2022", python_full_version: StringVersion { string: "3.11.7", version: "3.11.7" }, python_version: StringVersion { string: "3.11", version: "3.11" }, sys_platform: "linux" } }, scheme: Scheme { purelib: "/data/home/kcajf/code/project/.pixi/envs/default/lib/python3.11/site-packages", platlib: "/data/home/kcajf/code/project/.pixi/envs/default/lib/python3.11/site-packages", scripts: "/data/home/kcajf/code/project/.pixi/envs/default/bin", data: "/data/home/kcajf/code/project/.pixi/envs/default", include: "/data/home/kcajf/code/project/.pixi/envs/default/include/python3.11" }, virtualenv: Scheme { purelib: "lib/python3.11/site-packages", platlib: "lib/python3.11/site-packages", scripts: "bin", data: "", include: "include/site/python3.11" }, sys_prefix: "/data/home/kcajf/code/project/.pixi/envs/default", sys_base_exec_prefix: "/data/home/kcajf/code/project/.pixi/envs/default", sys_base_prefix: "/data/home/kcajf/code/project/.pixi/envs/default", sys_base_executable: Some("/data/home/kcajf/code/project/.pixi/envs/default/bin/python3.11"), sys_executable: "/data/home/kcajf/code/project/.pixi/envs/default/bin/python3.11", sys_path: ["/home/kcajf/.cache/rattler/cache/uv-cache/.tmpWpG9AL", "/data/home/kcajf/code/project/.pixi/envs/default/lib/python311.zip", "/data/home/kcajf/code/project/.pixi/envs/default/lib/python3.11", "/data/home/kcajf/code/project/.pixi/envs/default/lib/python3.11/lib-dynload", "/data/home/kcajf/code/project/.pixi/envs/default/lib/python3.11/site-packages", "/data/home/kcajf/code/project/.pixi/envs/default/lib/python3.11/site-packages/gsa_monitor_client-1.0.4-py3.12.egg"], stdlib: "/data/home/kcajf/code/project/.pixi/envs/default/lib/python3.11", tags: OnceCell(Uninit), target: None, prefix: None, pointer_size: _64, gil_disabled: false }
DEBUG uv_fs: Acquired lock for `.pixi/envs/default`
DEBUG pixi::install_pypi: Constructed site-packages with 250 packages
DEBUG pixi::install_pypi: Figuring out what to install/reinstall/remove
 INFO pixi::install_pypi: Nothing to do - Audited 23 distributions in 341ms
DEBUG rattler_shell::shell: Guessing the current shell is bash. Parent process name: bash and args: []

Here is the output with -vvv. the above took ~3 seconds

du -hs .pixi/envs/default/conda-meta
41M     .pixi/envs/default/conda-meta

the conda-meta folder is 41MB.

du -hs pixi.lock
444K    pixi.lock

[baszalmstra]

Ah! The conda-meta folder seems quite big! Perhaps reading that is something we came optimize!

[baszalmstra]

Are there any particularly large files? How many json files are in there?

[kcajf]

There are 435 files. Here is the tail of all larger than 100kB:

100K    .pixi/envs/default/conda-meta/gsl-2.7-he838d99_0.json
100K    .pixi/envs/default/conda-meta/pyzmq-26.0.3-py311h08a0b41_0.json
104K    .pixi/envs/default/conda-meta/gettext-tools-0.22.5-h59595ed_2.json
104K    .pixi/envs/default/conda-meta/graphviz-11.0.0-hc68bbd7_0.json
104K    .pixi/envs/default/conda-meta/xkeyboard-config-2.42-h4ab18f5_0.json
112K    .pixi/envs/default/conda-meta/dash-2.6.1-pyhd8ed1ab_0.json
120K    .pixi/envs/default/conda-meta/ipython-8.26.0-pyh707e725_0.json
132K    .pixi/envs/default/conda-meta/glib-2.80.3-h8a4344b_1.json
132K    .pixi/envs/default/conda-meta/gst-plugins-base-1.22.9-hfa15dee_1.json
132K    .pixi/envs/default/conda-meta/libgrpc-1.58.2-he06187c_0.json
136K    .pixi/envs/default/conda-meta/pygments-2.18.0-pyhd8ed1ab_0.json
180K    .pixi/envs/default/conda-meta/notebook-7.2.1-pyhd8ed1ab_0.json
180K    .pixi/envs/default/conda-meta/tk-8.6.13-noxft_h4845f30_101.json
204K    .pixi/envs/default/conda-meta/libhwloc-2.11.1-default_hecaa2ac_1000.json
208K    .pixi/envs/default/conda-meta/setuptools-71.0.1-pyhd8ed1ab_0.json
208K    .pixi/envs/default/conda-meta/tzdata-2024a-h0c530f3_0.json
212K    .pixi/envs/default/conda-meta/gtk2-2.24.33-h280cfa0_4.json
216K    .pixi/envs/default/conda-meta/libarrow-13.0.0-h8d17fec_10_cpu.json
228K    .pixi/envs/default/conda-meta/sqlalchemy-2.0.31-py311h331c9d8_0.json
248K    .pixi/envs/default/conda-meta/pytz-2024.1-pyhd8ed1ab_0.json
252K    .pixi/envs/default/conda-meta/rdma-core-52.0-he02047a_0.json
256K    .pixi/envs/default/conda-meta/debugpy-1.8.2-py311h4332511_0.json
264K    .pixi/envs/default/conda-meta/libgoogle-cloud-2.12.0-h19a6dae_3.json
268K    .pixi/envs/default/conda-meta/pyarrow-13.0.0-py311h39c9aba_10_cpu.json
276K    .pixi/envs/default/conda-meta/fonttools-4.53.1-py311h61187de_0.json
276K    .pixi/envs/default/conda-meta/networkx-2.8.8-pyhd8ed1ab_0.json
296K    .pixi/envs/default/conda-meta/holoviews-1.19.1-pyhd8ed1ab_0.json
304K    .pixi/envs/default/conda-meta/holidays-0.53-pyhd8ed1ab_0.json
320K    .pixi/envs/default/conda-meta/kernel-headers_linux-64-3.10.0-h4a8ded7_16.json
320K    .pixi/envs/default/conda-meta/libabseil-20230802.1-cxx17_h59595ed_0.json
344K    .pixi/envs/default/conda-meta/libstdcxx-devel_linux-64-12.2.0-h3b97bd3_19.json
352K    .pixi/envs/default/conda-meta/libpq-16.1-h33b98f1_7.json
376K    .pixi/envs/default/conda-meta/jupyterlab-4.2.4-pyhd8ed1ab_0.json
388K    .pixi/envs/default/conda-meta/matplotlib-base-3.8.3-py311h54ef318_0.json
400K    .pixi/envs/default/conda-meta/pyqt-5.15.9-py311hf0fb5b6_5.json
416K    .pixi/envs/default/conda-meta/babel-2.14.0-pyhd8ed1ab_0.json
500K    .pixi/envs/default/conda-meta/scipy-1.12.0-py311h64a7726_2.json
504K    .pixi/envs/default/conda-meta/bokeh-3.3.4-pyhd8ed1ab_0.json
560K    .pixi/envs/default/conda-meta/panel-1.3.8-pyhd8ed1ab_0.json
560K    .pixi/envs/default/conda-meta/sysroot_linux-64-2.17-h4a8ded7_16.json
584K    .pixi/envs/default/conda-meta/scikit-learn-1.4.0-py311hc009520_0.json
600K    .pixi/envs/default/conda-meta/numpy-1.26.3-py311h64a7726_0.json
648K    .pixi/envs/default/conda-meta/numba-0.59.0-py311h96b013e_1.json
672K    .pixi/envs/default/conda-meta/gettext-0.22.5-h59595ed_2.json
688K    .pixi/envs/default/conda-meta/aws-sdk-cpp-1.11.156-h10f61e1_5.json
792K    .pixi/envs/default/conda-meta/botocore-1.34.144-pyge310_1234567_0.json
812K    .pixi/envs/default/conda-meta/python-3.11.7-hab00c5b_1_cpython.json
868K    .pixi/envs/default/conda-meta/jedi-0.19.1-pyhd8ed1ab_0.json
1.1M    .pixi/envs/default/conda-meta/ncurses-6.5-h59595ed_0.json
1.3M    .pixi/envs/default/conda-meta/pandas-1.5.3-py311h2872171_1.json
5.0M    .pixi/envs/default/conda-meta/qt-main-5.15.8-h450f30e_18.json
5.3M    .pixi/envs/default/conda-meta/plotly-5.19.0-pyhd8ed1ab_0.json
5.8M    .pixi/envs/default/conda-meta/libboost-headers-1.84.0-ha770c72_2.json

[kcajf]

as well as / instead of speeding up the parsing of these files, could we avoid parsing them entirely by doing some mtime or hash check for the common case of "nothing has changed"?