Pixi list cannot see packages installed with uv pip install

[sidmadala]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

Shell Commands:

sidmadala@icgpu01:~/hocklab/test-repo$ pixi install
sidmadala@icgpu01:~/hocklab/test-repo$ pixi shell
(test) sidmadala@icgpu01:~/hocklab/test-repo$ uv pip install --no-build-isolation axolotl[flash-attn,deepspeed]==0.9.0
(test) sidmadala@icgpu01:~/hocklab/test-repo$ uv pip install vllm==0.8.5
(test) sidmadala@icgpu01:~/hocklab/test-repo$ pixi list | grep -E "axol|vllm"

Testing if axolotl and vllm were installed:

import axolotl
import vllm

print(axolotl.__version__)  # 0.9.0
print(vllm.__version__)  # 0.8.5

Pixi.toml:

[workspace]
authors = ["sidmadala"]
channels = ["conda-forge", "nvidia"]
name = "test"
platforms = ["linux-64"]
version = "0.1.0"

[tasks]

[dependencies]
python = "3.12.*"
cuda-toolkit = { version = "12.4.*", channel = "nvidia" }
uv = ">=0.7.2,<0.8"

[pypi-dependencies]
torch = "==2.6.0"
torchvision = "*"
huggingface-hub = { version = ">=0.30.0", extras = ["cli", "torch"] }
transformers = ">=4.51"
evaluate = ">=0.4.0"
datasets = ">=3.5.0"
accelerate = ">=1.6.0"
peft = ">=0.15.0"
bitsandbytes = ">=0.45.0"
trl = ">=0.17.0"
unsloth = "*"
sentencepiece = "*"
openai = "*"
scikit-learn = "*"
scipy = "*"
jupyterlab = "*"
matplotlib = "*"
seaborn = "*"
gradio = "*"
wandb = "*"
nltk = "*"
pandas = "*"
tenacity = "*"
requests = "*"
aiohttp = "*"
ratelimit = "*"
importlib-resources = "*"
mypy = "*"
python-dotenv = ">=1.0.0"
nvitop = "==1.5.0"
gpustat = "==1.1.1"

Issue description

Pixi is not able to see axolotl or vllm when installed using uv pip install. I cannot use uv add since I am using pixi.toml and not pyproject.toml due to issue #3499. While the packages seem to be installed correctly when testing if they can be imported, the result of (test) sidmadala@icgpu01:~/hocklab/test-repo$ pixi list | grep -E "axol|vllm" is empty, which confuses me. Any assistance would be much appreciated.

Expected behavior

Since Pixi cannot install axolotl via pixi add --pypi --no-build-isolation or in the pixi.toml itself (#3730), I would prefer if pixi can see what is installed via uv pip install --no-build-isolation since this behavior seems to work when using pip inside of a conda environment.

[noamgot]

You should add a no-build-isolation table manually - see here: https://pixi.sh/latest/reference/pixi_manifest/#no-build-isolation

I copied your pixi.toml file and managed to install the packages you installed with uv by using pixi add --pypi

Pay attention to the note in the docs:

To use no-build-isolation effectively, use conda dependencies to define the build environment. These are installed before the PyPI dependencies are resolved, this way these dependencies are available during the build process. In the example above adding torch as a PyPI dependency would be ineffective, as it would not yet be installed during the PyPI resolution phase.

I think I managed to do this despite torch being a pypi dependency because I did it in multiple stages. I'm not sure it will work out-of-the-box in other methods (e.g, by retrieving the pixi.toml and pixi.lock file and running pixi install)

[sidmadala]

Could you paste any modifications you made to the pixi.toml or installation process?

[noamgot]

Sorry, already deleted 😬
Assuming you installed an environment with your configuration, just add the problematic pypi packages to the table I mentioned above, and run the pixi add commands (instead of uv pip install)

[sidmadala]

My current setup is this which is still not working after following the suggestions:

pixi.toml:

[workspace]
authors = ["sidmadala"]
channels = ["conda-forge", "nvidia"]
name = "test"
platforms = ["linux-64"]
version = "0.1.0"

[tasks]

[dependencies]
python = "3.12.*"
cuda-toolkit = { version = "12.4.*", channel = "nvidia" }
uv = ">=0.7.2,<0.8"

[pypi-options]
no-build-isolation = ["axolotl"]

[pypi-dependencies]
torch = "==2.6.0"
torchvision = "*"
huggingface-hub = { version = ">=0.30.0", extras = ["cli", "torch"] }
transformers = ">=4.51"
evaluate = ">=0.4.0"
datasets = ">=3.5.0"
accelerate = ">=1.6.0"
peft = ">=0.15.0"
bitsandbytes = ">=0.45.0"
trl = ">=0.17.0"
unsloth = "*"
sentencepiece = "*"
openai = "*"
scikit-learn = "*"
scipy = "*"
jupyterlab = "*"
matplotlib = "*"
seaborn = "*"
gradio = "*"
wandb = "*"
nltk = "*"
pandas = "*"
tenacity = "*"
requests = "*"
aiohttp = "*"
ratelimit = "*"
importlib-resources = "*"
mypy = "*"
python-dotenv = ">=1.0.0"
nvitop = "==1.5.0"
gpustat = "==1.1.1"

vllm = "==0.8.5"
# axolotl = { version = "==0.9.0", extras = ["flash-attn", "deepspeed"] }

Commands:

sidmadala@icgpu01:~/hocklab/test-repo$ pixi install  # Installs all dependencies listed above
sidmadala@icgpu01:~/hocklab/test-repo$ pixi shell 
(test-pixi) sidmadala@icgpu01:~/hocklab/test-pixi$ pixi add --pypi axolotl[flash-attn,deepspeed]==0.9.0
Error:   × Failed to update PyPI packages for environment 'default'
  ├─▶ Failed to prepare distributions
  ├─▶ Failed to build `flash-attn==2.7.4.post1`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `setuptools.build_meta:__legacy__.build_wheel` failed (exit status: 1)
      
      [stderr]
      Traceback (most recent call last):
        File "<string>", line 14, in <module>
        File "/home/sidmadala/.cache/rattler/cache/uv-cache/builds-v0/.tmp9tC27W/lib/python3.12/site-packages/setuptools/build_meta.py", line 331, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/sidmadala/.cache/rattler/cache/uv-cache/builds-v0/.tmp9tC27W/lib/python3.12/site-packages/setuptools/build_meta.py", line 301, in _get_build_requires
          self.run_setup()
        File "/home/sidmadala/.cache/rattler/cache/uv-cache/builds-v0/.tmp9tC27W/lib/python3.12/site-packages/setuptools/build_meta.py", line 512, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/home/sidmadala/.cache/rattler/cache/uv-cache/builds-v0/.tmp9tC27W/lib/python3.12/site-packages/setuptools/build_meta.py", line 317, in run_setup
          exec(code, locals())
        File "<string>", line 22, in <module>
      ModuleNotFoundError: No module named 'torch'
      
      hint: This error likely indicates that `[email protected]` depends on `torch`, but doesn't declare it as a build dependency. If `flash-attn` is a first-party package, consider adding `torch` to its `build-
      system.requires`. Otherwise, `uv pip install torch` into the environment and re-run with `--no-build-isolation`.

[sidmadala]

For posterity, I seem to have solved the issue. Still not ideal (one command pixi install) but it works for now.

pixi.toml:

[workspace]
authors = ["sidmadala"]
channels = ["conda-forge", "nvidia"]
name = "test-pixi"
platforms = ["linux-64"]
version = "0.1.0"

[tasks]

[dependencies]
python = "3.12.*"
cuda-toolkit = { version = "12.4.*", channel = "nvidia" }
uv = ">=0.7.2,<0.8"

[pypi-options]
no-build-isolation = ["axolotl", "flash-attn"]  # THIS IS THE IMPORTANT SECTION!!!

[pypi-dependencies]
torch = "==2.6.0"
torchvision = "*"
huggingface-hub = { version = ">=0.30.0", extras = ["cli", "torch"] }
transformers = ">=4.51"
evaluate = ">=0.4.0"
datasets = ">=3.5.0"
accelerate = ">=1.6.0"
peft = ">=0.15.0"
bitsandbytes = ">=0.45.0"
trl = ">=0.17.0"
unsloth = "*"
sentencepiece = "*"
openai = "*"
scikit-learn = "*"
scipy = "*"
jupyterlab = "*"
matplotlib = "*"
seaborn = "*"
gradio = "*"
wandb = "*"
nltk = "*"
pandas = "*"
tenacity = "*"
requests = "*"
aiohttp = "*"
ratelimit = "*"
importlib-resources = "*"
mypy = "*"
python-dotenv = ">=1.0.0"
nvitop = "==1.5.0"
gpustat = "==1.1.1"

vllm = "==0.8.5"
# axolotl = { version = "==0.9.0", extras = ["flash-attn", "deepspeed"] }

Commands:

sidmadala@icgpu01:~/hocklab/test-repo$ pixi install  # Installs all dependencies listed above
sidmadala@icgpu01:~/hocklab/test-repo$ pixi shell 
(test-pixi) sidmadala@icgpu01:~/hocklab/test-pixi$ pixi add --pypi axolotl[flash-attn,deepspeed]==0.9.0
(test-pixi) sidmadala@icgpu01:~/hocklab/test-pixi$ p list | grep -E "torch|flash|deep|axol|vllm"
axolotl                                   0.9.0                                    1.2 MiB     pypi   axolotl-0.9.0-py3-none-any.whl
axolotl_contribs_lgpl                     0.0.6                                    30.7 KiB    pypi   axolotl_contribs_lgpl-0.0.6-py3-none-any.whl
axolotl_contribs_mit                      0.0.3                                    11.8 KiB    pypi   axolotl_contribs_mit-0.0.3-py3-none-any.whl
deepspeed                                 0.15.4                                   5.7 MiB     pypi   deepspeed-0.15.4-py3-none-any.whl
deepspeed_kernels                         0.0.1.dev1698255861                      181 MiB     pypi   deepspeed_kernels-0.0.1.dev1698255861-py3-none-manylinux1_x86_64.whl
flash_attn                                2.7.4.post1                              576.7 MiB   pypi   flash_attn-2.7.4.post1-cp312-cp312-linux_x86_64.whl
torch                                     2.6.0                                    1.4 GiB     pypi   torch-2.6.0-cp312-cp312-manylinux1_x86_64.whl
torchao                                   0.9.0                                    24.3 MiB    pypi   torchao-0.9.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl
torchaudio                                2.6.0                                    12.4 MiB    pypi   torchaudio-2.6.0-cp312-cp312-manylinux1_x86_64.whl
torchvision                               0.21.0                                   19 MiB      pypi   torchvision-0.21.0-cp312-cp312-manylinux1_x86_64.whl
vllm                                      0.8.5                                    1014 MiB    pypi   vllm-0.8.5-cp38-abi3-manylinux1_x86_64.whl

[noamgot]

FYI, you could just run pixi list "foo|bar" to achieve the same resolve as pixi list | grep -E "foo|bar" ...

[tdejager]

Hi, so what is the issue remaining :)? Sorry bit hard to follow for me

[noamgot]

IMO there's no real issue here
@sidmadala tried to add dependencies directly with uv, while one could add using pixi add --pypi (in this case, with the relevant modifications discussed above).

There's the subtle points stated in the docs - for pypi dependencies that require other dependencies for building, one should either replace the needed dependency with a respective conda dependency, or pre-install the needed (pypi) dependency and then install the relevant package that needs it (i.e., 2-stage installation). See the torch discussion above...

[tdejager]

Thanks that clarifies it, and that means pixi list works as expected?

[noamgot]

after adding with pixi add --pypi - yes, of course :)
the OP wanted pixi list to also identify packages added with uv pip install (after activating a pixi shell), which is impossible, AFAICT

[sidmadala]

Since conda list also shows packages installed with pip in conda environments, I felt it would be natural for pixi to have the same behavior. It also feels weird to be forced to write out --no-build-isolation packages in the pixi.toml instead of just doing pixi add --pypy --no-build-isolation axolotl[flash-attn,deepspeed] and having the pixi.toml updated automatically to reflect this specific type of installation. Something similar to this perhaps.

[tdejager]

Hey!

So I guess you are asking two things:

1. pixi list will crawl your environment and list packages not installed by pixi. But after the fact by uv I guess this is kind of out of scope for pixi list currently because it shows the locked dependencies. So we are not going to add this as of now. I can move this over to discussion.
2. Have some way to add a package from the CLI with --no-build-isolation enabled. This makes more sense to me but is a different issue. The example you pointed to is already complex, because I think you want to add the same flag to the extra's in this case they are the same as the package name but they need not be. We could have simple version that does not do that to start off with though.