Test python library: dependencies not automatically imported?

[tobiasBora]

I tried to review a python library (NixOS/nixpkgs#398351) by doing:

$ nixpkgs-review pr 398351
$ python
>>> import picos
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/qmsllz8m16lc6s2kph2skz4z9g670qdb-python3.13-picos-2.6.1/lib/python3.13/site-packages/picos/__init__.py", line 43, in <module>
    from . import settings, uncertain  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qmsllz8m16lc6s2kph2skz4z9g670qdb-python3.13-picos-2.6.1/lib/python3.13/site-packages/picos/uncertain.py", line 27, in <module>
    from .expressions.uncertain import (IntractableWorstCase,  # noqa
  File "/nix/store/qmsllz8m16lc6s2kph2skz4z9g670qdb-python3.13-picos-2.6.1/lib/python3.13/site-packages/picos/expressions/__init__.py", line 28, in <module>
    from .algebra import *  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qmsllz8m16lc6s2kph2skz4z9g670qdb-python3.13-picos-2.6.1/lib/python3.13/site-packages/picos/expressions/algebra.py", line 27, in <module>
    import cvxopt
  File "/nix/store/rk119pj16f652yld3bhn5v6m4fa0a8ks-python3.13-cvxopt-1.3.2/lib/python3.13/site-packages/cvxopt/__init__.py", line 57, in <module>
    import cvxopt.base
ModuleNotFoundError: No module named 'cvxopt.base'

But I'm confused by this error since cvxopt is already included in the dependencies of this library:

  dependencies = [
    numpy
    cvxopt
    scipy
  ];

The fix I found for now is to manually import cvxopt like:

$ nixpkgs-review pr 398351 -p python3Packages.cvxopt -p python3Packages.picos
$ python
>>> import picos
>>> 

and it works as expected. Yet, I don't understand why I need to do it manually.

[Mic92]

Python dependencies used to be propagated into a nix-shell. If this no longer works than this is something nixpkgs needs to fix. nixpkgs-review hasn't changed about the way things are imported.

[Mic92]

The only caveat is if you have a lot of changed packages:

nixpkgs-review/nixpkgs_review/nix/review-shell.nix

Line 40 in c252cff

packages = if builtins.length attrs > 50 then [ env ] else attrs;

in which case we use buildEnv rather where propagation no longer works.

[tobiasBora]

Hum, weird, I tried and nix-shell (outside nixpkgs review) does propagate the dependencies:

{ pkgs ? import <nixpkgs> { }, }:
pkgs.callPackage ({mkShell, python3}:
  mkShell {
    # libraries
    buildInputs = [
      (python3.withPackages (ps: [ps.picos]))
    ];
  }
) { }

$ nix-shell
$ python
Python 3.12.9 (main, Feb  4 2025, 14:38:38) [GCC 14.2.1 20241116] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import picos
>>> import cvxopt
>>> 

EDIT:

To be sure, I also tried with python3Packages:

{ pkgs ? import <nixpkgs> { }, }:
pkgs.callPackage ({mkShell, python3Packages}:
  mkShell {
    # libraries
    buildInputs = [
      python3Packages.picos
    ];
  }
) { }

Could it be related to how the python path is handled? See e.g. #194

[Mic92]

You can compare your devshell inside nixpkgs-review with your own it's in the current directory.

[Mic92]

Multiple python version maybe cause issues, but it's unclear to me how to handle this use case.

[tobiasBora]

Multiple python version maybe cause issues, but it's unclear to me how to handle this use case.

Oh, I think this is indeed the origin of the problem, as pointed by the other thread. In fact, just doing:

$ nixpkgs-review pr 398351 -p python3Packages.picos

is enough to have a working python, since it will build picos for a single python version.

But if we do simply

$ nixpkgs-review pr 398351

then it tries to build picos for 2 python versions at once (3.12 and 3.13 here), and then it gets confused.

After a bit of search, I could go a bit further: the PYTHONPATH only contains stuff for python 3.13:

$ echo $PYTHONPATH
/nix/store/qmsllz8m16lc6s2kph2skz4z9g670qdb-python3.13-picos-2.6.1/lib/python3.13/site-packages:/nix/store/7krvqbjqra9d9mciv65a4viz0y3ibqll-python3.13-numpy-2.2.4/lib/python3.13/site-packages:/nix/store/xhq29h77qh5rhiwgcbirq1vy8j432qpr-python3-3.13.2/lib/python3.13/site-packages:/nix/store/rk119pj16f652yld3bhn5v6m4fa0a8ks-python3.13-cvxopt-1.3.2/lib/python3.13/site-packages:/nix/store/jmwqcly2dwq5pjdk28v84c5d0xl41qck-python3.13-scipy-1.15.2/lib/python3.13/site-packages

while python runs the 3.12 version:

$ which python
/nix/store/fqm9bqqlmaqqr02qbalm1bazp810qfiw-python3-3.12.9/bin/python

However, if instead of running the 3.12 version I run the 3.13, it works out of the box!

$ nixpkgs-review pr 398351
[nix-shell:~/.cache/nixpkgs-review/pr-398351-10]$ python3.13
Python 3.13.2 (main, Feb  4 2025, 14:51:09) [GCC 14.2.1 20250322] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import picos
>>> 

Unsurprisingly, running instead the 3.12 version fails.

Now, when creating a shell with:

{ pkgs ? import <nixpkgs> { }, }:
pkgs.callPackage ({mkShell, python3Packages}:
  mkShell {
    # libraries
    buildInputs = [
      python3Packages.picos
    ];
  }
) { }

the resulting python uses PYTHONPATH's env var to determine the available libraries…

So one (non-working) idea could be to put in the PYTHONPATH the path of both libraries… Sadly, putting both versions in the PYTHONPATH fails, since if we put both, python will take the first library in the path that matches the library name, and then error if the python versions between the library and the binary is not correct:

[nix-shell:~/.cache/nixpkgs-review/pr-398351-11]$ echo $PYTHONPATH
/nix/store/qmsllz8m16lc6s2kph2skz4z9g670qdb-python3.13-picos-2.6.1/lib/python3.13/site-packages:/nix/store/7krvqbjqra9d9mciv65a4viz0y3ibqll-python3.13-numpy-2.2.4/lib/python3.13/site-packages:/nix/store/xhq29h77qh5rhiwgcbirq1vy8j432qpr-python3-3.13.2/lib/python3.13/site-packages:/nix/store/rk119pj16f652yld3bhn5v6m4fa0a8ks-python3.13-cvxopt-1.3.2/lib/python3.13/site-packages:/nix/store/jmwqcly2dwq5pjdk28v84c5d0xl41qck-python3.13-scipy-1.15.2/lib/python3.13/site-packages

[nix-shell:~/.cache/nixpkgs-review/pr-398351-11]$ export PYTHONPATH=/nix/store/7sm3aahsf0790arpbfjkpv8c86dhc26g-python3.12-picos-2.6.0/lib/python3.12/site-packages:/nix/store/s3k7qby931y3hc7b2phvyay054idkfcg-python3.12-numpy-2.2.3/lib/python3.12/site-packages:/nix/store/f2krmq3iv5nibcvn4rw7nrnrciqprdkh-python3-3.12.9/lib/python3.12/site-packages:/nix/store/sm0zn0db7m2x28inkddknlqr96ahr7n8-python3.12-cvxopt-1.3.2/lib/python3.12/site-packages:/nix/store/kci42v10i2gxw47p7hx2m5jbp8xjskpg-python3.12-scipy-1.15.2/lib/python3.12/site-packages:$PYTHONPATH

[nix-shell:~/.cache/nixpkgs-review/pr-398351-11]$ echo $PYTHONPATH
/nix/store/7sm3aahsf0790arpbfjkpv8c86dhc26g-python3.12-picos-2.6.0/lib/python3.12/site-packages:/nix/store/s3k7qby931y3hc7b2phvyay054idkfcg-python3.12-numpy-2.2.3/lib/python3.12/site-packages:/nix/store/f2krmq3iv5nibcvn4rw7nrnrciqprdkh-python3-3.12.9/lib/python3.12/site-packages:/nix/store/sm0zn0db7m2x28inkddknlqr96ahr7n8-python3.12-cvxopt-1.3.2/lib/python3.12/site-packages:/nix/store/kci42v10i2gxw47p7hx2m5jbp8xjskpg-python3.12-scipy-1.15.2/lib/python3.12/site-packages:/nix/store/qmsllz8m16lc6s2kph2skz4z9g670qdb-python3.13-picos-2.6.1/lib/python3.13/site-packages:/nix/store/7krvqbjqra9d9mciv65a4viz0y3ibqll-python3.13-numpy-2.2.4/lib/python3.13/site-packages:/nix/store/xhq29h77qh5rhiwgcbirq1vy8j432qpr-python3-3.13.2/lib/python3.13/site-packages:/nix/store/rk119pj16f652yld3bhn5v6m4fa0a8ks-python3.13-cvxopt-1.3.2/lib/python3.13/site-packages:/nix/store/jmwqcly2dwq5pjdk28v84c5d0xl41qck-python3.13-scipy-1.15.2/lib/python3.13/site-packages

[nix-shell:~/.cache/nixpkgs-review/pr-398351-11]$ python
Python 3.12.9 (main, Feb  4 2025, 14:38:38) [GCC 14.2.1 20250322] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import picos
>>> 

[nix-shell:~/.cache/nixpkgs-review/pr-398351-11]$ python3.13
Python 3.13.2 (main, Feb  4 2025, 14:51:09) [GCC 14.2.1 20250322] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import picos
Traceback (most recent call last):
  File "<python-input-0>", line 1, in <module>
    import picos
  File "/nix/store/7sm3aahsf0790arpbfjkpv8c86dhc26g-python3.12-picos-2.6.0/lib/python3.12/site-packages/picos/__init__.py", line 43, in <module>
    from . import settings, uncertain  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/7sm3aahsf0790arpbfjkpv8c86dhc26g-python3.12-picos-2.6.0/lib/python3.12/site-packages/picos/uncertain.py", line 27, in <module>
    from .expressions.uncertain import (IntractableWorstCase,  # noqa
    ...<4 lines>...
                                        WassersteinAmbiguitySet)
  File "/nix/store/7sm3aahsf0790arpbfjkpv8c86dhc26g-python3.12-picos-2.6.0/lib/python3.12/site-packages/picos/expressions/__init__.py", line 28, in <module>
    from .algebra import *  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/7sm3aahsf0790arpbfjkpv8c86dhc26g-python3.12-picos-2.6.0/lib/python3.12/site-packages/picos/expressions/algebra.py", line 27, in <module>
    import cvxopt
  File "/nix/store/sm0zn0db7m2x28inkddknlqr96ahr7n8-python3.12-cvxopt-1.3.2/lib/python3.12/site-packages/cvxopt/__init__.py", line 57, in <module>
    import cvxopt.base
ModuleNotFoundError: No module named 'cvxopt.base'
>>> 

On the other hand, if we use python3.withPackages (instead of installing the libraries one by one) like here:

{ pkgs ? import <nixpkgs> { }, }:
pkgs.callPackage ({mkShell, python3Packages, python3}:
  mkShell {
    # libraries
    buildInputs = [
      (python3.withPackages (ps: [ps.picos]))
    ];
  }
) { }

then PYTHONPATH is not even set:

$ nix-shell
$ echo $PYTHONPATH

$ python
Python 3.12.9 (main, Feb  4 2025, 14:38:38) [GCC 14.2.1 20241116] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import picos
>>> 

Conclusion

So I can think of the following solutions:

• Review each version separately, like nixpkgs-review pr 398351 -p python312Packages.picos then nixpkgs-review pr 398351 -p python313Packages.picos. This works (just need to run nixpkgs-review one time for each python version), but it should probably be documented that testing python libraries requires some testing like this since we can't anymore just run nixpkgs-review pr XXX and expect things to work.
• Review all packages at once like nixpkgs-review pr 398351 -p "(python312.withPackages (ps: [ps.picos]))" -p "(python313.withPackages (ps: [ps.picos]))": This fails with an error from nixpkgs: These packages do not exist: (python312.withPackages (ps: [ps.picos])) (python313.withPackages (ps: [ps.picos])). Couldn't nixpkgs-review allow expressions like this? Just created a separate issue in Allow expressions within -p #485
• quick and dirty: maybe a hopefully simple to implement fix to make nixpkgs-review work out of the box would be to make python point to the version that is contained in PYTHONPATH, this way users are less confused when trying libraries.
• Modify python so that it can deal with PYTHONPATH containing libraries from multiple versions: not implemented: not sure how easy it is to do, certainly need to write a wrapper for python that would keep only the valid libraries in the path… certainly something to do directly in nixpkgs.
• Detect directly in nixpkgs-review that we build python libraries, and run therefore pythonX.withPackages (…) + unset PYTHONPATH. This may help to also solve Error when python lib in multiple python version is in PATH #194 that seem to have bugs due to PYTHONPATH. This seems to be the cleanest option (as it may also fix Error when python lib in multiple python version is in PATH #194), but no idea how easy it is to implement…

[wolfgangwalther]

This problem would also appear when I start a regular nix-shell with multiple python versions, right?

I wonder whether fixing nixpkgs to support multiple PYTHONPATH environment variables, like PYTHON312PATH and PYTHON313PATH and then wrapping python to read those and map them to PYTHONPATH could work?

[Mic92]

python.withPackages might be a solution but not a complete one because also buildPythonApplication can leak PYTHONPATH. Not sure if python.withPackages will ignore PYTHONPATH.