pyzmq==27.0.1 error on AIX 7.2

[AAB94]

Hi all,
I have an AIX 7.2.0.0 setup where, using pyenv we install python 3.11.3, then we install pyzmq
If I run just import zmq I get an error, I even tried setting the compiler to IBM for C, C++ and using
pip install pyzmq --no-binary pyzmq --no-cache-dir, the result is the same.

(venv) bash-5.2$ python
Python 3.11.3 (main, Nov 22 2024, 16:34:57) [GCC 10.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>> import zmq
Illegal instruction (core dumped)

We use AIX open source tool box, to use dnf package manager to install gcc, g++ and other dependencies needed to run pyenv, deploy virtualenv, install python3.11.3

pyzmq==25.0.0 works fine without issues, but anything after fails

I have tried building libzmq from sources using C,C++ compiler on AIX and then linking it to a C,C++ sample code,
but the C/C++ binaries work fine.

[minrk]

Can you run an install with:

SKBUILD_BUILD_VERBOSE=1 python3 -m pip install -v --no-binary pyzmq --no-cache pyzmq

and share the output?

This is likely an issue with cmake on the system, since pyzmq doesn't pick the compilers or compiler flags at all, it trusts Python itself and cmake's FindPython to work.

You might check if the basic C and C++ examples from scikit-build-core work, too:

c:

SKBUILD_BUILD_VERBOSE=1 python3 -m pip install -v --ignore-installed 'git+https://github.com/scikit-build/scikit-build-core#egg=example&subdirectory=docs/examples/getting_started/c'

python3 -c 'import example; print(example.square(5))'

c++:

SKBUILD_BUILD_VERBOSE=1 python3 -m pip install -v --ignore-installed 'git+https://github.com/scikit-build/scikit-build-core#egg=example&subdirectory=docs/examples/getting_started/nanobind'

python3 -c 'import example; print(example.square(5))'

[AAB94]

Hi @minrk

Installing collected packages: example
  Running command git rev-parse HEAD
  bcdd6650ef6607c5e840c3b5d2e66783a3487155
Successfully installed example-0.0.1
(venv) bash-5.2$ python
Python 3.11.3 (main, Nov 22 2024, 16:34:57) [GCC 10.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import exmaple
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'exmaple'
>>> import example
Illegal instruction (core dumped)
(venv) bash-5.2$

oddly on AIX 7.3 we don't face the issue
I tried setting the CC and CXX to IBM and GCC 10 same result

[minrk]

Okay, this means that cmake (or Python in general) cannot build Python extensions, so this is not related to pyzmq.

Make sure to include the verbose output from the command I gave above to see the compilation commands that are executed, which might give you a hint. I don't know anything about AIX, so I can't really help identify what would be wrong, but it might help you or someone who know aix

[AAB94]

Using pip 22.3.1 from /path_to/venv/lib/python3.11/site-packages/pip (python 3.11)
Collecting example
  Cloning https://github.com/scikit-build/scikit-build-core to /tmp/pip-install-1exz5ol2/example_78d1e69adf884e21bc32170a4d9654a8
  Running command git version
  git version 2.41.0
  Running command git clone --filter=blob:none https://github.com/scikit-build/scikit-build-core /tmp/pip-install-1exz5ol2/example_78d1e69adf884e21bc32170a4d9654a8
  Cloning into '/tmp/pip-install-1exz5ol2/example_78d1e69adf884e21bc32170a4d9654a8'...
  Updating files:  57% (236/414)
  Updating files:  58% (241/414)
  Updating files:  59% (245/414)
  Updating files:  60% (249/414)
  Updating files:  61% (253/414)
  Updating files:  62% (257/414)
  Updating files:  63% (261/414)
  Updating files:  64% (265/414)
  Updating files:  65% (270/414)
  Updating files:  66% (274/414)
  Updating files:  67% (278/414)
  Updating files:  68% (282/414)
  Updating files:  69% (286/414)
  Updating files:  70% (290/414)
  Updating files:  71% (294/414)
  Updating files:  72% (299/414)
  Updating files:  73% (303/414)
  Updating files:  74% (307/414)
  Updating files:  75% (311/414)
  Updating files:  76% (315/414)
  Updating files:  77% (319/414)
  Updating files:  78% (323/414)
  Updating files:  79% (328/414)
  Updating files:  80% (332/414)
  Updating files:  81% (336/414)
  Updating files:  82% (340/414)
  Updating files:  83% (344/414)
  Updating files:  84% (348/414)
  Updating files:  85% (352/414)
  Updating files:  86% (357/414)
  Updating files:  87% (361/414)
  Updating files:  88% (365/414)
  Updating files:  89% (369/414)
  Updating files:  90% (373/414)
  Updating files:  91% (377/414)
  Updating files:  92% (381/414)
  Updating files:  93% (386/414)
  Updating files:  94% (390/414)
  Updating files:  95% (394/414)
  Updating files:  96% (398/414)
  Updating files:  97% (402/414)
  Updating files:  98% (406/414)
  Updating files:  99% (410/414)
  Updating files: 100% (414/414)
  Updating files: 100% (414/414), done.
  Running command git rev-parse HEAD
  bcdd6650ef6607c5e840c3b5d2e66783a3487155
  Resolved https://github.com/scikit-build/scikit-build-core to commit bcdd6650ef6607c5e840c3b5d2e66783a3487155
  Running command git rev-parse HEAD
  bcdd6650ef6607c5e840c3b5d2e66783a3487155
  Running command pip subprocess to install build dependencies
  Collecting scikit-build-core
    Using cached scikit_build_core-0.11.5-py3-none-any.whl (185 kB)
  Collecting packaging>=23.2
    Using cached packaging-25.0-py3-none-any.whl (66 kB)
  Collecting pathspec>=0.10.1
    Using cached pathspec-0.12.1-py3-none-any.whl (31 kB)
  Installing collected packages: pathspec, packaging, scikit-build-core
  Successfully installed packaging-25.0 pathspec-0.12.1 scikit-build-core-0.11.5
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Getting requirements to build wheel ... done
  Running command Preparing metadata (pyproject.toml)
  *** scikit-build-core 0.11.5 using CMake 3.30.3 (metadata_wheel)
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: example
  Running command Building wheel for example (pyproject.toml)
  *** scikit-build-core 0.11.5 using CMake 3.30.3 (wheel)
  *** Configuring CMake...
  loading initial cache file /tmp/tmppk7uzr3p/build/CMakeInit.txt
  -- The C compiler identification is GNU 10.3.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /opt/freeware/bin/gcc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Found Python: /opt/pyenv/versions/3.11.3/include/python3.11 (found version "3.11.3") found components: Development.Module
  -- Configuring done (0.9s)
  -- Generating done (0.0s)
  -- Build files have been written to: /tmp/tmppk7uzr3p/build
  *** Building project with Unix Makefiles...
  Change Dir: '/tmp/tmppk7uzr3p/build'

  Run Build Command(s): /opt/freeware/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile
  /opt/freeware/bin/cmake -S/tmp/pip-install-1exz5ol2/example_78d1e69adf884e21bc32170a4d9654a8/docs/examples/getting_started/c -B/tmp/tmppk7uzr3p/build --check-build-system CMakeFiles/Makefile.cmake 0
  /opt/freeware/bin/cmake -E cmake_progress_start /tmp/tmppk7uzr3p/build/CMakeFiles /tmp/tmppk7uzr3p/build//CMakeFiles/progress.marks
  /usr/bin/gmake  -f CMakeFiles/Makefile2 all
  gmake[1]: Entering directory '/tmp/tmppk7uzr3p/build'
  /usr/bin/gmake  -f CMakeFiles/example.dir/build.make CMakeFiles/example.dir/depend
  gmake[2]: Entering directory '/tmp/tmppk7uzr3p/build'
  cd /tmp/tmppk7uzr3p/build && /opt/freeware/bin/cmake -E cmake_depends "Unix Makefiles" /tmp/pip-install-1exz5ol2/example_78d1e69adf884e21bc32170a4d9654a8/docs/examples/getting_started/c /tmp/pip-install-1exz5ol2/example_78d1e69adf884e21bc32170a4d9654a8/docs/examples/getting_started/c /tmp/tmppk7uzr3p/build /tmp/tmppk7uzr3p/build /tmp/tmppk7uzr3p/build/CMakeFiles/example.dir/DependInfo.cmake "--color="
  gmake[2]: Leaving directory '/tmp/tmppk7uzr3p/build'
  /usr/bin/gmake  -f CMakeFiles/example.dir/build.make CMakeFiles/example.dir/build
  gmake[2]: Entering directory '/tmp/tmppk7uzr3p/build'
  [ 50%] Building C object CMakeFiles/example.dir/example.c.o
  /opt/freeware/bin/gcc -Dexample_EXPORTS -isystem /opt/pyenv/versions/3.11.3/include/python3.11 -maix64 -fPIC -O3 -DNDEBUG -fPIC -MD -MT CMakeFiles/example.dir/example.c.o -MF CMakeFiles/example.dir/example.c.o.d -o CMakeFiles/example.dir/example.c.o -c /tmp/pip-install-1exz5ol2/example_78d1e69adf884e21bc32170a4d9654a8/docs/examples/getting_started/c/example.c
  [100%] Linking C shared module example.so
  /opt/freeware/bin/cmake -E cmake_link_script CMakeFiles/example.dir/link.txt --verbose=1
  "/opt/freeware/share/cmake-3.30/Modules/Platform/AIX/ExportImportList" -o CMakeFiles/example.dir/exports.exp -c /opt/freeware/bin/gcc CMakeFiles/example.dir/example.c.o
  /opt/freeware/bin/gcc -fPIC -Wl,-bE:CMakeFiles/example.dir/exports.exp -maix64 -fPIC -O3 -DNDEBUG -Wl,-b,erok -maix64 -shared -Wl,-bnoipath  -o example.so CMakeFiles/example.dir/example.c.o  -Wl,-blibpath:/opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/10/ppc64:/opt/freeware/lib/ppc64:/opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/10:/opt/freeware/lib:/usr/lib:/lib:
  gmake[2]: Leaving directory '/tmp/tmppk7uzr3p/build'
  [100%] Built target example
  gmake[1]: Leaving directory '/tmp/tmppk7uzr3p/build'
  /opt/freeware/bin/cmake -E cmake_progress_start /tmp/tmppk7uzr3p/build/CMakeFiles 0

  *** Installing project into wheel...
  -- Install configuration: "Release"
  -- Installing: /tmp/tmppk7uzr3p/wheel/platlib/./example.so
  -- Set non-toolchain portion of runtime path of "/tmp/tmppk7uzr3p/wheel/platlib/./example.so" to "/opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/10/ppc64:/opt/freeware/lib/ppc64:/opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/10:/opt/freeware/lib:/usr/lib:/lib"
  *** Making wheel...
  *** Created example-0.0.1-cp311-cp311-aix_7204_9988_64.whl
  Building wheel for example (pyproject.toml) ... done
  Created wheel for example: filename=example-0.0.1-cp311-cp311-aix_7204_9988_64.whl size=2051 sha256=b50921367f461f7fc0a7ebbdae7faba8e238e32994e0f008a61eb72731d0ab6d
  Stored in directory: /tmp/pip-ephem-wheel-cache-ezi6xbnm/wheels/93/61/78/73ce2eb51a34ef254e73cc382ffe95d108e49f29f1e0957cac
Successfully built example
Installing collected packages: example
  Running command git rev-parse HEAD
  bcdd6650ef6607c5e840c3b5d2e66783a3487155
Successfully installed example-0.0.1

[minrk]

Thanks. I guess some of that compilation or linking is wrong, but I couldn't tell you what. You could try comparing it to the compilation and linking commands you get when you build something that does work.

[AAB94]

Hi @minrk

can we specify the compiler/linker to use when building pyzmq
or must be gcc

[minrk]

pyzmq doesn't express any requirements or preferences on the compiler, that would be set by Python, CMake, and/or the OS. Typically the standard $CC and $CXX environment variables are respected if you want to override the defaults. If pyzmq 25 didn't have this problem, it is likely the introduction of cmake that made the difference. You might test a basic cmake C++ hello world project if that alone can explain the issue.

[AAB94]

Hi @minrk
I have compiled/built libsodium and libzmq on AIX successfully, can you share the steps to build pyzmq using the libraries I built?
Also should libsodium be a static library that is linked statically to libzmq?

[AAB94]

Hi @minrk

I have set ZMQ_PREFIX="",
I download pyzmq source for 27.0.2 but after pyzmq is built I dont see any auth folder, I checked and zmq 4.3.5 curve_keygen tool works

[minrk]

can you share the steps to build pyzmq using the libraries I built?

The same steps I shared above should work. You can also find more detailed directions from the docs, but the main thing is to set ZMQ_PREFIX to where your libzmq is if pkg-config, etc. can't find it.

python3 -m pip install pyzmq --no-binary pyzmq

Also should libsodium be a static library that is linked statically to libzmq?

pyzmq doesn't link libsodium, so it doesn't matter. pyzmq only links to libzmq, which is what uses libsodium or not.

I download pyzmq source for 27.0.2 but after pyzmq is built I dont see any auth folder

I'm not sure what you mean by looking for an auth folder. Can you explain more directly what you are trying to do that doesn't work and what errors you see, ideally with specific code and tracebacks?

[AAB94]

Hi @minrk
Thanks for the reply and time let me clarify
I download this https://github.com/zeromq/pyzmq/archive/refs/tags/v27.0.2.tar.gz

cmake -DCMAKE_INSTALL_PREFIX=/<>/pyzmq -DPYTHON_EXECUTABLE=/<>/venv/bin -DPYTHON_LIBRARIES=/<>/venv/bin ..
cmake --build . --target install --config Release

This is python 3.11.3, the machine has gcc 10 installed.
venv has cython installed

pyzmq after its built when I open the folder has pyzmq/zmq/backend but does not have pyzmq/zmq/auth folders under it.
The sources that I downloaded has it, do they just need to be copied?

[minrk]

That's not how you install Python packages, you use python3 -m pip install ., as indicated in all of the examples of how to build and install pyzmq above and in the docs. You don't invoke cmake directly, The build backend (scikit-build-core) is responsible for that, and also responsible for installing the Python files, which cmake isn't related to.