Issues while installing mpi4py on linux based cluster

[rB080]

Hi, I am trying to run a pip install mpi4py but it throws off with an error. The exact error is here:

Defaulting to user installation because normal site-packages is not writeable
Collecting mpi4py
  Downloading mpi4py-3.1.4.tar.gz (2.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 18.3 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: mpi4py
  Building wheel for mpi4py (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for mpi4py (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [156 lines of output]
      running bdist_wheel
      running build
      running build_src
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-39
      creating build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/run.py -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/__init__.py -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/bench.py -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/__main__.py -> build/lib.linux-x86_64-cpython-39/mpi4py
      creating build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/server.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/_base.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/_lib.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/__init__.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/_core.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/aplus.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/pool.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/__main__.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      creating build/lib.linux-x86_64-cpython-39/mpi4py/util
      copying src/mpi4py/util/__init__.py -> build/lib.linux-x86_64-cpython-39/mpi4py/util
      copying src/mpi4py/util/pkl5.py -> build/lib.linux-x86_64-cpython-39/mpi4py/util
      copying src/mpi4py/util/dtlib.py -> build/lib.linux-x86_64-cpython-39/mpi4py/util
      copying src/mpi4py/py.typed -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/dl.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/run.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/__main__.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/__init__.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/bench.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/MPI.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/MPI.pxd -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/__init__.pxd -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/libmpi.pxd -> build/lib.linux-x86_64-cpython-39/mpi4py
      creating build/lib.linux-x86_64-cpython-39/mpi4py/include
      creating build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
      copying src/mpi4py/include/mpi4py/mpi4py.h -> build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
      copying src/mpi4py/include/mpi4py/mpi4py.MPI.h -> build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
      copying src/mpi4py/include/mpi4py/mpi4py.MPI_api.h -> build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
      copying src/mpi4py/include/mpi4py/mpi4py.i -> build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
      copying src/mpi4py/include/mpi4py/mpi.pxi -> build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
      copying src/mpi4py/futures/_lib.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/pool.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/server.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/__main__.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/aplus.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/__init__.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/_core.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/util/pkl5.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/util
      copying src/mpi4py/util/__init__.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/util
      copying src/mpi4py/util/dtlib.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/util
      running build_clib
      MPI configuration: [mpi] from 'mpi.cfg'
      MPI C compiler:    /share/apps/anaconda3-2022.05/bin/mpicc
      MPI C++ compiler:  /share/apps/anaconda3-2022.05/bin/mpicxx
      MPI F compiler:    /share/apps/anaconda3-2022.05/bin/mpifort
      MPI F90 compiler:  /share/apps/anaconda3-2022.05/bin/mpif90
      MPI F77 compiler:  /share/apps/anaconda3-2022.05/bin/mpif77
      checking for library 'lmpe' ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      building 'mpe' dylib library
      creating build/temp.linux-x86_64-cpython-39
      creating build/temp.linux-x86_64-cpython-39/src
      creating build/temp.linux-x86_64-cpython-39/src/lib-pmpi
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c src/lib-pmpi/mpe.c -o build/temp.linux-x86_64-cpython-39/src/lib-pmpi/mpe.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      warning: build_clib: command '/share/apps/anaconda3-2022.05/bin/mpicc' failed with exit code 127

      warning: build_clib: building optional library "mpe" failed

      checking for library 'vt-mpi' ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      checking for library 'vt.mpi' ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      building 'vt' dylib library
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c src/lib-pmpi/vt.c -o build/temp.linux-x86_64-cpython-39/src/lib-pmpi/vt.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      warning: build_clib: command '/share/apps/anaconda3-2022.05/bin/mpicc' failed with exit code 127

      warning: build_clib: building optional library "vt" failed

      checking for library 'vt-mpi' ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      checking for library 'vt.mpi' ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      building 'vt-mpi' dylib library
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c src/lib-pmpi/vt-mpi.c -o build/temp.linux-x86_64-cpython-39/src/lib-pmpi/vt-mpi.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      warning: build_clib: command '/share/apps/anaconda3-2022.05/bin/mpicc' failed with exit code 127

      warning: build_clib: building optional library "vt-mpi" failed

      checking for library 'vt-hyb' ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      checking for library 'vt.ompi' ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      building 'vt-hyb' dylib library
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c src/lib-pmpi/vt-hyb.c -o build/temp.linux-x86_64-cpython-39/src/lib-pmpi/vt-hyb.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      warning: build_clib: command '/share/apps/anaconda3-2022.05/bin/mpicc' failed with exit code 127

      warning: build_clib: building optional library "vt-hyb" failed

      running build_ext
      MPI configuration: [mpi] from 'mpi.cfg'
      MPI C compiler:    /share/apps/anaconda3-2022.05/bin/mpicc
      MPI C++ compiler:  /share/apps/anaconda3-2022.05/bin/mpicxx
      MPI F compiler:    /share/apps/anaconda3-2022.05/bin/mpifort
      MPI F90 compiler:  /share/apps/anaconda3-2022.05/bin/mpif90
      MPI F77 compiler:  /share/apps/anaconda3-2022.05/bin/mpif77
      checking for dlopen() availability ...
      checking for header 'dlfcn.h' ...
      gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -I/share/apps/anaconda3-2022.05/include/python3.9 -c _configtest.c -o _configtest.o
      success!
      removing: _configtest.c _configtest.o
      success!
      checking for library 'dl' ...
      gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -I/share/apps/anaconda3-2022.05/include/python3.9 -c _configtest.c -o _configtest.o
      gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat _configtest.o -Lbuild/temp.linux-x86_64-cpython-39 -ldl -o _configtest
      success!
      removing: _configtest.c _configtest.o _configtest
      checking for function 'dlopen' ...
      gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -I/share/apps/anaconda3-2022.05/include/python3.9 -c _configtest.c -o _configtest.o
      gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat _configtest.o -Lbuild/temp.linux-x86_64-cpython-39 -ldl -o _configtest
      success!
      removing: _configtest.c _configtest.o _configtest
      building 'mpi4py.dl' extension
      gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -DHAVE_DLFCN_H=1 -DHAVE_DLOPEN=1 -I/share/apps/anaconda3-2022.05/include/python3.9 -c src/dynload.c -o build/temp.linux-x86_64-cpython-39/src/dynload.o
      gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -shared -Wl,-rpath,/share/apps/anaconda3-2022.05/lib -Wl,-rpath-link,/share/apps/anaconda3-2022.05/lib -L/share/apps/anaconda3-2022.05/lib -L/share/apps/anaconda3-2022.05/lib -Wl,-rpath,/share/apps/anaconda3-2022.05/lib -Wl,-rpath-link,/share/apps/anaconda3-2022.05/lib -L/share/apps/anaconda3-2022.05/lib build/temp.linux-x86_64-cpython-39/src/dynload.o -Lbuild/temp.linux-x86_64-cpython-39 -ldl -o build/lib.linux-x86_64-cpython-39/mpi4py/dl.cpython-39-x86_64-linux-gnu.so
      checking for MPI compile and link ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -I/share/apps/anaconda3-2022.05/include/python3.9 -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      error: Cannot compile MPI programs. Check your configuration!!!
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for mpi4py
Failed to build mpi4py
ERROR: Could not build wheels for mpi4py, which is required to install pyproject.toml-based projects

This however works on conda, but even if I can see the package on conda list, it doesn't show up when I try importing it from python. Is there a fix for this? Is this a python version issue? I'm using python 3.10.

[wangxiaobo007]

same question

[m-bain]

Install with video2dataset, it's much faster

https://github.com/iejMac/video2dataset

[LymanY]

u can use conda instead of pip

[shinever22]

Hi~We are researching a project, and the Webvid-2.5M dataset is very important to us. We found that the link is no longer working. Could you please provide us with a new link?