-
Notifications
You must be signed in to change notification settings - Fork 114
Installation FAQ (pip, setup.py)
This page covers questions about installing GalSim with pip or setup.py. For questions about installing with SCons, go here.
Sorry this page is so spare currently. This is a new installation process, so we're not sure yet what the "frequently-asked" questions will be. For now, see the installation instructions here
-
Dependencies
-
Installing with pip
-
Installing with setup.py
-
Installing with conda
-
Importing galsim after installation
When running pip install galsim, you may get the message:
****
Installation requires setuptools version >= 38.
Please upgrade or install with pip install -U setuptools
****
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/tmp/pip-build-a04km4cq/galsim/setup.py", line 30, in <module>
from setuptools.command.build_clib import build_clib
ImportError: No module named 'setuptools.command.build_clib'
The suggestion to run pip install -U setuptools is normally effective to get this working.
Sometimes you may run into an SSL certificate verify error. E.g. the following output:
Collecting galsim
Downloading https://files.pythonhosted.org/packages/f5/c9/eae469e73476b208701a7cb21a22daec61ec16cfb121e1abb328a9344e9f/GalSim-2.0.0rc1.tar.gz (36.4MB)
100% |████████████████████████████████| 36.4MB 33kB/s
Complete output from command python setup.py egg_info:
Download error on https://pypi.org/simple/eigency/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600) -- Some packages may not be found!
Couldn't find index page for 'eigency' (maybe misspelled?)
Download error on https://pypi.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600) -- Some packages may not be found!
No local packages or working download links found for eigency>=1.77
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/tmp/pip-build-7cpz0c4h/galsim/setup.py", line 864, in <module>
zip_safe=False,
File "/sw/lib/python3.4/site-packages/setuptools/__init__.py", line 128, in setup
_install_setup_requires(attrs)
File "/sw/lib/python3.4/site-packages/setuptools/__init__.py", line 123, in _install_setup_requires
dist.fetch_build_eggs(dist.setup_requires)
File "/sw/lib/python3.4/site-packages/setuptools/dist.py", line 514, in fetch_build_eggs
replace_conflicting=True,
File "/sw/lib/python3.4/site-packages/pkg_resources/__init__.py", line 770, in resolve
replace_conflicting=replace_conflicting
File "/sw/lib/python3.4/site-packages/pkg_resources/__init__.py", line 1053, in best_match
return self.obtain(req, installer)
File "/sw/lib/python3.4/site-packages/pkg_resources/__init__.py", line 1065, in obtain
return installer(requirement)
File "/sw/lib/python3.4/site-packages/setuptools/dist.py", line 581, in fetch_build_egg
return cmd.easy_install(req)
File "/sw/lib/python3.4/site-packages/setuptools/command/easy_install.py", line 667, in easy_install
raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('eigency>=1.77')
Using setuptools version 39.2.0
Python version = 3.4.3 (default, Sep 26 2015, 00:09:24)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
Adding eigency to build_dep
GalSim version is 2.0.0rc1
The result is that pip cannot automatically install the dependencies for you. So the solution is to just install each one by hand. In the above case, just pip install -U eigency. Then, in my case, that also failed to find its dependency, setuptools_scm. So I had to do pip install -U setuptools_scm. Etc.
There may be a different solution to fix the underlying problem with the SSL certificate, but manual installation of the dependencies worked, so I didn't investigate further.
Bryan Gillis reported the following error when trying to build GalSim: (The error actually happened when compiling the dependency, LSSTDESC/Coord, but the error is unrelated to the module being compiled.)
gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Iinclude -I/cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.0/usr/include/python3.6m -c src/Angle.cpp -o build/temp.linux-x86_64-3.6/src/Angle.o
g++ -pthread -shared -Wl,-z,relro build/temp.linux-x86_64-3.6/src/Angle.o -L/usr/lib64 -lpython3.6m -o build/lib.linux-x86_64-3.6/coord/_coord.cpython-36m-x86_64-linux-gnu.so
/usr/bin/ld: cannot find -lpython3.6m
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1
The problem was that the library libpython3.6m.so was not in his LIBRARY_PATH. The directory with this library needs to be in both LIBRARY_PATH and LD_LIBRARY_PATH. The former is where the linker looks at build time, and the latter is where the system looks at run time. So you need to have both of these system variables set properly.
The solution is to make sure both of the following are set properly, usually in your .bash_profile. In his case, this looked like:
export LIBRARY_PATH=$LIBARY_PATH:/cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.0/usr/lib64/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/cvmfs/euclid-dev.in2p3.fr/CentOS7/EDEN-2.0/usr/lib64/
Obviously, you should replace the directory name abouve with wherever your system has the libpython*.so library file.
If you build with sudo and then later build without sudo, this can happen. The easiest resolution is probably to sudo rm -r ~/.ccache to blow away the current cache (with sudo) and start over.
David Kirkby reported the following problem using pip install galsim with Anaconda Python:
Collecting galsim
Downloading https://files.pythonhosted.org/packages/96/a1/4da265850d10f13b0aa99fa20db13a5ee81865fb8e07e7544fd69a3d0c5a/GalSim-2.0.2.tar.gz (36.8MB)
100% |████████████████████████████████| 36.8MB 737kB/s
Missing build time requirements in pyproject.toml for galsim from https://files.pythonhosted.org/packages/96/a1/4da265850d10f13b0aa99fa20db13a5ee81865fb8e07e7544fd69a3d0c5a/GalSim-2.0.2.tar.gz#sha256=659ba3d3b0c4fe2b77361822ce89ee1ed6a454fe1a8ed14f7a3e752b52304996: 'wheel'.
This version of pip does not implement PEP 517 so it cannot build a wheel without 'setuptools' and 'wheel'.
Installing build dependencies ... done
Complete output from command python setup.py egg_info:
warning: no files found matching 'eigency/*.pyd'
warning: no previously-included files found matching 'eigency/*.cpp'
warning: no previously-included files matching 'CMakeLists.txt' found under directory 'eigency/eigen_3.2.8/Eigen'
...snip...
distutils.errors.DistutilsError: Setup script exited with error: SandboxViolation: mkdir('/private/var/var', 511) {}
The package setup script has attempted to modify files on your system
that are not within the EasyInstall build area, and has been aborted.
This package cannot be safely installed by EasyInstall, and may not
support alternate installation locations even if you run its setup
script by hand. Please inform the package's author and the EasyInstall
maintainers to find out if a fix or workaround is available.
Using setuptools version 40.0.0
Python version = 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:14:59)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
Adding eigency to build_dep
GalSim version is 2.0.2
This seems to be an error either in eigency or setuptools wherein eigency tries to write some temporary files in the normal place (e.g. either /tmp or /private/var), and setuptools doesn't allow it.
This might not be conda-specific, but with conda there is a particularly easy workaround:
conda install eigen
to use its Eigen distribution rather than relying on eigency. Then run pip install galsim again.
David Kirkby reported the following problem using pip install galsim with Anaconda Python:
Installing build dependencies ... done
Complete output from command python setup.py egg_info:
Using setuptools version 40.0.0
Python version = 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:14:59)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
GalSim version is 2.0.2
running egg_info
creating pip-egg-info/GalSim.egg-info
writing pip-egg-info/GalSim.egg-info/PKG-INFO
writing dependency_links to pip-egg-info/GalSim.egg-info/dependency_links.txt
writing entry points to pip-egg-info/GalSim.egg-info/entry_points.txt
writing requirements to pip-egg-info/GalSim.egg-info/requires.txt
writing top-level names to pip-egg-info/GalSim.egg-info/top_level.txt
writing manifest file 'pip-egg-info/GalSim.egg-info/SOURCES.txt'
error: Could not find PyBind11
Include directory for fftw3 is /usr/local/include
PyBind11 is version 2.2.3
Looking for pybind11 header files:
/Users/david/.local/include/python3.6m (no)
/Users/david/anaconda3/envs/desi/include/python3.6m (no)
Could not find pybind11 header files.
They should have been in one of the following two locations:
/Users/david/.local/include/python3.6m
/Users/david/anaconda3/envs/desi/include/python3.6m
I'm not sure why pybind11 doesn't report its own location properly sometimes when installed with pip in a conda distribution. However, using the conda-forge installation instead usually seems to work:
conda install -c conda-forge pybind11
``
Then repeat `pip install galsim`
#### `ImportError: dlopen(.../site-packages/galsim/_galsim.cpython-36m-darwin.so, 2): Symbol not found`
David Kirkby [reported](https://github.com/GalSim-developers/GalSim/issues/978#issuecomment-405236614) the following problem importing galsim after installing with `pip install galsim` with Anaconda Python:
ImportError: dlopen(/Users/david/anaconda3/envs/desi/lib/python3.6/site-packages/galsim/_galsim.cpython-36m-darwin.so, 2): Symbol not found: __ZN3tmv14BandMatrixViewINSt3__17complexIdEELi0EE13conjugateSelfEv Referenced from: /usr/local/lib/libgalsim.1.3.dylib Expected in: /usr/local/lib/libtmv_symband.0.dylib in /usr/local/lib/libgalsim.1.3.dylib
This can happen when you have previously installed GalSim 1.x with the old SCons installation and are now installing GalSim 2.x with pip or setup.py. If the old GalSim libraries are in your LIBRARY_PATH, then they can sometimes be linked rather than the new 2.0 library.
The easiest solution is to manually remove these obsolete libraries and reinstall. E.g.
sudo rm /usr/local/lib/libgalsim* pip uninstall galsim pip install --no-cache-dir galsim
Note the `--no-cache-dir` option to make sure that pip actually rebuilds everything rather than just reusing the previous library with the bad linkage.
----
## 5. Importing galsim after installation
#### `ImportError: cannot import name '_galsim'` when importing galsim
a) Are you doing `import galsim` in the GalSim source directory?
If so, you might see something like the following:
import galsim Traceback (most recent call last): File "", line 1, in File "/Users/Mike/GalSim/galsim/init.py", line 96, in from .position import Position, PositionI, PositionD File "/Users/Mike/GalSim/galsim/position.py", line 22, in from . import _galsim ImportError: cannot import name '_galsim'
Python looks in the current directory first when importing something. So in this case, it sees the `galsim` directory and tries to import that rather than from the installed location. But the build process doesn't build the `_galsim.so` library there. (It's in a subdirectory of the `build` directory.) So you get this error.
The solution is to change to a different directory and running Python from there.
b) It may be finding the wrong installation of galsim.
Sometimes you can end up with multiple GalSim versions installed on your machine, and python can get confused about which one is the right one to use. Especially if you have switched among the pip, SCons, and setup.py installation methods.
Normally you can check which version of something in imported by checking e.g. `galsim.__file__`. But if it
cannot import, this method doesn't work. Instead, you need to do the following:
import imp imp.find_module('galsim') (None, '/Users/Mike/anaconda3/lib/python3.6/site-packages/galsim', ('', '', 5))
That second item in the tuple is the location that it is trying to import from when you do `import galsim`. If that doesn't match the location that you think galsim is installed, then try deleting that installation and trying again.
One easy way to make sure you only have one version of galsim installed is to do `pip uninstall galsim` repeatedly until it says there is no version of galsim installed. Then reinstall it once.
#### `ImportError: undefined symbol` when importing or using galsim.
a) This is another possible error if you have multiple versions of galsim installed and it is finding the wrong one at run time.
E.g. Niall MacCrann reported the following error:
import galsim Traceback (most recent call last): File "", line 1, in File "/global/project/projectdirs/des/maccrann/py3/lib/python3.6/site-packages/galsim/init.py", line 94, in from .position import PositionI, PositionD File "/global/project/projectdirs/des/maccrann/py3/lib/python3.6/site-packages/galsim/position.py", line 22, in from . import _galsim ImportError: /global/project/projectdirs/des/maccrann/py3/lib/python3.6/site-packages/galsim/_galsim.so: undefined symbol: _ZN6galsim7Silicon10accumulateIfEEdRKNS_11PhotonArrayENS_14UniformDeviateENS_9ImageViewIT_EENS_8PositionIiEE
He found that deleting the files in this directory and doing a clean install fixed it.
Sometimes, especially when switching from a SCons-based installation to a pip or setup.py installation, not all of the previous version's files are correctly removed or updated, so manual deleting may be required. Further updates should work fine if you keep using the same installation method.
b) Another possibility is that the `_galsim.so` file is correct, but it is linked to the wrong `libgalsim.so` library.
E.g. the following error:
import galsim Traceback (most recent call last): File "", line 1, in File "/Users/Mike/anaconda3/lib/python3.6/site-packages/galsim/init.py", line 96, in from .position import Position, PositionI, PositionD File "/Users/Mike/anaconda3/lib/python3.6/site-packages/galsim/position.py", line 22, in from . import _galsim ImportError: dlopen(/Users/Mike/anaconda3/lib/python3.6/site-packages/galsim/_galsim.cpython-36m-darwin.so, 2): Symbol not found: __ZN6galsim11invertImageISt7complexIdEEEvNS_9ImageViewIT_EE Referenced from: /Users/Mike/anaconda3/lib/python3.6/site-packages/galsim/_galsim.cpython-36m-darwin.so Expected in: flat namespace in /Users/Mike/anaconda3/lib/python3.6/site-packages/galsim/_galsim.cpython-36m-darwin.so
This probably means that there is some libgalsim.so file in your LIBRARY_PATH, which was found at compile time, probably from an older SCons installation procedure. If setup.py finds that at link time, it can link to that older library, rather than the `libgalsim.a` library that it just compiled.
The way to check is to use either `ldd` (on Unix) or `otool -L` (on Mac) to check what external libraries `_galsim.so` is linking to:
$ otool -L /Users/Mike/anaconda3/lib/python3.6/site-packages/galsim/_galsim.cpython-36m-darwin.so /Users/Mike/anaconda3/lib/python3.6/site-packages/galsim/_galsim.cpython-36m-darwin.so: /sw/lib/libgalsim.2.0.dylib (compatibility version 2.0.0, current version 2.0.0) /sw/lib/libfftw3.3.dylib (compatibility version 7.0.0, current version 7.2.0) @rpath/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.19.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0) /Users/Mike/anaconda3/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
The above is an example of an erroneous build. The `libgalsim` line should not be there, since we now statically link all the C++ code into the `_galsim.so` library.
The solution is to delete the file in the above location (`/Users/Mike/lib/libgalsim.2.0.dylib` in the above case), remove the GalSim/build directory, and rerun `python setup.py install`.
If you used pip to install, it might still have linked to this wrong library. To force pip to reinstall, you can do
pip uninstall galsim [possibly multiple times to make sure all versions are removed] pip --no-cache-dir install galsim
That should then rebuild and hopefully link properly.
In the above example, after I removed the file `/sw/lib/libgalsim.2.0.dylib`, the output of `otool` is:
$ otool -L build/*/galsim/_galsim.cpython-36m-darwin.so build/lib.macosx-10.7-x86_64-3.6/galsim/_galsim.cpython-36m-darwin.so: /sw/lib/libfftw3.3.dylib (compatibility version 7.0.0, current version 7.2.0) @rpath/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.19.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0) /Users/Mike/anaconda3/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
#### `Segmentation faults when running scripts using galsim`
Obviously, there are many possibilities here, but so far the main reason seems to be similar to the above ImportError problem. There may be some incompatibility between the `_galsim` library and the back-end `libgalsim` library. It may be that `import galsim` doesn't notice the incompatibility, but when the code tries to do something that uses the back-end C++ code, there is an incompatibility and you get a segmentation fault.
The solution is the same. Delete the old, now invalid libgalsim library, and rebuild GalSim.