Installing pyasdf alongside pandas and pytables on a cluster example

Installing pyasdf alongside pandas/pytables with parallel IO enabled can be a little tricky, because pytables has an h5py dependency that might break the link to the parallel h5py dependency of pyasdf. If not installed carefully, obspy issue #1677 can pop up... although I'm not exactly sure where it comes from.

It took a little bit of searching and trial and error to install pyasdf with pandas/pytables on the cluster at the USC Centre for High Performance Computing, so I want to document what worked for me. I hope this will help someone else out.

To start, I am using openmpi v1.8.8, so I source the setup file for this:

[malcolcw@hpc-login2 malcolcw]$ source /usr/usc/openmpi/default/setup.sh

and there is a build of HDF5 v1.8.18 with parallel features enabled, so I add this to my environment:

[malcolcw@hpc-login2 malcolcw]$ source /usr/usc/hdf5/1.8.18/parallel/setup.sh

Then I create a new conda environment with python 2.7.14 and activate it:

[malcolcw@hpc-login2 malcolcw]$ conda create --name py27 -c conda-forge python=2.7
[malcolcw@hpc-login2 malcolcw]$ source activate py27

Next I install some pyasdf 0.4.0 dependencies via conda - per installation instructions:

[malcolcw@hpc-login2 malcolcw]$ conda install -c conda-forge obspy colorama pytest pip flake8 dill prov

and mpi4py via pip:

[malcolcw@hpc-login2 malcolcw]$ pip install mpi4py

Now I build h5py against the HDF5 library provided on the cluster, using pip:

[malcolcw@hpc-login2 malcolcw]$ export CC=mpicc
[malcolcw@hpc-login2 malcolcw]$ export HDF5_MPI="ON"
[malcolcw@hpc-login2 malcolcw]$ pip install --no-binary=h5py --no-deps h5py

I like to use pandas and pandas.HDFStore so first I build numexpr and pandas:

[malcolcw@hpc-login2 malcolcw]$ pip install --no-binary=numexpr --no-deps numexpr
[malcolcw@hpc-login2 malcolcw]$ pip install --no-binary=pandas --no-deps pandas

and then I build pytables against the correct HDF5 library:

[malcolcw@hpc-login2 malcolcw]$ export HDF5_DIR=/usr/usc/hdf5/1.8.18/parallel
[malcolcw@hpc-login2 malcolcw]$ pip install --no-binary=tables --no-deps tables

Finally I install pyasdf:

[malcolcw@hpc-login2 malcolcw]$ pip install pyasdf

At the end I am left with the following packages in my environment,

name: py27
channels:
  - conda-forge
  - defaults
dependencies:
  - asn1crypto=0.24.0=py27_0
  - attrs=17.4.0=py_0
  - backports=1.0=py27_1
  - backports.functools_lru_cache=1.5=py27_0
  - backports_abc=0.5=py27_0
  - ca-certificates=2018.1.18=0
  - certifi=2018.1.18=py27_0
  - cffi=1.11.5=py27_0
  - chardet=3.0.4=py27_0
  - colorama=0.3.9=py27_0
  - configparser=3.5.0=py27_0
  - cryptography=2.2.1=py27_0
  - cycler=0.10.0=py27_0
  - dbus=1.10.22=0
  - decorator=4.2.1=py27_0
  - dill=0.2.7.1=py27_0
  - enum34=1.1.6=py27_1
  - expat=2.2.5=0
  - flake8=3.5.0=py27_0
  - fontconfig=2.12.6=0
  - freetype=2.8.1=0
  - funcsigs=1.0.2=py_2
  - functools32=3.2.3.2=py27_2
  - future=0.16.0=py27_0
  - futures=3.2.0=py27_0
  - gettext=0.19.8.1=0
  - glib=2.55.0=0
  - gst-plugins-base=1.8.0=0
  - gstreamer=1.8.0=1
  - html5lib=1.0.1=py_0
  - icu=58.2=0
  - idna=2.6=py27_1
  - ipaddress=1.0.18=py27_0
  - isodate=0.6.0=py_0
  - jpeg=9b=2
  - keepalive=0.5=py27_0
  - kiwisolver=1.0.1=py27_1
  - libffi=3.2.1=3
  - libiconv=1.15=0
  - libpng=1.6.34=0
  - libxcb=1.13=0
  - libxml2=2.9.8=0
  - libxslt=1.1.32=0
  - lxml=4.2.1=py27_0
  - matplotlib=2.2.2=py27_1
  - mccabe=0.6.1=py27_0
  - mkl_fft=1.0.1=py27_1
  - mkl_random=1.0.1=py27_0
  - mock=2.0.0=py27_0
  - more-itertools=4.1.0=py_0
  - ncurses=5.9=10
  - networkx=2.1=py27_0
  - obspy=1.1.0=py27_1
  - openssl=1.0.2n=0
  - pbr=4.0.0=py27_0
  - pcre=8.41=1
  - pip=9.0.3=py27_0
  - pluggy=0.6.0=py_0
  - prov=1.5.2=py27_0
  - py=1.5.3=py_0
  - pycodestyle=2.3.1=py27_0
  - pycparser=2.18=py27_0
  - pydotplus=2.0.2=py27_0
  - pyflakes=1.6.0=py27_0
  - pyopenssl=17.5.0=py27_1
  - pyparsing=2.2.0=py27_0
  - pyqt=5.6.0=py27_4
  - pysocks=1.6.8=py27_1
  - pytest=3.5.0=py27_0
  - python=2.7.14=5
  - python-dateutil=2.7.2=py_0
  - pytz=2018.3=py_0
  - qt=5.6.2=7
  - rdflib=4.2.2=py27_0
  - readline=7.0=0
  - requests=2.18.4=py27_1
  - setuptools=39.0.1=py27_0
  - singledispatch=3.4.0.3=py27_0
  - sip=4.18=py27_1
  - six=1.11.0=py27_1
  - sparqlwrapper=1.8.0=py27_0
  - sqlalchemy=1.2.5=py27_0
  - sqlite=3.20.1=2
  - subprocess32=3.2.7=py27_0
  - tk=8.6.7=0
  - tornado=5.0.1=py27_1
  - urllib3=1.22=py27_0
  - webencodings=0.5=py27_0
  - wheel=0.30.0=py27_2
  - xorg-libxau=1.0.8=3
  - xorg-libxdmcp=1.1.2=3
  - xz=5.2.3=0
  - zlib=1.2.11=0
  - intel-openmp=2018.0.0=8
  - libgcc-ng=7.2.0=hdf63c60_3
  - libgfortran-ng=7.2.0=hdf63c60_3
  - libstdcxx-ng=7.2.0=hdf63c60_3
  - mkl=2018.0.2=1
  - numpy=1.14.2=py27hdbf6ddf_1
  - scipy=1.0.1=py27hfc37229_0
  - pip:
    - h5py==2.7.1
    - mpi4py==3.0.0
    - numexpr==2.6.4
    - pandas==0.22.0
    - pyasdf==0.4.0
    - tables==3.4.2