Skip to content

Building ScyllaDB

Jump to bottom
aborkar-ibm edited this page Dec 22, 2020 · 39 revisions

Building ScyllaDB

The instructions provided below specify the steps to build ScyllaDB version 4.2.1 on Linux on IBM Z for the following distributions:

  • RHEL 7.8, 7.9, 8.1, 8.2, 8.3
  • Ubuntu 18.04

Ensure that the kernel you are using is the latest available.

General Notes:

  • ScyllaDB and its dependencies are large projects and take a significant amount of time to build. The build can be sped up on large systems by increasing the -j parameter passed to make and/or ninja. The build was tested on a machine with 8 vCPUs and 32GB RAM.
  • Building ScyllaDB takes a large amount of disk space. Ensure you have at least 75GB available on the filesystem you are using. Significantly more space will be necessary if you wish to add debug information to the test binaries.
  • A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.
  • An installation directory /<prefix>/ is referred to in these instructions, by default this is /usr/local.
  • A target architecture /<target>/ is referred to in these instructions - this is the value passed to the GCC compiler.

1. Build using script

If you want to build ScyllaDB manually, go to STEP 9.

Use the following commands to build ScyllaDB using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/ScyllaDB/4.2.1/build_scylladb.sh

bash build_scylladb.sh [-t] [-y] -z <target>

Provide -t option for executing build with tests.
Provide -y to install required packages without confirmation. Provide -z option to select target architecture for build e.g. z13, z14

If the build completes successfully, go to STEP 6. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

2. Install Dependencies

export SOURCE_ROOT=/<source_root>/
export TARGET=native                      # z13, z14, native
export PREFIX=/usr/local                  # /<prefix>/

2.1 Prepare for installation

  • RHEL 7.8, 7.9

    Downgrade broken glibc version if installed.

    sudo yum list installed glibc-2.17-307.el7.1.s390
if [[ $? ]]; then
 sudo yum downgrade -y glibc glibc-common
 sudo yum downgrade -y krb5-libs
 sudo yum downgrade -y libss e2fsprogs-libs e2fsprogs libcom_err
 sudo yum downgrade -y libselinux-utils libselinux-python libselinux
fi

2.2 Install Packages

  • RHEL 7.x

    sudo yum install -y java-1.8.0-openjdk-devel libaio-devel \
 systemtap-sdt-devel lksctp-tools-devel xfsprogs-devel snappy-devel \
 libyaml-devel openssl-devel libevent-devel \
 libtasn1-devel libmpcdec lz4-devel \
 libatomic libatomic_ops-devel perl-devel \
 automake make git gcc gcc-c++ maven \
 unzip bzip2 wget curl xz-devel texinfo \
 libffi-devel hwloc-devel libpciaccess-devel libxml2-devel \
 libtool diffutils libtool-ltdl-devel trousers-devel \
 libunistring-devel libicu-devel readline-devel \
 lua-devel patch systemd-devel valgrind-devel

  • RHEL 8.x

    sudo yum install -y gcc gcc-c++ libatomic libatomic_ops-devel \
 java-1.8.0-openjdk-devel \
 lksctp-tools-devel xfsprogs-devel snappy-devel \
 libyaml-devel openssl-devel libevent-devel \
 libtasn1-devel libmpcdec \
 libidn2-devel numactl-devel c-ares-devel \
 gnutls-devel gnutls-c++ gnutls-dane \
 perl-devel \
 python38 python38-devel python38-pip python38-PyYAML \
 python38-setuptools python38-requests \
 make automake git maven ant ninja-build \
 unzip bzip2 wget curl xz-devel texinfo \
 libffi-devel hwloc-devel libpciaccess-devel libxml2-devel \
 libtool diffutils libtool-ltdl-devel trousers-devel p11-kit-devel \
 libunistring-devel libicu-devel readline-devel \
 lua-devel patch systemd-devel valgrind-devel

  • Ubuntu 18.04

    sudo apt-get update >/dev/null
sudo apt-get install -y software-properties-common
sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
sudo apt-get update >/dev/null
sudo apt-get install -y --no-install-recommends gcc-10 g++-10
sudo apt-get install -y openjdk-8-jdk libaio-dev \
 systemtap-sdt-dev lksctp-tools xfsprogs \
 libyaml-dev openssl libevent-dev \
 libmpfr-dev libmpcdec-dev \
 libssl1.0-dev libsystemd-dev libhwloc-dev \
 libsctp-dev libsnappy-dev libpciaccess-dev libxml2-dev xfslibs-dev \
 libgnutls28-dev libiconv-hook-dev mpi-default-dev libbz2-dev \
 libxslt-dev libjsoncpp-dev libc-ares-dev \
 libprotobuf-dev protobuf-compiler libcrypto++-dev \
 libtool perl ant libffi-dev \
 automake make git maven ninja-build \
 unzip bzip2 wget curl xz-utils texinfo \
 diffutils liblua5.3-dev libnuma-dev libunistring-dev \
 pigz ragel rapidjson-dev stow patch locales valgrind

3. Build GCC 10 and binutils:

ScyllaDB requires GCC 10 or higher.

  • On RHEL 7.x, 8.x:
  cd "$SOURCE_ROOT"
  URL=http://ftpmirror.gnu.org/binutils/binutils-2.34.tar.gz
  curl -sSL $URL | tar xzf - || error "binutils 10.2.0"
  cd binutils-2.34
  mkdir objdir
  cd objdir

  CC=/usr/bin/gcc ../configure --prefix=${PREFIX} --build=s390x-linux-gnu
  make -j 8
  sudo make install

  cd "$SOURCE_ROOT"
  URL=https://ftpmirror.gnu.org/gcc/gcc-10.2.0/gcc-10.2.0.tar.gz
  curl -sSL $URL | tar xzf - || error "GCC 10.2.0"

  cd gcc-10.2.0
  ./contrib/download_prerequisites
  mkdir objdir
  cd objdir

  ../configure --enable-languages=c,c++ --prefix=${PREFIX} \
    --enable-shared --enable-threads=posix \
    --disable-multilib --disable-libmpx \
    --with-system-zlib --with-long-double-128 --with-arch=zEC12 \
    --disable-libphobos --disable-werror \
    --build=s390x-linux-gnu --host=s390x-linux-gnu --target=s390x-linux-gnu

  make -j 8 bootstrap
  sudo make install

  export PATH=${PREFIX}/bin${PATH:+:${PATH}}

  LD_LIBRARY_PATH=${PREFIX}/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
  LD_LIBRARY_PATH+=:${PREFIX}/lib
  LD_LIBRARY_PATH+=:/usr/lib64
  export LD_LIBRARY_PATH

  LD_RUN_PATH=${PREFIX}/lib64${LD_RUN_PATH:+:${LD_RUN_PATH}}
  LD_RUN_PATH+=:${PREFIX}/lib
  LD_RUN_PATH+=:/usr/lib64
  export LD_RUN_PATH

  export CC=${PREFIX}/bin/gcc
  export CXX=${PREFIX}/bin/g++
  • On Ubuntu 18.04:
  export PATH=${PREFIX}/bin${PATH:+:${PATH}}

  LD_LIBRARY_PATH=${PREFIX}/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
  LD_LIBRARY_PATH+=:${PREFIX}/lib
  LD_LIBRARY_PATH+=:/usr/lib64
  export LD_LIBRARY_PATH

  LD_RUN_PATH=${PREFIX}/lib64${LD_RUN_PATH:+:${LD_RUN_PATH}}
  LD_RUN_PATH+=:${PREFIX}/lib
  LD_RUN_PATH+=:/usr/lib64
  export LD_RUN_PATH

  export CC=/usr/bin/gcc-10
  export CXX=/usr/bin/g++-10

4. Build and install common dependencies for all distributions

  • Python 3.8 and the required modules:
  cd "$SOURCE_ROOT"
  URL="https://www.python.org/ftp/python/3.8.6/Python-3.8.6.tgz"
  curl -sSL $URL | tar xzf -
  cd Python-3.8.6
  ./configure
  make
  sudo make install
  pip3 install --user --upgrade pip
  pip3 install --user pyparsing colorama pyyaml cassandra-driver boto3 requests pytest
  • Cmake 3.17.4:
  cd "$SOURCE_ROOT"
  URL=https://github.com/Kitware/CMake/releases/download/v3.17.4/cmake-3.17.4.tar.gz
  curl -sSL $URL | tar xzf -
  cd cmake-3.17.4
  ./bootstrap
  make
  sudo make install
  • lz4 1.9.3:
  cd "$SOURCE_ROOT"
  URL=https://github.com/lz4/lz4/archive/v1.9.3.tar.gz
  curl -sSL $URL | tar xzf - || error "lz4 1.9.3"
  cd lz4-1.9.3
  sudo make install
  • xxHash 0.8.0:
  cd "$SOURCE_ROOT"
  URL=https://github.com/Cyan4973/xxHash/archive/v0.8.0.tar.gz
  curl -sSL $URL | tar xzf - || error "xxHash 0.8.0"
  cd xxHash-0.8.0
  sudo make install
  • zstd 1.4.5:
  cd "$SOURCE_ROOT"
  URL=https://github.com/facebook/zstd/releases/download/v1.4.5/zstd-1.4.5.tar.gz
  curl -sSL $URL | tar xzf - || error "zstd 1.4.5"
  cd zstd-1.4.5
  URL=https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/ScyllaDB/4.2.1/patch/
  curl -sSL ${URL}/zstd.diff | patch -p1 || error "zstd.diff"
  cd lib
  make
  sudo make install

5. Build and install additional dependencies for RHEL 7.x and 8.x:

  • Ragel
  cd "$SOURCE_ROOT"
  URL=http://www.colm.net/files/ragel/ragel-6.10.tar.gz
  curl -sSL $URL | tar xzf -
  cd ragel-6.10
  ./configure
  make -j 8
  sudo make install
  • cryptopp
  cd "$SOURCE_ROOT"
  mkdir cryptopp
  cd cryptopp
  curl -skSLO https://www.cryptopp.com/cryptopp820.zip
  unzip cryptopp820.zip
  CXXFLAGS="-std=c++11 -g -O2" make
  sudo make install
  • jsoncpp
  cd "$SOURCE_ROOT"
  URL=https://github.com/open-source-parsers/jsoncpp/archive/1.7.7.tar.gz
  curl -sSL $URL | tar xzf -
  cd jsoncpp-1.7.7
  mkdir -p build/release
  cd build/release
  /usr/local/bin/cmake ../..
  make -j 8
  sudo make install
  • Protocolbuffers
  cd "$SOURCE_ROOT"
  git clone https://github.com/protocolbuffers/protobuf.git
  cd protobuf
  git checkout v3.11.2
  ./autogen.sh
  ./configure
  make
  sudo make install
  • RapidJSon
  cd "$SOURCE_ROOT"
  git clone https://github.com/Tencent/rapidjson.git
  cd rapidjson
  git checkout v1.1.0
  sudo cp -r ./include/rapidjson ${PREFIX}/include
  • stow
  cd "$SOURCE_ROOT"
  URL=http://ftpmirror.gnu.org/gnu/stow/stow-2.3.1.tar.gz
  curl -sSL $URL | tar xzf - || error "Stow 2.3.1"
  cd stow-2.3.1
  ./configure
  sudo make install

6. Build and install additional dependencies for RHEL 7.x:

  • Ninja
  cd "$SOURCE_ROOT"
  curl -sSLO https://github.com/ninja-build/ninja/archive/v1.10.1.zip
  unzip v1.10.1.zip
  cd ninja-1.10.1
  ./configure.py --bootstrap
  sudo cp ninja ${PREFIX}/bin
  • libidn2
  cd ${SOURCE_ROOT}
  URL=https://ftp.gnu.org/gnu/libidn/libidn2-2.3.0.tar.gz
  curl -sSL $URL | tar xzf -
  cd libidn2-2.3.0
  ./configure --disable-doc --disable-gtk-doc
  make
  sudo make install
  • numactl
  cd ${SOURCE_ROOT}
  git clone https://github.com/numactl/numactl.git
  cd numactl
  git checkout v2.0.14
  ./autogen.sh
  ./configure
  make
  sudo make install
  • c-ares
  cd ${SOURCE_ROOT}
  URL=https://c-ares.haxx.se/download/c-ares-1.15.0.tar.gz
  curl -sSL $URL | tar xzf -
  cd c-ares-1.15.0
  ./configure
  make
  sudo make install
  • Ant
  cd "$SOURCE_ROOT"
  URL=https://downloads.apache.org/ant/binaries/apache-ant-1.10.9-bin.tar.gz
  curl -sSL $URL | tar xzf -
  export ANT_HOME="$SOURCE_ROOT/apache-ant-1.10.9"
  export PATH=$PATH:"$ANT_HOME/bin"
  • LUA
  cd "$SOURCE_ROOT"
  URL=http://www.lua.org/ftp/lua-5.3.5.tar.gz
  curl -sSL $URL | tar xzf -
  cd lua-5.3.5
  make linux
  sudo make install
  • p11-kit
  ver=0.23.21
  cd "$SOURCE_ROOT"
  URL=https://github.com/p11-glue/p11-kit/releases/download/0.23.21/p11-kit-0.23.21.tar.xz
  curl -sSL $URL | tar xJf - || error "p11-kit 0.23.21"
  cd p11-kit-0.23.21
  ./configure --prefix=${PREFIX}
  make
  sudo make install
  • gmplib
  cd "$SOURCE_ROOT"
  URL=https://gmplib.org/download/gmp/gmp-6.2.1.tar.xz
  curl -sSL $URL | tar xJf - || error "gmplib 6.2.1"
  cd gmp-6.2.1
  ./configure --prefix=${PREFIX}
  make
  make check
  sudo make install
  • nettle
  cd "$SOURCE_ROOT"
  URL=https://ftp.gnu.org/gnu/nettle/nettle-3.6.tar.gz
  curl -sSL $URL | tar xzf - || error "nettle 3.6"
  cd nettle-3.6
  ./configure --prefix=${PREFIX}
  make
  make check
  sudo make install
  • gnutls
  cd "$SOURCE_ROOT"
  URL=https://www.gnupg.org/ftp/gcrypt/gnutls/v3.6/gnutls-3.6.15.tar.xz
  curl -sSL $URL | tar xJf - || error "gnutls 3.6.15"
  cd gnutls-3.6.15
  PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}} ./configure --prefix=${PREFIX}
  make
  make check
  sudo make install

7. Build and install further dependencies common to all distributions

  • Antlr
  cd "$SOURCE_ROOT"

  URL=https://github.com/antlr/antlr3/archive/3.5.2.tar.gz
  curl -sSL $URL | tar xzf -
  cd antlr3-3.5.2
  URL=https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/ScyllaDB/4.2.1/patch/
  curl -sSL ${URL}/antlr3.diff | patch -p1 || error "antlr3.diff"

  sudo cp runtime/Cpp/include/antlr3* ${PREFIX}/include/

  cd antlr-complete
  MAVEN_OPTS="-Xmx4G" mvn
  echo 'java -cp '"$(pwd)"'/target/antlr-complete-3.5.2.jar org.antlr.Tool $@' | sudo tee ${PREFIX}/bin/antlr3
  sudo chmod +x ${PREFIX}/bin/antlr3
  • Boost

Set the build environment

On RHEL 7.x, 8.x: TOOLSET=gcc

On Ubuntu 18.04: TOOLSET=gcc-10

  CENV=(PATH=$PATH LD_LIBRARY_PATH=$LD_LIBRARY_PATH LD_RUN_PATH=$LD_RUN_PATH CC=$CC CXX=$CXX)

  cd "$SOURCE_ROOT"
  URL=https://dl.bintray.com/boostorg/release/1.74.0/source/boost_1_74_0.tar.gz
  curl -sSL $URL | tar xzf -
  cd boost_1_74_0

  sed -i 's/array\.hpp/array_wrapper.hpp/g' boost/numeric/ublas/matrix.hpp
  sed -i 's/array\.hpp/array_wrapper.hpp/g' boost/numeric/ublas/storage.hpp

  ./bootstrap.sh

  options=( toolset=$TOOLSET variant=release link=shared runtime-link=shared threading=multi --without-python )

  ./b2 ${options[@]} stage
  sudo ${CENV[@]} ./b2 ${options[@]} install
  • Thrift
  cd "$SOURCE_ROOT"
  URL=http://archive.apache.org/dist/thrift/0.13.0/thrift-0.13.0.tar.gz
  curl -sSL $URL | tar xzf -
  cd thrift-0.13.0
  ./configure --without-java --without-lua --without-go --disable-tests --disable-tutorial
  make -j 8
  sudo make install
  • fmt
  cd "$SOURCE_ROOT"
  git clone https://github.com/fmtlib/fmt.git
  cd fmt
  git checkout 6.2.1
  mkdir build
  cd build
  /usr/local/bin/cmake -DFMT_TEST=OFF -DCMAKE_CXX_STANDARD=17 ..
  make
  sudo make install
  • yaml-cpp
  cd "$SOURCE_ROOT"
  URL=https://github.com/jbeder/yaml-cpp/archive/yaml-cpp-0.6.3.tar.gz
  curl -sSL $URL | tar xzf -
  cd yaml-cpp-yaml-cpp-0.6.3
  mkdir build
  cd build
  /usr/local/bin/cmake ..
  make
  sudo make install

8. Build ScyllaDB

8.1 On Ubuntu 18.04:

  sudo locale-gen en_US.UTF-8
  export LC_ALL=C
  unset LANGUAGE

8.2 Checkout and Patch

The following commands will clone the repository for v4.2.1, initialize its submodules and apply the patches.

  cd "$SOURCE_ROOT"
  git clone https://github.com/scylladb/scylla.git
  cd scylla
  git checkout scylla-4.2.1
  git submodule update --init --recursive

  URL=https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/ScyllaDB/4.2.1/patch/                                                     

  curl -sSL ${URL}/seastar.diff | patch -d seastar -p1 || echo "Error"
  curl -sSL ${URL}/scylla.diff | patch -p1 || echo "Error"

8.3 Configure and compile ScyllaDB

The following commands will build a release binary that is optimized for, and will only run, on the specified TARGET:

  export PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig:${PKG_CONFIG_PATH}
  export PATH=$PATH:~/.local/bin/
  
  cflags="-I${PREFIX}/include -I${PREFIX}/include/boost -L${PREFIX}/lib -L${PREFIX}/lib64 "

  ./configure.py --mode release --target ${TARGET} --debuginfo 1 \
    --static-thrift --cflags "${cflags}" --ldflags="-Wl,--build-id=sha1" \
    --compiler "${CXX}" --c-compiler "${CC}"

  ninja -j 8

9. Test ScyllaDB

Once ScyllaDB has been built you can run the tests. By default the tests will not contain debug information. To enable debug information add --tests-debuginfo 1 to the configure.py parameters above. This will make the build significantly larger (10s of gigabytes).

  cd "$SOURCE_ROOT/scylla"
  ./test.py --mode release

10. Performance Tuning

The performance tuning scripts that ship with ScyllaDB don't fully support LinuxONE at this stage. To maximize performance receive packet steering (RPS) should be enabled on network adapters.

11. Run ScyllaDB

The scylla binary can be found at /<source_root>/scylla/build/release/scylla.

$SOURCE_ROOT/scylla/build/release/scylla --help

For more information on how to run and test ScyllaDB see the links below.

References

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Clone this wiki locally