Building ScyllaDB

The instructions provided below specify the steps to build ScyllaDB version 4.4.1 on Linux on IBM Z for the following distributions:

RHEL (7.8, 7.9, 8.1, 8.2, 8.3)
Ubuntu (18.04, 20.04)

Ensure that the kernel you are using is the latest available.

General Notes:

ScyllaDB and its dependencies are large projects and take a significant amount of time to build. The build can be sped up on large systems by increasing the -j parameter passed to make and/or ninja. The build was tested on a machine with 8 vCPUs and 32GB/16GB RAM. When building on a machine with 16GB RAM or less, a large swap file is needed.
Building ScyllaDB takes a large amount of disk space. Ensure you have at least 75GB available on the filesystem you are using. Significantly more space will be necessary if you wish to add debug information to the test binaries.
A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.
An installation directory /<prefix>/ is referred to in these instructions, by default this is /usr/local.
A target architecture /<target>/ is referred to in these instructions - this is the value passed to the GCC compiler.

1. Build using script

If you want to build ScyllaDB manually, go to STEP 9.

Use the following commands to build ScyllaDB using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/ScyllaDB/4.4.1/build_scylladb.sh

bash build_scylladb.sh [-t] [-y] -z <target>

Provide -t option for executing build with tests.
Provide -y to install required packages without confirmation. Provide -z option to select target architecture for build e.g. z13, z14

If the build completes successfully, go to STEP 6. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

2. Install Dependencies

export SOURCE_ROOT=/<source_root>/
export TARGET=native                      # z13, z14, native
export PREFIX=/usr/local                  # /<prefix>/

2.1 Prepare for installation

RHEL (7.8, 7.9)

Downgrade broken glibc version if installed.

sudo yum list installed glibc-2.17-307.el7.1.s390
if [[ $? ]]; then
 sudo yum downgrade -y glibc glibc-common
 sudo yum downgrade -y krb5-libs
 sudo yum downgrade -y libss e2fsprogs-libs e2fsprogs libcom_err
 sudo yum downgrade -y libselinux-utils libselinux-python libselinux
fi

2.2 Install Packages

RHEL 7.x

sudo yum install -y java-1.8.0-openjdk-devel libaio-devel \
 systemtap-sdt-devel lksctp-tools-devel xfsprogs-devel snappy-devel \
 libyaml-devel openssl-devel libevent-devel \
 libtasn1-devel libmpcdec lz4-devel \
 libatomic libatomic_ops-devel perl-devel \
 automake make git gcc gcc-c++ maven \
 unzip bzip2 wget curl xz-devel texinfo \
 libffi-devel libpciaccess-devel libxml2-devel \
 libtool diffutils libtool-ltdl-devel trousers-devel \
 libunistring-devel libicu-devel readline-devel \
 lua-devel patch systemd-devel valgrind-devel

RHEL 8.x

sudo yum install -y gcc gcc-c++ libatomic libatomic_ops-devel \
 java-1.8.0-openjdk-devel \
 lksctp-tools-devel xfsprogs-devel snappy-devel \
 libyaml-devel openssl-devel libevent-devel \
 libtasn1-devel libmpcdec \
 libidn2-devel numactl-devel c-ares-devel \
 gnutls-devel gnutls-c++ gnutls-dane \
 perl-devel \
 python38 python38-devel python38-pip python38-PyYAML \
 python38-setuptools python38-requests \
 make automake git maven ant ninja-build \
 unzip bzip2 wget curl xz-devel texinfo \
 libffi-devel libpciaccess-devel libxml2-devel \
 libtool diffutils libtool-ltdl-devel trousers-devel p11-kit-devel \
 libunistring-devel libicu-devel readline-devel \
 lua-devel patch systemd-devel valgrind-devel

Ubuntu 18.04, 20.04

sudo apt-get update >/dev/null
sudo apt-get install -y software-properties-common
sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
sudo apt-get update >/dev/null
sudo apt-get install -y --no-install-recommends gcc g++
sudo apt-get install -y openjdk-8-jdk libaio-dev \
 systemtap-sdt-dev lksctp-tools xfsprogs \
 libyaml-dev openssl libevent-dev \
 libmpfr-dev libmpcdec-dev \
 libssl-dev libsystemd-dev \
 libsctp-dev libsnappy-dev libpciaccess-dev libxml2-dev xfslibs-dev \
 libgnutls28-dev libiconv-hook-dev liblzma-dev libbz2-dev \
 libxslt-dev libjsoncpp-dev libc-ares-dev \
 libprotobuf-dev protobuf-compiler libcrypto++-dev \
 libtool perl ant libffi-dev \
 automake make git maven ninja-build \
 unzip bzip2 wget curl xz-utils texinfo \
 diffutils liblua5.3-dev libnuma-dev libunistring-dev \
 pigz ragel rapidjson-dev stow patch locales valgrind libudev-dev

3. Build GCC 10 and binutils:

ScyllaDB v4.4.1 requires GCC 10.1.1 or higher.

  cd "$SOURCE_ROOT"
  URL=http://ftpmirror.gnu.org/binutils/binutils-2.36.tar.gz
  curl -sSL $URL | tar xzf - || error "binutils 2.36"
  cd binutils-2.36
  mkdir objdir
  cd objdir

  CC=/usr/bin/gcc ../configure --prefix=${PREFIX} --build=s390x-linux-gnu
  make -j 8
  sudo make install

  cd "$SOURCE_ROOT"
  URL=https://ftp.gnu.org/gnu/gcc/gcc-10.2.0/gcc-10.2.0.tar.gz
  curl -sSL $URL | tar xzf - || error "GCC 10.2.0"

  cd gcc-10.2.0
  ./contrib/download_prerequisites
  mkdir objdir
  cd objdir

  ../configure --enable-languages=c,c++ --prefix=${PREFIX} \
    --enable-shared --enable-threads=posix \
    --disable-multilib --disable-libmpx \
    --with-system-zlib --with-long-double-128 --with-arch=zEC12 \
    --disable-libphobos --disable-werror \
    --build=s390x-linux-gnu --host=s390x-linux-gnu --target=s390x-linux-gnu

  make -j 8 bootstrap
  sudo make install

  export PATH=${PREFIX}/bin${PATH:+:${PATH}}

  LD_LIBRARY_PATH=${PREFIX}/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
  LD_LIBRARY_PATH+=:${PREFIX}/lib
  LD_LIBRARY_PATH+=:/usr/lib64
  export LD_LIBRARY_PATH

  LD_RUN_PATH=${PREFIX}/lib64${LD_RUN_PATH:+:${LD_RUN_PATH}}
  LD_RUN_PATH+=:${PREFIX}/lib
  LD_RUN_PATH+=:/usr/lib64
  export LD_RUN_PATH

  export CC=${PREFIX}/bin/gcc
  export CXX=${PREFIX}/bin/g++

4. Setup Python environment

Build Python 3.8 (only on RHEL 7.x, Ubuntu(18.04, 20.04)):

  cd "$SOURCE_ROOT"
  URL="https://www.python.org/ftp/python/3.8.6/Python-3.8.6.tgz"
  curl -sSL $URL | tar xzf -
  cd Python-3.8.6
  ./configure
  make
  sudo make install

Update symblic link for Python 3.8 (only on RHEL 8.3):

  sudo update-alternatives --set python3 "/usr/bin/python3.8"

Install the required modules for Python 3.8:

  pip3 install --user --upgrade pip
  pip3 install --user pyparsing colorama pyyaml cassandra-driver boto3 requests pytest

5. Build and install common dependencies for all distributions

Cmake:

  cd "$SOURCE_ROOT"
  URL=https://github.com/Kitware/CMake/releases/download/v3.17.4/cmake-3.17.4.tar.gz
  curl -sSL $URL | tar xzf -
  cd cmake-3.17.4
  ./bootstrap
  make
  sudo make install

xxHash:

  cd "$SOURCE_ROOT"
  URL=https://github.com/Cyan4973/xxHash/archive/v0.8.0.tar.gz
  curl -sSL $URL | tar xzf - || error "xxHash 0.8.0"
  cd xxHash-0.8.0
  sudo make install

zstd:

  cd "$SOURCE_ROOT"
  URL=https://github.com/facebook/zstd/releases/download/v1.4.5/zstd-1.4.5.tar.gz
  curl -sSL $URL | tar xzf - || error "zstd 1.4.5"
  cd zstd-1.4.5
  URL=https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/ScyllaDB/4.4.1/patch/
  curl -sSL ${URL}/zstd.diff | patch -p1 || error "zstd.diff"
  cd lib
  make
  sudo make install

cryptopp

  cd "$SOURCE_ROOT"
  mkdir cryptopp
  cd cryptopp
  curl -ksSLO https://github.com/weidai11/cryptopp/archive/refs/tags/CRYPTOPP_8_2_0.zip
  unzip CRYPTOPP_8_2_0.zip
  cd cryptopp-CRYPTOPP_8_2_0
  CXXFLAGS="-std=c++11 -g -O2" make
  sudo make install

hwloc

  cd "$SOURCE_ROOT"
  URL=https://download.open-mpi.org/release/hwloc/v2.4/hwloc-2.4.1.tar.gz
  curl -sSL $URL | tar xzf - || error "hwloc 2.4.1"
  cd hwloc-2.4.1
  ./configure
  sudo make install

6. Build and install additional dependencies(only on RHEL 8.x and Ubuntu(18.04, 20.04)):

lz4

  cd "$SOURCE_ROOT"
  URL=https://github.com/lz4/lz4/archive/v1.9.3.tar.gz
  curl -sSL $URL | tar xzf - || error "lz4 1.9.3"
  cd lz4-1.9.3
  sudo make install

7. Build and install additional dependencies(only on RHEL 7.x and 8.x):

Ragel

  cd "$SOURCE_ROOT"
  URL=http://www.colm.net/files/ragel/ragel-6.10.tar.gz
  curl -sSL $URL | tar xzf -
  cd ragel-6.10
  ./configure
  make -j 8
  sudo make install

jsoncpp

  cd "$SOURCE_ROOT"
  URL=https://github.com/open-source-parsers/jsoncpp/archive/1.7.7.tar.gz
  curl -sSL $URL | tar xzf -
  cd jsoncpp-1.7.7
  mkdir -p build/release
  cd build/release
  /usr/local/bin/cmake ../..
  make -j 8
  sudo make install

Protocolbuffers

  cd "$SOURCE_ROOT"
  git clone https://github.com/protocolbuffers/protobuf.git
  cd protobuf
  git checkout v3.11.2
  ./autogen.sh
  ./configure
  make
  sudo make install

RapidJSon

  cd "$SOURCE_ROOT"
  git clone https://github.com/Tencent/rapidjson.git
  cd rapidjson
  git checkout v1.1.0
  sudo cp -r ./include/rapidjson ${PREFIX}/include

stow

  cd "$SOURCE_ROOT"
  URL=http://ftpmirror.gnu.org/gnu/stow/stow-2.3.1.tar.gz
  curl -sSL $URL | tar xzf - || error "Stow 2.3.1"
  cd stow-2.3.1
  ./configure
  sudo make install

8. Build and install additional dependencies(only on RHEL 7.x):

Ninja

  cd "$SOURCE_ROOT"
  curl -sSLO https://github.com/ninja-build/ninja/archive/v1.10.1.zip
  unzip v1.10.1.zip
  cd ninja-1.10.1
  ./configure.py --bootstrap
  sudo cp ninja ${PREFIX}/bin

libidn2

  cd ${SOURCE_ROOT}
  URL=https://ftp.gnu.org/gnu/libidn/libidn2-2.3.0.tar.gz
  curl -sSL $URL | tar xzf -
  cd libidn2-2.3.0
  ./configure --disable-doc --disable-gtk-doc
  make
  sudo make install

numactl

  cd ${SOURCE_ROOT}
  git clone https://github.com/numactl/numactl.git
  cd numactl
  git checkout v2.0.14
  ./autogen.sh
  ./configure
  make
  sudo make install

c-ares

  cd ${SOURCE_ROOT}
  URL=https://c-ares.haxx.se/download/c-ares-1.15.0.tar.gz
  curl -sSL $URL | tar xzf -
  cd c-ares-1.15.0
  ./configure
  make
  sudo make install

Ant

  cd "$SOURCE_ROOT"
  URL=https://downloads.apache.org/ant/binaries/apache-ant-1.10.9-bin.tar.gz
  curl -sSL $URL | tar xzf -
  export ANT_HOME="$SOURCE_ROOT/apache-ant-1.10.9"
  export PATH=$PATH:"$ANT_HOME/bin"

LUA

  cd "$SOURCE_ROOT"
  URL=http://www.lua.org/ftp/lua-5.3.5.tar.gz
  curl -sSL $URL | tar xzf -
  cd lua-5.3.5
  make linux
  sudo make install

p11-kit

  ver=0.23.21
  cd "$SOURCE_ROOT"
  URL=https://github.com/p11-glue/p11-kit/releases/download/0.23.21/p11-kit-0.23.21.tar.xz
  curl -sSL $URL | tar xJf - || error "p11-kit 0.23.21"
  cd p11-kit-0.23.21
  ./configure --prefix=${PREFIX}
  make
  sudo make install

gmplib

  cd "$SOURCE_ROOT"
  URL=https://gmplib.org/download/gmp/gmp-6.2.1.tar.xz
  curl -sSL $URL | tar xJf - || error "gmplib 6.2.1"
  cd gmp-6.2.1
  ./configure --prefix=${PREFIX}
  make
  make check
  sudo make install

nettle

  cd "$SOURCE_ROOT"
  URL=https://ftp.gnu.org/gnu/nettle/nettle-3.6.tar.gz
  curl -sSL $URL | tar xzf - || error "nettle 3.6"
  cd nettle-3.6
  ./configure --prefix=${PREFIX}
  make
  make check
  sudo make install

gnutls

  cd "$SOURCE_ROOT"
  URL=https://www.gnupg.org/ftp/gcrypt/gnutls/v3.6/gnutls-3.6.15.tar.xz
  curl -sSL $URL | tar xJf - || error "gnutls 3.6.15"
  cd gnutls-3.6.15
  PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}} ./configure --prefix=${PREFIX}
  make
  make check
  sudo make install

9. Build and install further dependencies common to all distributions

Antlr

  cd "$SOURCE_ROOT"

  URL=https://github.com/antlr/antlr3/archive/3.5.2.tar.gz
  curl -sSL $URL | tar xzf -
  cd antlr3-3.5.2
  URL=https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/ScyllaDB/4.4.1/patch/
  curl -sSL ${URL}/antlr3.diff | patch -p1 || error "antlr3.diff"

  sudo cp runtime/Cpp/include/antlr3* ${PREFIX}/include/

  cd antlr-complete
  MAVEN_OPTS="-Xmx4G" mvn
  echo 'java -cp '"$(pwd)"'/target/antlr-complete-3.5.2.jar org.antlr.Tool $@' | sudo tee ${PREFIX}/bin/antlr3
  sudo chmod +x ${PREFIX}/bin/antlr3

Boost

Set up the build environment

  TOOLSET=gcc
  CENV=(PATH=$PATH LD_LIBRARY_PATH=$LD_LIBRARY_PATH LD_RUN_PATH=$LD_RUN_PATH CC=$CC CXX=$CXX)

  cd "$SOURCE_ROOT"
  URL=https://boostorg.jfrog.io/artifactory/main/release/1.74.0/source/boost_1_74_0.tar.gz
  curl -sSL $URL | tar xzf -
  cd boost_1_74_0

  sed -i 's/array\.hpp/array_wrapper.hpp/g' boost/numeric/ublas/matrix.hpp
  sed -i 's/array\.hpp/array_wrapper.hpp/g' boost/numeric/ublas/storage.hpp

  ./bootstrap.sh

  options=( toolset=$TOOLSET variant=release link=shared runtime-link=shared threading=multi --without-python )

  ./b2 ${options[@]} stage
  sudo ${CENV[@]} ./b2 ${options[@]} install

Thrift

  cd "$SOURCE_ROOT"
  URL=http://archive.apache.org/dist/thrift/0.13.0/thrift-0.13.0.tar.gz
  curl -sSL $URL | tar xzf -
  cd thrift-0.13.0
  ./configure --without-java --without-lua --without-go --disable-tests --disable-tutorial
  make -j 8
  sudo make install

fmt

  cd "$SOURCE_ROOT"
  git clone https://github.com/fmtlib/fmt.git
  cd fmt
  git checkout 6.2.1
  mkdir build
  cd build
  /usr/local/bin/cmake -DFMT_TEST=OFF -DCMAKE_CXX_STANDARD=17 ..
  make
  sudo make install

yaml-cpp

  cd "$SOURCE_ROOT"
  URL=https://github.com/jbeder/yaml-cpp/archive/yaml-cpp-0.6.3.tar.gz
  curl -sSL $URL | tar xzf -
  cd yaml-cpp-yaml-cpp-0.6.3
  mkdir build
  cd build
  /usr/local/bin/cmake ..
  make
  sudo make install

10. Build ScyllaDB

10.1 Adjust locale settings (only on Ubuntu(18.04, 20.04)):

  sudo locale-gen en_US.UTF-8
  export LC_ALL=C
  unset LANGUAGE

10.2 Checkout and Patch

The following commands will clone the repository for v4.4.1, initialize its submodules and apply the patches.

  cd "$SOURCE_ROOT"
  git clone https://github.com/scylladb/scylla.git
  cd scylla
  git checkout scylla-4.4.1
  git submodule update --init --force --recursive

  URL=https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/ScyllaDB/4.4.1/patch/                                                     

  curl -sSL ${URL}/seastar.diff | patch -d seastar -p1 || echo "Error"
  curl -sSL ${URL}/scylla.diff | patch -p1 || echo "Error"

10.3 Configure and compile ScyllaDB

The following commands will build a release binary that is optimized for, and will only run, on the specified TARGET:

  export PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig:${PKG_CONFIG_PATH}
  export PATH=$PATH:~/.local/bin/
  
  #Fix warning about deprecated boost/function_output_iterator.hpp (Optional) 
  sudo sed -i 's/boost\/function_output_iterator\.hpp/boost\/iterator\/function_output_iterator\.hpp/g' \
    $PREFIX/include/boost/signals2/detail/null_output_iterator.hpp

  cflags="-I${PREFIX}/include -I${PREFIX}/include/boost -L${PREFIX}/lib -L${PREFIX}/lib64 -fcoroutines "

  ./configure.py --mode="release" --target="${TARGET}" --debuginfo=1 \
    --static-thrift --cflags="${cflags}" --ldflags="-Wl,--build-id=sha1 " \
    --compiler="${CXX}" --c-compiler="${CC}"

  ninja build -j 8

Increase the request capacity in /proc/sys/fs/aio-max-nr for setting up Async I/O (only on Ubuntu 20.04):

echo "fs.aio-max-nr = 1048576" |& sudo tee /etc/sysctl.conf
sudo sysctl -p

11. Test ScyllaDB

Once ScyllaDB has been built you can run the tests. By default the tests will not contain debug information. To enable debug information add --tests-debuginfo 1 to the configure.py parameters above. This will make the build significantly larger (10s of gigabytes).

  cd "$SOURCE_ROOT/scylla"
  ./test.py --mode release

12. Performance Tuning

The performance tuning scripts that ship with ScyllaDB don't fully support LinuxONE at this stage. To maximize performance receive packet steering (RPS) should be enabled on network adapters.

13. Run ScyllaDB

The scylla binary can be found at /<source_root>/scylla/build/release/scylla.

$SOURCE_ROOT/scylla/build/release/scylla --help

For more information on how to run and test ScyllaDB see the links below.

References

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Building ScyllaDB

Building ScyllaDB

1. Build using script

2. Install Dependencies

2.1 Prepare for installation

2.2 Install Packages

3. Build GCC 10 and binutils:

4. Setup Python environment

5. Build and install common dependencies for all distributions

6. Build and install additional dependencies(only on RHEL 8.x and Ubuntu(18.04, 20.04)):

7. Build and install additional dependencies(only on RHEL 7.x and 8.x):

8. Build and install additional dependencies(only on RHEL 7.x):

9. Build and install further dependencies common to all distributions

10. Build ScyllaDB

10.1 Adjust locale settings (only on Ubuntu(18.04, 20.04)):

10.2 Checkout and Patch

10.3 Configure and compile ScyllaDB

11. Test ScyllaDB

12. Performance Tuning

13. Run ScyllaDB

References

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!