Building Elasticsearch

Building Elasticsearch

The instructions provided below specify the steps to build Elasticsearch 7.17.7 on Linux on IBM Z for following distributions:

• RHEL (7.8, 7.9, 8.4, 8.6, 8.7, 9.0, 9.1)
• SLES (12 SP5, 15 SP4)
• Ubuntu (18.04, 20.04, 22.04)

General Notes

• When following the steps below please use a standard permission user unless otherwise specified.
• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Build and Install Elasticsearch

1) Build using script

If you want to build Elasticsearch using manual steps, go to STEP 2.

Use the following commands to build Elasticsearch using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/7.17.7/build_elasticsearch.sh

# Build Elasticsearch
bash build_elasticsearch.sh  [Provide -t option for executing build with tests]

If the build completes successfully, go to STEP 9. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

2) Install build dependencies

export SOURCE_ROOT=/<source_root>/

• RHEL (7.8, 7.9, 8.4, 8.6, 8.7, 9.0, 9.1)

  sudo yum install -y curl git gzip tar wget patch

• SLES (12 SP5, 15 SP4)

  sudo zypper install -y curl git gzip tar wget patch

• Ubuntu (18.04, 20.04, 22.04)

  sudo apt-get update
  sudo apt-get install -y curl git gzip tar wget patch locales
  sudo locale-gen en_US.UTF-8

3) Install Java

• With Eclipse Adoptium Temurin Runtime (previously known as AdoptOpenJDK hotspot)

  • Download and install Eclipse Adoptium Temurin Runtime (Java 17) from here.

• With OpenJDK 17

  • RHEL (8.4, 8.6, 8.7, 9.0, 9.1)

    sudo yum install -y java-17-openjdk-devel

  • SLES 15 SP4

    sudo zypper install -y java-17-openjdk java-17-openjdk-devel

  • Ubuntu (18.04, 20.04, 22.04)

    sudo apt-get install -y openjdk-17-jre openjdk-17-jdk ca-certificates-java

Note: At the time of creation of these build instructions, Elasticsearch was verified with Eclipse Adoptium Temurin Runtime (Java 17) version (build 17.0.5+8) and OpenJDK 17.

4) Set the environment variables

export LANG="en_US.UTF-8"
export JAVA_HOME=<Path to JDK>
export ES_JAVA_HOME=/<Path to JDK>/
export JAVA17_HOME=/<Path to JDK>/
export PATH=$ES_JAVA_HOME/bin:$PATH

Note: Ensure system locale is set up correctly for Elasticsearch to build without encoding errors.

5) Download Elasticsearch and apply patches

cd $SOURCE_ROOT
git clone https://github.com/elastic/elasticsearch
cd elasticsearch
git checkout v7.17.7

• Apply patches to create s390x distribution

 export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/7.17.7/patch/elasticsearch.patch"
 curl -o elasticsearch.patch $PATCH_URL
 git apply elasticsearch.patch

6) Build

cd $SOURCE_ROOT/elasticsearch
CPU_NUM="$(grep -c ^processor /proc/cpuinfo)"
./gradlew :distribution:archives:oss-linux-s390x-tar:assemble --max-workers="$CPU_NUM"  --parallel

7) Testing (Optional)

cd $SOURCE_ROOT/elasticsearch
export JAVA_TOOL_OPTIONS="-Dfile.encoding=UTF8"
export RUNTIME_JAVA_HOME=/<Path to JDK>/ # Java runtime to test against
export JAVA17_HOME=/<Path to JDK>/ # If Java runtime to test against is JDK 17
export JAVA11_HOME=/<Path to JDK>/ # If Java runtime to test against is JDK 11

./gradlew --continue test -Dtests.haltonfailure=false -Dtests.jvm.argline="-Xss2m"

Notes:

• You can set RUNTIME_JAVA_HOME optionally to the location of the JDK (another version of the JDK that act as the runtime) that you'd like to use during testing. Eclipse Adoptium Temurin 17 (build 17.0.5+8), OpenJDK 17, Eclipse Adoptium Temurin 11 (build 11.0.18+10) and OpenJDK 11 were used at the time of this writing.
• Certain test cases may require an individual rerun to pass. There may be false negatives due to seccomp not supporting s390x properly.
• If there is an stack overflow error, increase the -Xss arg value in the above command.
• Some X-Pack test cases will fail as X-Pack plugins are not supported on s390x, such as Machine Learning features.
• The test case failures in modules server:test, x-pack:plugin:core:test and x-pack:plugin:security:cli:test can be ignored. They fail on both Intel and s390x.
• The node.processors setting is now bounded by the number of available processors. Some X-Pack Test case may fail for this reason. To fix this, ensure the value of node.processors setting does not exceed the number of available processors.
• For more information regarding Elasticsearch testing, please refer to their testing documentation.
• User can also create distributions as deb, rpm and docker using below commands. In case of docker distribution, User needs to change Dockerfile to use built tar distribution instead of downloading it.

./gradlew :distribution:packages:s390x-oss-deb:assemble
./gradlew :distribution:packages:s390x-oss-rpm:assemble
./gradlew :distribution:docker:oss-docker-s390x-build-context:assemble

8) Install Elasticsearch

cd $SOURCE_ROOT/elasticsearch
sudo mkdir /usr/share/elasticsearch
sudo tar -xzf distribution/archives/oss-linux-s390x-tar/build/distributions/elasticsearch-oss-7.17.7-SNAPSHOT-linux-s390x.tar.gz -C /usr/share/elasticsearch --strip-components 1
sudo ln -sf /usr/share/elasticsearch/bin/* /usr/bin/

sudo /usr/sbin/groupadd elastic
sudo chown [username]:elastic -R /usr/share/elasticsearch/

• Update configurations to disable unsupported xpack.ml

sudo echo 'xpack.ml.enabled: false' >> /usr/share/elasticsearch/config/elasticsearch.yml

9) Verify Elasticsearch Server

> elasticsearch --version
Version: 7.17.7-SNAPSHOT, Build: oss/tar/78dcaaa8cee33438b91eca7f5c7f56a70fec9e80/2023-01-27T15:53:00.423326932Z, JVM: 17.0.5

10) Start Elasticsearch Server

elasticsearch &

Use curl -X GET http://localhost:9200/ when running Elasticsearch. The output should be similar to this:

{
  "name" : "5cfcf1e3d622",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "ga-j0JDORNy-p92Ji7De0w",
  "version" : {
    "number" : "7.17.7-SNAPSHOT",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "78dcaaa8cee33438b91eca7f5c7f56a70fec9e80",
    "build_date" : "2023-01-27T15:53:00.423326932Z",
    "build_snapshot" : true,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Note:

• If Access denied to Keystore error occurs on starting Elasticsearch, try updating/creating new password for the keystore using

elasticsearch-keystore create -p

Installing Elasticsearch Curator client (Optional)

Elasticsearch Curator can be used to curate, or manage, Elasticsearch indices and snapshots. It can be installed using below steps:

1) Install dependencies

• RHEL (7.8, 7.9, 8.4, 8.6, 8.7, 9.0, 9.1)

  sudo yum install -y python3-devel libyaml-devel
  sudo pip3 install elasticsearch==7.17.7

• SLES (12 SP5)

  sudo zypper install -y libyaml-devel
  sudo -H env PATH=$PATH pip3 install elasticsearch==7.17.7

  • Python 3.11.x -- Instructions for building Python 3.11.x can be found here

• SLES (15 SP4)

  sudo zypper install -y python3-devel python3-pip libyaml-devel glibc-locale-base
  sudo pip3 install elasticsearch==7.17.7
  export LC_CTYPE="en_US.UTF-8"

• Ubuntu (18.04, 20.04, 22.04)

  sudo apt-get update
  sudo apt-get install -y python3-pip libyaml-dev
  sudo pip3 install elasticsearch==7.17.7

2) Install Elasticsearch Curator

  sudo pip3 install elasticsearch==7.17.7
  sudo pip3 install elasticsearch-curator==5.8.4

Note:

• In case of an error sudo: pip3: command not found, Run above command as sudo env PATH=$PATH -H pip3 install elasticsearch-curator==5.8.4.

3) Verify Elasticsearch Curator installation

> curator --version
curator, version 5.8.4

Note:

• In case an error related to ASCII encoding is thrown by Click, ensure your locale is correctly set. For more details, refer this.

References:

• https://www.elastic.co/products/elasticsearch
• https://www.elastic.co/guide/en/elasticsearch/client/curator/5.8/index.html