Building Elasticsearch

Building Elasticsearch

Note: This package uses Log4j. Please see details here, for the updates on security vulnerabilities.

Below versions of Elasticsearch are available in respective distributions at the time of creation of these build instructions.

The instructions provided below specify the steps to build Elasticsearch 7.16.3 on Linux on IBM Z for following distributions:

• RHEL (7.8, 7.9, 8.2, 8.4, 8.5)
• SLES (12 SP5, 15 SP3)
• Ubuntu (18.04, 20.04)

General Notes

• When following the steps below please use a standard permission user unless otherwise specified.
• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Build and Install Elasticsearch

1) Build using script

If you want to build Elasticsearch using manual steps, go to STEP 2.

Use the following commands to build Elasticsearch using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/7.16.3/build_elasticsearch.sh

# Build Elasticsearch
bash build_elasticsearch.sh  [Provide -t option for executing build with tests]

If the build completes successfully, go to STEP 8. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

2) Install build dependencies

export SOURCE_ROOT=/<source_root>/

• RHEL (7.8, 7.9, 8.2, 8.4, 8.5)

  sudo yum install -y curl git gzip tar wget patch

  • Download and install Eclipse Adoptium Temurin Runtime (Java 16) from here.

• SLES (12 SP5, 15 SP3)

  sudo zypper install -y curl git gzip tar wget patch

  • Download and install Eclipse Adoptium Temurin Runtime (Java 16) from here.

• Ubuntu (18.04, 20.04)

  sudo apt-get update
  sudo apt-get install -y curl git gzip tar wget patch locales
  sudo locale-gen en_US.UTF-8

  • Download and install Eclipse Adoptium Temurin Runtime (Java 16) from here.

Note: At the time of creation of these build instructions, Elasticsearch was verified with Eclipse Adoptium Temurin Runtime (Java 16) version (build 16.0.2+7).

3) Set the environment variables

export LANG="en_US.UTF-8"
export JAVA_HOME=<Path to JDK>
export ES_JAVA_HOME=/<Path to JDK>/
export JAVA16_HOME=/<Path to JDK>/
export PATH=$ES_JAVA_HOME/bin:$PATH

Note: Ensure system locale is set up correctly for Elasticsearch to build without encoding errors.

4) Download Elasticsearch and apply patches

cd $SOURCE_ROOT
git clone https://github.com/elastic/elasticsearch
cd elasticsearch
git checkout v7.16.3

• Apply gradle patches to create s390x distribution

 export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/7.16.3/patch/elasticsearch.patch"
 curl -o elasticsearch.patch $PATCH_URL
 git apply elasticsearch.patch

5) Build

cd $SOURCE_ROOT/elasticsearch
CPU_NUM="$(grep -c ^processor /proc/cpuinfo)"
./gradlew :distribution:archives:oss-linux-s390x-tar:assemble --max-workers="$CPU_NUM"  --parallel

6) Testing (Optional)

cd $SOURCE_ROOT/elasticsearch
 export JAVA_HOME=/<Path to JDK>/
 export RUNTIME_JAVA_HOME=/<Path to JDK>/
./gradlew --continue test -Dtests.haltonfailure=false -Dtests.jvm.argline="-Xss2m"

Notes:

• You can set RUNTIME_JAVA_HOME optionally to the location of the JDK(another version of the JDK that act as the runtime) that you'd like to use during testing. AdoptOpenJDK 16 with Hotspot was used at the time of this writing.
• Certain test cases may require an individual rerun to pass. There may be false negatives due to seccomp not supporting s390x properly.
• If there is an stack overflow error, increase the -Xss arg value in the above command.
• Some X-Pack test cases will fail as X-Pack plugins are not supported on s390x, such as Machine Learning features.
• The test case failures in modules server:test and :libs:elasticsearch-ssl-config:test can be ignored. They fail on both intel and s390x.
• The node.processors setting is now bounded by the number of available processors. Some X-Pack Test case may fail for this reason. To fix this, ensure the value of node.processors setting does not exceed the number of available processors.
• Illegal reflective access warnings in the logs are known issues. See https://github.com/elastic/elasticsearch/issues/40334.
• For more information regarding Elasticsearch testing, please refer to their testing documentation.
• User can also create distributions as deb, rpm and docker using below commands. In case of docker distribution, User needs to change Dockerfile to use built tar distribution instead of downloading it.

./gradlew :distribution:packages:s390x-oss-deb:assemble
./gradlew :distribution:packages:s390x-oss-rpm:assemble
./gradlew :distribution:docker:oss-docker-s390x-build-context:assemble

7) Install Elasticsearch

cd $SOURCE_ROOT/elasticsearch
sudo mkdir /usr/share/elasticsearch
sudo tar -xzf distribution/archives/oss-linux-s390x-tar/build/distributions/elasticsearch-oss-7.16.3-SNAPSHOT-linux-s390x.tar.gz -C /usr/share/elasticsearch --strip-components 1
sudo ln -sf /usr/share/elasticsearch/bin/* /usr/bin/

sudo /usr/sbin/groupadd elastic
sudo chown [username]:elastic -R /usr/share/elasticsearch/

• Update configurations to disable unsupported xpack.ml

sudo echo 'xpack.ml.enabled: false' >> /usr/share/elasticsearch/config/elasticsearch.yml

8) Verify Elasticsearch Server

> elasticsearch --version
Version: 7.16.3-SNAPSHOT, Build: oss/tar/4e6e4eab2297e949ec994e688dad46290d018022/2022-01-20T07:28:37.896634170Z, JVM: 16.0.2

9) Start Elasticsearch Server

elasticsearch &

Use curl -X GET http://localhost:9200/ when running Elasticsearch. The output should be similar to this:

{
  "name" : "",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "tUvQfnpgSO2wXCu8eTVsmA",
  "version" : {
    "number" : "7.16.3-SNAPSHOT",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "4e6e4eab2297e949ec994e688dad46290d018022",
    "build_date" : "2022-01-20T07:28:37.896634170Z",
    "build_snapshot" : true,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Note:

• Elasticsearch needs to be run by a non-root user which has standard permission rights, hence we have changed the owner in the above step.
• If Access denied to Keystore error occurs on starting Elasticsearch, try updating/created new password for the keystore using

elasticsearch-keystore create -p

Installing Elasticsearch Curator client (Optional)

Elasticsearch Curator can be used to curate, or manage, Elasticsearch indices and snapshots. It can be installed using below steps:

1) Install dependencies

• RHEL (7.8, 7.9, 8.2, 8.4, 8.5)

  sudo yum install -y python3-devel libyaml-devel
  sudo pip3 install elasticsearch==7.16.3

• SLES (12 SP5)

  sudo zypper install -y libyaml-devel
  sudo -H env PATH=$PATH pip3 install elasticsearch==7.16.3

  • Python 3.10.x -- Instructions for building Python 3.10.x can be found here

• SLES (15 SP3)

  sudo zypper install -y python3-devel python3-pip libyaml-devel glibc-locale-base
  sudo pip3 install elasticsearch==7.16.3
  export LC_CTYPE="en_US.UTF-8"

• Ubuntu (18.04, 20.04)

  sudo apt-get update
  sudo apt-get install -y python3-pip libyaml-dev
  sudo pip3 install elasticsearch==7.16.3

2) Install Elasticsearch Curator

  sudo pip3 install elasticsearch==7.13.4
  sudo pip3 install elasticsearch-curator==5.8.4

Note:

• In case of an error sudo: pip3: command not found, Run above command as sudo env PATH=$PATH -H pip3 install elasticsearch-curator==5.8.4.

3) Verify Elasticsearch Curator installation

> curator --version
curator, version 5.8.4

Note:

• In case an error related to ASCII encoding is thrown by Click, ensure your locale is correctly set. For more details, refer this.

References:

• https://www.elastic.co/products/elasticsearch
• https://www.elastic.co/guide/en/elasticsearch/client/curator/5.x/index.html