Building Elasticsearch

Building Elasticsearch

Below versions of Elasticsearch are available in respective distributions at the time of creation of these build instructions.

The instructions provided below specify the steps to build Elasticsearch 7.9.1 on Linux on IBM Z for following distributions:

• RHEL (7.6, 7.7, 7.8, 8.1, 8.2)
• SLES 12 SP5
• Ubuntu 18.04

General Notes

• When following the steps below please use a standard permission user unless otherwise specified.
• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Build and Install Elasticsearch

1) Build using script

If you want to build Elasticsearch using manual steps, go to STEP 2.

Use the following commands to build Elasticsearch using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/7.9.1/build_elasticsearch.sh

# Build Elasticsearch
bash build_elasticsearch.sh  [Provide -t option for executing build with tests, -j AdoptJDK11_openj9 to run with AdoptOpenJDK 11 (With OpenJ9)]

If the build completes successfully, go to STEP 8. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

2) Install Build Dependencies

export SOURCE_ROOT=/<source_root>/

• RHEL (7.6, 7.7, 7.8, 8.1, 8.2)

  sudo yum install -y curl git gzip tar wget patch

  • With Open JDK:

    sudo yum install -y java-11-openjdk java-11-openjdk-devel
    

  • With AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9 , OpenJDK11 with Hotspot)
    • Download and install AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9 , OpenJDK11 with Hotspot) from here.

• SLES 12 SP5

  sudo zypper install -y curl git gzip tar wget patch

  • With Open JDK:

    sudo zypper install -y java-11-openjdk java-11-openjdk-devel
    

  • With AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9 , OpenJDK11 with Hotspot)
    • Download and install AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9 , OpenJDK11 with Hotspot) from here.

• Ubuntu 18.04

  sudo apt-get update
  sudo apt-get install -y curl git gzip tar wget patch locales
  sudo locale-gen en_US.UTF-8

  • With Open JDK

    sudo apt-get install -y openjdk-11-jdk
    

  • With AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9 , OpenJDK11 with Hotspot)

    • Download and install AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9 , OpenJDK11 with Hotspot) from here.

Note: At the time of creation of these build instructions, Elasticsearch was verified with AdoptOpenJDK 11 version (build 11.0.8+10).

3) Set the environment variables

unset JAVA_TOOL_OPTIONS
export LANG="en_US.UTF-8"
export JAVA_TOOL_OPTIONS="-Dfile.encoding=UTF8"
export JAVA_HOME=/<Path to JDK>/
export JAVA11_HOME=/<Path to JDK>/
export PATH=$JAVA_HOME/bin:$PATH

Note: Ensure system locale is set up correctly for Elasticsearch to build without encoding errors.

4) Download Elasticsearch and apply patches

cd $SOURCE_ROOT
git clone https://github.com/elastic/elasticsearch
cd elasticsearch
git checkout v7.9.1

• Apply gradle patches to create s390x distribution

 export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/7.9.1/patch"
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/archives/linux-s390x-tar
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/archives/oss-linux-s390x-tar
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/packages/s390x-deb
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/packages/s390x-oss-deb
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/packages/s390x-oss-rpm
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/packages/s390x-rpm
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/docker/docker-s390x-export
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/docker/oss-docker-s390x-export
 wget $PATCH_URL/docker_build_context_build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/docker/docker-s390x-build-context
 mv $SOURCE_ROOT/elasticsearch/distribution/docker/docker-s390x-build-context/docker_build_context_build.gradle $SOURCE_ROOT/elasticsearch/distribution/docker/docker-s390x-build-context/build.gradle
 wget $PATCH_URL/oss_docker_build_context_build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/docker/oss-docker-s390x-build-context
 mv $SOURCE_ROOT/elasticsearch/distribution/docker/oss-docker-s390x-build-context/oss_docker_build_context_build.gradle $SOURCE_ROOT/elasticsearch/distribution/docker/oss-docker-s390x-build-context/build.gradle
 wget -O - $PATCH_URL/diff.patch | git apply

5) Build

cd $SOURCE_ROOT/elasticsearch
./gradlew :distribution:archives:oss-linux-s390x-tar:assemble --parallel

6) Testing (Optional)

cd $SOURCE_ROOT/elasticsearch
./gradlew --continue test -Dtests.haltonfailure=false -Dtests.jvm.argline="-Xss2m"

Notes:

• You can set RUNTIME_JAVA_HOME optionally to the location of the JDK(another version of the JDK that act as the runtime) that you'd like to use during testing. AdoptOpenJDK 11 with Hotspot was used at the time of this writing.
• If using java 11, ensure that you have JAVA11_HOME set to the location of the version 11 JDK. Likewise if using java 8, set JAVA8_HOME, etc...
• Certain test cases may require an individual rerun to pass. There may be false negatives due to seccomp not supporting s390x properly.
• If there is an stack overflow error, increase the -Xss arg value in the above command.
• Some X-Pack test cases will fail as X-Pack plugins are not supported on s390x, such as Machine Learning features.
• _The node.processors setting is now bounded by the number of available processors. Some X-Pack Test case may fail for this reason. To fix this, ensure the value of node.processors setting does not exceed the number of available processors.
• Illegal reflective access warnings in the logs are known issues. See https://github.com/elastic/elasticsearch/issues/40334.
• For more information regarding Elasticsearch testing, please refer to their testing documentation.
• User can also create distributions as deb, rpm and docker using below commands. In case of docker distribution, User needs to change Dockerfile to use built tar distribution instead of downloading it.
• The test case failures in the moduleqa:evil-tests:test are related to environmental issues and can be ignored. Please refer to this issue for more info. Setting the $ES_TMPDIR can help passing the test.
• _The test case failures in the modulebuild-tools:compileTestFixturesJava are related to JDK 11 and can be ignored. It fails on both intel and s390x.

./gradlew :distribution:packages:s390x-oss-deb:assemble
./gradlew :distribution:packages:s390x-oss-rpm:assemble
./gradlew :distribution:docker:oss-docker-s390x-build-context:assemble

7) Install Elasticsearch

cd $SOURCE_ROOT/elasticsearch
sudo mkdir /usr/share/elasticsearch
sudo tar -xzf distribution/archives/oss-linux-s390x-tar/build/distributions/elasticsearch-oss-7.8.1-SNAPSHOT-linux-s390x.tar.gz -C /usr/share/elasticsearch --strip-components 1
sudo ln -sf /usr/share/elasticsearch/bin/* /usr/bin/

sudo groupadd [username]
sudo chown [username]:[username] -R /usr/share/elasticsearch/

8) Verify Elasticsearch Server

> elasticsearch --version
Version: 7.9.1-SNAPSHOT, Build: oss/tar/083627f112ba94dffc1232e8b42b73492789ef91/2020-10-05T08:52:59.776397Z, JVM: 11.0.8

9) Start Elasticsearch Server

elasticsearch

Use curl -X GET http://localhost:9200/ when running Elasticsearch. The output should be similar to this:

{
  "name" : "localhost",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "7.9.1-SNAPSHOT",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "083627f112ba94dffc1232e8b42b73492789ef91",
    "build_date" : "2020-10-05T10:51:41.758966995Z",
    "build_snapshot" : true,
    "lucene_version" : "8.6.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Note:

• Elasticsearch needs to be run by a non-root user which has standard permission rights, hence we have changed the owner in the above step.
• _If Access denied to Keystore error occurs on starting Elasticsearch, try updating/created new password for the keystore using

elasticsearch-keystore create -p

Installing Elasticsearch Curator client (Optional)

Elasticsearch Curator can be used to curate, or manage, Elasticsearch indices and snapshots. It can be installed using below steps:

1) Install dependencies

• RHEL (7.6, 7.7, 7.8, 8.1, 8.2)

  sudo yum install -y python3-devel

• SLES 12 SP5

  sudo zypper install -y python3 python3-pip

• Ubuntu 18.04

  sudo apt-get update
  sudo apt-get install -y python3-pip

2) Install Elasticsearch Curator

  sudo -H pip3 install elasticsearch-curator

Note:

• In case of an error sudo: pip3: command not found, Run above command as sudo env PATH=$PATH -H pip3 install elasticsearch-curator.

3) Verify Elasticsearch Curator installation

  curator --version

Note:

• In case an error related to ASCII encoding is thrown by Click, ensure your locale is correctly set. For more details, refer this.

References:

https://www.elastic.co/products/elasticsearch
https://www.elastic.co/guide/en/elasticsearch/client/curator/5.x/index.html