Building Elasticsearch

Below versions of Elasticsearch are available in respective distributions at the time of creation of these build instructions.

The instructions provided below specify the steps to build Elasticsearch 7.10.0 on Linux on IBM Z for following distributions:

RHEL (7.8, 7.9, 8.1, 8.2, 8.3)
SLES (12 SP5, 15 SP1, 15 SP2)
Ubuntu (18.04, 20.04)

General Notes

When following the steps below please use a standard permission user unless otherwise specified.
A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Build and Install Elasticsearch

1) Build using script

If you want to build Elasticsearch using manual steps, go to STEP 2.

Use the following commands to build Elasticsearch using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/7.10.0/build_elasticsearch.sh

# Build Elasticsearch
bash build_elasticsearch.sh  [Provide -t option for executing build with tests, -j AdoptJDK11_openj9 to run with AdoptOpenJDK 11 (With OpenJ9)]

If the build completes successfully, go to STEP 8. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

2) Install Build Dependencies

export SOURCE_ROOT=/<source_root>/

RHEL (7.8, 7.9, 8.1, 8.2, 8.3)
```
sudo yum install -y curl git gzip tar wget patch
```
- With Open JDK 11 :
```
sudo yum install -y java-11-openjdk java-11-openjdk-devel
```
- With AdoptOpenJDK (OpenJDK15 with Eclipse OpenJ9 , OpenJDK15 with Hotspot)
  - Download and install AdoptOpenJDK (OpenJDK11 with Eclipse OpenJ9 , OpenJDK11 with Hotspot) from here.
SLES (12 SP5, 15 SP1, 15 SP2)
```
sudo zypper install -y curl git gzip tar wget patch
```
- With Open JDK 11:
```
sudo zypper install -y java-11-openjdk java-11-openjdk-devel
```
- With AdoptOpenJDK (OpenJDK15 with Eclipse OpenJ9 , OpenJDK15 with Hotspot)
  - Download and install AdoptOpenJDK (OpenJDK15 with Eclipse OpenJ9 , OpenJDK15 with Hotspot) from here.
Ubuntu (18.04, 20.04)
```
sudo apt-get update
sudo apt-get install -y curl git gzip tar wget patch locales
sudo locale-gen en_US.UTF-8
```
- With Open JDK 11:
```
sudo apt-get install -y openjdk-11-jdk
```
- With AdoptOpenJDK (OpenJDK15 with Eclipse OpenJ9 , OpenJDK15 with Hotspot)
  - Download and install AdoptOpenJDK (OpenJDK15 with Eclipse OpenJ9 , OpenJDK15 with Hotspot) from here.

Note: At the time of creation of these build instructions, Elasticsearch was verified with AdoptOpenJDK 11 version (build 15.0.1+9).

3) Set the environment variables

unset JAVA_TOOL_OPTIONS
export LANG="en_US.UTF-8"
export JAVA_TOOL_OPTIONS="-Dfile.encoding=UTF8"
export JAVA_HOME=/<Path to JDK>/
export JAVA15_HOME=/<Path to JDK>/   #Incase of AdoptOpenJDK
export JAVA11_HOME=/<Path to JDK>/   #Incase of OpenJDK
export PATH=$JAVA_HOME/bin:$PATH

Note: Ensure system locale is set up correctly for Elasticsearch to build without encoding errors.

4) Download Elasticsearch and apply patches

cd $SOURCE_ROOT
git clone https://github.com/elastic/elasticsearch
cd elasticsearch
git checkout v7.10.0

Apply gradle patches to create s390x distribution

 export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/7.10.0/patch"
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/archives/linux-s390x-tar
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/archives/oss-linux-s390x-tar
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/packages/s390x-deb
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/packages/s390x-oss-deb
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/packages/s390x-oss-rpm
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/packages/s390x-rpm
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/docker/docker-s390x-export
 wget $PATCH_URL/build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/docker/oss-docker-s390x-export
 wget $PATCH_URL/docker_build_context_build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/docker/docker-s390x-build-context
 mv $SOURCE_ROOT/elasticsearch/distribution/docker/docker-s390x-build-context/docker_build_context_build.gradle $SOURCE_ROOT/elasticsearch/distribution/docker/docker-s390x-build-context/build.gradle
 wget $PATCH_URL/oss_docker_build_context_build.gradle -P $SOURCE_ROOT/elasticsearch/distribution/docker/oss-docker-s390x-build-context
 mv $SOURCE_ROOT/elasticsearch/distribution/docker/oss-docker-s390x-build-context/oss_docker_build_context_build.gradle $SOURCE_ROOT/elasticsearch/distribution/docker/oss-docker-s390x-build-context/build.gradle
 wget -O - $PATCH_URL/diff.patch | git apply

5) Build

cd $SOURCE_ROOT/elasticsearch
./gradlew :distribution:archives:oss-linux-s390x-tar:assemble --parallel

6) Testing (Optional)

cd $SOURCE_ROOT/elasticsearch
./gradlew --continue test -Dtests.haltonfailure=false -Dtests.jvm.argline="-Xss2m"

Notes:

You can set RUNTIME_JAVA_HOME optionally to the location of the JDK(another version of the JDK that act as the runtime) that you'd like to use during testing. AdoptOpenJDK 11 with Hotspot was used at the time of this writing.
If using java 15, ensure that you have JAVA15_HOME set to the location of the version 15 JDK. Likewise if using java 11, set JAVA11_HOME, etc...
Certain test cases may require an individual rerun to pass. There may be false negatives due to seccomp not supporting s390x properly.
If there is an stack overflow error, increase the -Xss arg value in the above command.
Some X-Pack test cases will fail as X-Pack plugins are not supported on s390x, such as Machine Learning features.
_The node.processors setting is now bounded by the number of available processors. Some X-Pack Test case may fail for this reason. To fix this, ensure the value of node.processors setting does not exceed the number of available processors.
Illegal reflective access warnings in the logs are known issues. See https://github.com/elastic/elasticsearch/issues/40334.
For more information regarding Elasticsearch testing, please refer to their testing documentation.
User can also create distributions as deb, rpm and docker using below commands. In case of docker distribution, User needs to change Dockerfile to use built tar distribution instead of downloading it.
The test case failures in the moduleqa:evil-tests:test are related to environmental issues and can be ignored. Please refer to this issue for more info. Setting the $ES_TMPDIR can help passing the test.
_The test case failures in the modulebuild-tools:compileTestFixturesJava are related to JDK 11 and can be ignored. It fails on both intel and s390x.

./gradlew :distribution:packages:s390x-oss-deb:assemble
./gradlew :distribution:packages:s390x-oss-rpm:assemble
./gradlew :distribution:docker:oss-docker-s390x-build-context:assemble

7) Install Elasticsearch

cd $SOURCE_ROOT/elasticsearch
sudo mkdir /usr/share/elasticsearch
sudo tar -xzf distribution/archives/oss-linux-s390x-tar/build/distributions/elasticsearch-oss-7.10.0-SNAPSHOT-linux-s390x.tar.gz -C /usr/share/elasticsearch --strip-components 1
sudo ln -sf /usr/share/elasticsearch/bin/* /usr/bin/

sudo groupadd [username]
sudo chown [username]:[username] -R /usr/share/elasticsearch/

8) Verify Elasticsearch Server

> elasticsearch --version
Version: 7.10.0-SNAPSHOT, Build: oss/tar/51e9d6f22758d0374a0f3f5c6e8f3a7997850f96/2020-12-01T09:31:28.802545Z, JVM: 11.0.9.1

9) Start Elasticsearch Server

elasticsearch

Use curl -X GET http://localhost:9200/ when running Elasticsearch. The output should be similar to this:

{
  "name" : "61b23cb2ffa8",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "ld9S0bOQR0mL8qVsZ3dn5A",
  "version" : {
    "number" : "7.10.0-SNAPSHOT",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "51e9d6f22758d0374a0f3f5c6e8f3a7997850f96",
    "build_date" : "2020-12-01T09:31:28.802545Z",
    "build_snapshot" : true,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Note:

Elasticsearch needs to be run by a non-root user which has standard permission rights, hence we have changed the owner in the above step.
_If Access denied to Keystore error occurs on starting Elasticsearch, try updating/created new password for the keystore using

elasticsearch-keystore create -p

Installing Elasticsearch Curator client (Optional)

Elasticsearch Curator can be used to curate, or manage, Elasticsearch indices and snapshots. It can be installed using below steps:

1) Install dependencies

RHEL (7.8, 7.9, 8.1, 8.2, 8.3)

sudo yum install -y python3-devel
sudo mkdir /usr/local/lib/python3.6
sudo ln -s /usr/lib/python3.6/site-packages /usr/local/lib/python3.6/

SLES (12 SP5, 15 SP1, 15 SP2)

sudo zypper install -y python3 python3-pip

Ubuntu (18.04, 20.04)

sudo apt-get update
sudo apt-get install -y python3-pip

2) Install Elasticsearch Curator

  export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/7.10.0/patch"
  
  wget https://github.com/elastic/curator/archive/V5.8.3.tar.gz -O elasticsearch-curator.tar.gz
  tar -xzvf elasticsearch-curator.tar.gz
  rm -rf elasticsearch-curator.tar.gz
  cd curator-5.8.3/
  curl  -o requirements.patch $PATCH_URL/requirements.patch
  curl  -o setup.py.patch $PATCH_URL/setup.py.patch
  curl  -o setup.cfg.patch $PATCH_URL/setup.cfg.patch
  patch requirements.txt < requirements.patch
  patch setup.py < setup.py.patch
  patch setup.cfg < setup.cfg.patch
  sudo python3 setup.py install

Note:

In case of an error sudo: pip3: command not found, Run above command as sudo env PATH=$PATH -H pip3 install elasticsearch-curator.

3) Verify Elasticsearch Curator installation

  curator --version

Note:

In case an error related to ASCII encoding is thrown by Click, ensure your locale is correctly set. For more details, refer this.

References:

https://www.elastic.co/products/elasticsearch
https://www.elastic.co/guide/en/elasticsearch/client/curator/5.x/index.html

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Building Elasticsearch

Building Elasticsearch

Build and Install Elasticsearch

1) Build using script

2) Install Build Dependencies

3) Set the environment variables

4) Download Elasticsearch and apply patches

5) Build

6) Testing (Optional)

7) Install Elasticsearch

8) Verify Elasticsearch Server

9) Start Elasticsearch Server

Installing Elasticsearch Curator client (Optional)

1) Install dependencies

2) Install Elasticsearch Curator

3) Verify Elasticsearch Curator installation

References:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!