Building Elasticsearch

Below versions of Elasticsearch are available in respective distributions at the time of creation of these build instructions.

The instructions provided below specify the steps to build Elasticsearch 8.6.0 on Linux on IBM Z for following distributions:

RHEL (7.8, 7.9, 8.6, 8.7, 9.0, 9.1)
SLES (12 SP5, 15 SP4)
Ubuntu (20.04, 22.04)

Prerequisites:

Docker packages are provided for RHEL, SLES and Ubuntu in their respective repositories. You can also use the static binaries provided here. More information about Docker CE can be found here
Ensure the current user belongs to group docker:

Use the below command to add group docker if it does not exist:
```
sudo groupadd docker
```
Use the below command to add current user to group docker if it has not been done:
```
sudo usermod -aG docker $USER && newgrp docker
```

General Notes

When following the steps below please use a standard permission user unless otherwise specified.
A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Build and Install Elasticsearch

1) Build using script

If you want to build Elasticsearch using manual steps, go to STEP 2.

Use the following commands to build Elasticsearch using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/8.6.0/build_elasticsearch.sh

# Build Elasticsearch
bash build_elasticsearch.sh  [Provide -t option for executing build with tests]

If the build completes successfully, go to STEP 9. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

2) Install build dependencies

export SOURCE_ROOT=/<source_root>/
export GRADLE_USER_HOME=$SOURCE_ROOT/.gradle

RHEL (7.8, 7.9, 8.6, 8.7, 9.0, 9.1)

sudo yum install -y curl git gzip tar wget patch make gcc gcc-c++

SLES 12 SP5

sudo zypper install -y curl git gzip tar wget patch make gcc gcc-c++ libnghttp2-devel

SLES 15 SP4

sudo zypper install -y curl git gzip tar wget patch make gcc gcc-c++

Ubuntu (20.04, 22.04)

sudo apt-get update
sudo apt-get install -y curl git gzip tar wget patch locales make gcc g++
sudo locale-gen en_US.UTF-8

3) Install Java

With Eclipse Adoptium Temurin Runtime (previously known as AdoptOpenJDK hotspot)
- Download and install Eclipse Adoptium Temurin Runtime (Java 17) from here.

With OpenJDK 17

RHEL (8.6, 8.7, 9.0, 9.1)

sudo yum install -y java-17-openjdk-devel

SLES 15 SP4

sudo zypper install -y java-17-openjdk java-17-openjdk-devel

Ubuntu (20.04, 22.04)

sudo apt-get install -y openjdk-17-jre openjdk-17-jdk ca-certificates-java

Note: At the time of creation of these build instructions, Elasticsearch was verified with Eclipse Adoptium Temurin Runtime (Java 17) version (build 17.0.5+8) and OpenJDK 17.

4) Set the environment variables

export LANG="en_US.UTF-8"
export JAVA_HOME=<Path to JDK>
export ES_JAVA_HOME=/<Path to JDK>/
export JAVA17_HOME=/<Path to JDK>/
export PATH=$ES_JAVA_HOME/bin:$PATH

Note: Ensure system locale is set up correctly for Elasticsearch to build without encoding errors.

5) Build JANSI v2.4.0

cd $SOURCE_ROOT
git clone https://github.com/fusesource/jansi.git
cd jansi
git checkout jansi-2.4.0
make clean-native native OS_NAME=Linux OS_ARCH=s390x

mkdir -p $SOURCE_ROOT/jansi-jar
cd $SOURCE_ROOT/jansi-jar
wget https://repo1.maven.org/maven2/org/fusesource/jansi/jansi/2.4.0/jansi-2.4.0.jar
jar xvf jansi-2.4.0.jar
cd org/fusesource/jansi/internal/native/Linux
mkdir s390x
cp $SOURCE_ROOT/jansi/target/native-Linux-s390x/libjansi.so s390x/
cd $SOURCE_ROOT/jansi-jar
jar cvf jansi-2.4.0.jar .

mkdir -p $SOURCE_ROOT/.gradle/caches/modules-2/files-2.1/org.fusesource.jansi/jansi/2.4.0/321c614f85f1dea6bb08c1817c60d53b7f3552fd/
cp jansi-2.4.0.jar $SOURCE_ROOT/.gradle/caches/modules-2/files-2.1/org.fusesource.jansi/jansi/2.4.0/321c614f85f1dea6bb08c1817c60d53b7f3552fd/
export sha256=$(sha256sum jansi-2.4.0.jar | awk '{print $1}')

6) Download Elasticsearch and apply patches

cd $SOURCE_ROOT
git clone https://github.com/elastic/elasticsearch
cd elasticsearch
git checkout v8.6.0

Apply gradle patches to create s390x distribution

export PATCH_URL="https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Elasticsearch/8.6.0/patch/elasticsearch.patch"
curl -o elasticsearch.patch $PATCH_URL
git apply elasticsearch.patch
sed -i '3284,3284 s|6cd91991323dd7b2fb28ca93d7ac12af5a86a2f53279e2b35827b30313fd0b9f|'"${sha256}"'|g' $SOURCE_ROOT/elasticsearch/gradle/verification-metadata.xml

There is a known issue related to deserialization w.r.t Big Endian system, refer this for more details. Below patch will allow the Elasticsearch build to complete without hitting endianness error.

diff --git a/server/src/main/java/org/elasticsearch/common/util/BigIntArray.java b/server/src/main/java/org/elasticsearch/common/util/BigIntArray.java
index e3cf7389f7e..554ce4061b6 100644
--- a/server/src/main/java/org/elasticsearch/common/util/BigIntArray.java
+++ b/server/src/main/java/org/elasticsearch/common/util/BigIntArray.java
@@ -25,11 +25,6 @@ import static org.elasticsearch.common.util.PageCacheRecycler.INT_PAGE_SIZE;
  * configurable length.
  */
 final class BigIntArray extends AbstractBigArray implements IntArray {
-    static {
-        if (ByteOrder.nativeOrder() != ByteOrder.LITTLE_ENDIAN) {
-            throw new Error("The deserialization assumes this class is written with little-endian ints.");
-        }
-    }

     private static final BigIntArray ESTIMATOR = new BigIntArray(0, BigArrays.NON_RECYCLING_INSTANCE, false);

diff --git a/server/src/main/java/org/elasticsearch/common/util/BigDoubleArray.java b/server/src/main/java/org/elasticsearch/common/util/BigDoubleArray.java
index ecfbfc5b9c6..3af7a533420 100644
--- a/server/src/main/java/org/elasticsearch/common/util/BigDoubleArray.java
+++ b/server/src/main/java/org/elasticsearch/common/util/BigDoubleArray.java
@@ -26,11 +26,6 @@ import static org.elasticsearch.common.util.PageCacheRecycler.DOUBLE_PAGE_SIZE;
  */
 final class BigDoubleArray extends AbstractBigArray implements DoubleArray {

-    static {
-        if (ByteOrder.nativeOrder() != ByteOrder.LITTLE_ENDIAN) {
-            throw new Error("The deserialization assumes this class is written with little-endian numbers.");
-        }
-    }

     private static final BigDoubleArray ESTIMATOR = new BigDoubleArray(0, BigArrays.NON_RECYCLING_INSTANCE, false);

7) Build

cd $SOURCE_ROOT/elasticsearch
CPU_NUM="$(grep -c ^processor /proc/cpuinfo)"
./gradlew :distribution:archives:linux-s390x-tar:assemble --max-workers="$CPU_NUM"  --parallel

8) Testing (Optional)

cd $SOURCE_ROOT/elasticsearch
export JAVA_TOOL_OPTIONS="-Dfile.encoding=UTF8"
export JAVA_HOME=/<Path to JDK>/
export RUNTIME_JAVA_HOME=/<Path to JDK>/
./gradlew --continue test -Dtests.haltonfailure=false -Dtests.jvm.argline="-Xss2m"

Notes:

You can set RUNTIME_JAVA_HOME optionally to the location of the JDK(another version of the JDK that act as the runtime) that you'd like to use during testing. Eclipse Adoptium Temurin 17 (build 17.0.5+8) and OpenJDK 17 was used at the time of this writing.
If there is an stack overflow error, increase the -Xss arg value in the above command.
Some X-Pack test cases will fail as X-Pack plugins are not supported on s390x, such as Machine Learning features.
The test case failures in modules server:test can be ignored. They fail on both intel and s390x.
The node.processors setting is now bounded by the number of available processors. Some X-Pack Test case may fail for this reason. To fix this, ensure the value of node.processors setting does not exceed the number of available processors.
For more information regarding Elasticsearch testing, please refer to their testing documentation.
User can also create distributions as deb, rpm and docker using below commands.

./gradlew :distribution:packages:s390x-deb:assemble
./gradlew :distribution:packages:s390x-rpm:assemble
./gradlew :distribution:docker:docker-s390x-export:assemble

9) Install Elasticsearch

cd $SOURCE_ROOT/elasticsearch
sudo mkdir /usr/share/elasticsearch
sudo tar -xzf distribution/archives/linux-s390x-tar/build/distributions/elasticsearch-8.6.0-SNAPSHOT-linux-s390x.tar.gz -C /usr/share/elasticsearch --strip-components 1
sudo ln -sf /usr/share/elasticsearch/bin/* /usr/bin/

sudo /usr/sbin/groupadd elastic
sudo chown [username]:elastic -R /usr/share/elasticsearch/

Update configurations to disable unsupported xpack.ml

sudo echo 'xpack.ml.enabled: false' >> /usr/share/elasticsearch/config/elasticsearch.yml

10) Verify Elasticsearch Server

> elasticsearch --version
Version: 8.6.0-SNAPSHOT, Build: tar/f67ef2df40237445caa70e2fef79471cc608d70d/2023-02-03T05:55:54.988376171Z, JVM: 17.0.5

11) Start Elasticsearch Server

elasticsearch &

Use curl --cacert /usr/share/elasticsearch/config/certs/http_ca.crt -u elastic https://localhost:9200 when running Elasticsearch. The output should be similar to this:

{
  "name" : "b6866e364539",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "q_1gqYXeSW6XaPVEfpwjwA",
  "version" : {
    "number" : "8.6.0-SNAPSHOT",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "f67ef2df40237445caa70e2fef79471cc608d70d",
    "build_date" : "2023-01-20T05:01:03.217555362Z",
    "build_snapshot" : true,
    "lucene_version" : "9.4.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

References:

https://www.elastic.co/products/elasticsearch

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Building Elasticsearch

Building Elasticsearch

Prerequisites:

Build and Install Elasticsearch

1) Build using script

2) Install build dependencies

3) Install Java

4) Set the environment variables

5) Build JANSI v2.4.0

6) Download Elasticsearch and apply patches

7) Build

8) Testing (Optional)

9) Install Elasticsearch

10) Verify Elasticsearch Server

11) Start Elasticsearch Server

References:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!