Building Apache Storm

Apache Storm

The instructions provided below specify the steps to build Apache Storm version 1.1.1 on Linux on IBM Z for the following distributions:

• RHEL (7.3, 7.4)
• SLES (12 SP2, 12 SP3)
• Ubuntu (16.04, 18.04)

General Notes:

• When following the steps below please use a standard permission user unless otherwise specified.
• A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Step 1: Building and installing Apache Storm

1.1) Install Dependencies

• RHEL (7.3, 7.4)

  sudo yum install git tar java-1.8.0-openjdk-devel.s390x  

• SLES (12 SP2, 12 SP3)

  sudo zypper install git java-1_8_0-openjdk tar  

• Ubuntu (16.04, 18.04)

  sudo apt-get install git tar ant wget openjdk-8-jdk maven curl gpgv nodejs gnupg2 python

1.2) Install Maven 3.2.5 (For RHEL and SLES only)

wget http://www.eu.apache.org/dist/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
tar -xzf apache-maven-3.2.5-bin.tar.gz
export PATH=$PWD/apache-maven-3.2.5/bin:$PATH

Note: Maven v3.2.5 or below is required. There is a bug in later versions of maven as linked to from https://issues.apache.org/jira/browse/MSHADE-206 that cause shaded dependencies to not be packaged correctly.

1.3) Install NodeJS SDK (For RHEL and SLES only)

Install nodejs sdk from https://developer.ibm.com/node/sdk/ for Linux on System z 64-bit

Add the nodejs executable to the path:

export PATH=$PATH:<Nodejs-Install-Directory>/bin  

1.4) Install rvm and nvm

curl -L https://get.rvm.io | bash -s stable --autolibs=enabled && source $HOME/.profile  
command curl -sSL https://rvm.io/mpapis.asc | gpg2 --import -  
wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.26.1/install.sh | bash && source $HOME/.bashrc

1.5) Set environment variables

Setup up Maven and JAVA:

export JAVA_HOME=<path to java>          
export PATH=$JAVA_HOME/bin:$PATH
export _JAVA_OPTIONS=-Xmx4096m
export JVM_ARGS="-Xms4096m -Xmx4096m"
export MAVEN_OPTS="-Xms4096m -Xmx8192m -Xss4096k"

1.6) Download the source code

cd /<source_root>/
git clone https://github.com/apache/storm.git  
cd storm  
git checkout tags/v1.1.1

1.7) Build Apache Storm

mvn clean install

Note: If you wish to skip the unit tests you can do this by adding -DskipTests to the command line.

Step 2: Testing(Optional)

2.1) Run unit test cases

To run Clojure and Java unit tests but no integration tests execute the command

cd /<source_root>/storm
mvn test

Note:

• Integration tests require that you activate the profile integration-test and that you specify the maven-failsafe-plugin in the module pom file.
• JmsSpoutTest fails intermittently.

Step 3: Create a Storm distribution (packaging)

cd /<source_root>/storm/storm-dist/binary
mvn package  

Note:

• After running mvn package you may be asked to enter your GPG/PGP credentials (once for each binary file, in fact). This happens because the packaging step will create *.asc digital signatures for all the binaries, and in the workflow above your GPG private key will be used to create those signatures.
• You can disbale GPG signing by adding -Dgpg.skip=true
• The binaries can be found at /<source_root>/storm/storm-dist/binary/target/.

References:

• http://storm.apache.org/releases/current/index.html
• https://github.com/apache/storm/blob/v1.1.1/DEVELOPER.md