Build and Deploy

Local or native jar build

Requirements

To Build :

• Apache Maven (Tested on v3.3.x)
• JDK (Tested on Oracle JDK 1.8)
• Working internet connection to retrieve maven dependencies

The following dependencies will be downloaded from Maven central. Feel free to look inside the pom.xml for the current versions being used.

• Apache Spark
• Apache Nutch
• Apache Kafka Client
• Apache Solr Client
• Scala

Note that the libraries like Solr-client, spark, kafka etc should match with your own deployment version. For instance, if you have Spark Cluster deployment of v1.6 with Scala 2.11, make sure to set them the same versions for the client libraries in pom.xml.

To deploy and run :

In Local Mode

• Apache Solr Server or Solr Cloud (Tested on v6.0.1): The necessary configuration files are placed under conf/solr directory

Build Instructions

• $ mvn clean install should give a sparkler-app/target/sparkler-app-xx.jar
• Use bin/sparkler.sh script for convinience

---

Docker build

cd to the root directory of the project and issue the following commands:

mvn clean
docker build -f sparkler-deployment/docker/Dockerfile  -t sparkler-local

To start for one time use:

docker run -it sparkler-local

To start and reuse the container

Start in daemon mode: docker run -it -d sparkler-local

The above command should print container_id

Attach a shell to the container docker exec -it <container_id> /bin/bash

Inside the docker

• /data/solr/bin/solr - start / stop solr using this tool
• /data/sparkler/bin/sparkler.sh - cli interface to sparkler