SetupSpark.md

Setup Spark

• Version here: 2.4.0

Current version of Spark natively contain pre-built Scala binary.

Current installation process is for Spark Standalone Mode

Download spark

fab download-spark

Install Spark

fab install-spark -v

Test

$ fab ssh-connect 0 -h # connect to master

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0
      /_/
scala> sc.version
res0: String = 2.4.0
scala> :quit

Install PySpark

pyspark

Configure for Pi Cluster

spark-env.sh

Memory...

TBD

slaves

like Hadoop workers

Submit Application

Spark - Submitting Applications

PySpark with Jupyter Notebook

Get Started with PySpark and Jupyter Notebook in 3 Minutes

fab install-jupyter

More detail see SetupJupyter.md

Job History

• Spark - Monitoring and Instrumentation
• Mastering Apache Spark - Spark History Server
• IBM - Enabling the Spark History service

Links

• Official Website
• Spark Github

Book

• Mastering Apache Spark