Sparkify Music Spark Data Lake Udacity Nanodegree Project

Introduction

A music streaming startup, Sparkify, has grown their user base and song database even more and wants to move their data warehouse to a data lake. Their data resides in S3, in a directory of JSON logs on user activity on the app, as well as a directory with JSON metadata on the songs in their app.

As their data engineer, you are tasked with building an ETL pipeline that extracts their data from S3, processes them using Spark, and loads the data back into S3 as a set of dimensional tables. This will allow their analytics team to continue finding insights in what songs their users are listening to.

Datasets

Song Dataset

The first dataset is a subset of real data from the Million Song Dataset. Each file is in JSON format and contains metadata about a song and the artist of that song. The files are partitioned by the first three letters of each song's track ID.

For example, here are filepaths to two files in this dataset:

song_data/A/B/C/TRABCEI128F424C983.json song_data/A/A/B/TRAABJL12903CDCF1A.json

Log Dataset

The second dataset consists of log files in JSON format generated by this event simulator based on the songs in the dataset above. These simulate app activity logs from an imaginary music streaming app based on configuration settings.

The log files in the dataset you'll be working with are partitioned by year and month.

For example, here are filepaths to two files in this dataset:

log_data/2018/11/2018-11-12-events.json log_data/2018/11/2018-11-13-events.json

Target Structure

Fact Table

songplays

-- records in log data associated with song plays i.e. records with page NextSong

schema:

songplay_id, start_time, user_id, level, song_id, artist_id, session_id, location, user_agent

Dimension Tables

users

-- users in the app

schema:

user_id, first_name, last_name, gender, level

songs

-- songs in music database

schema:

song_id, title, artist_id, year, duration

artists

-- artists in music database

schema:

artist_id, name, location, lattitude, longitude

time

-- timestamps of records in songplays broken down into specific units

schema

start_time, hour, day, week, month, year, weekday

ETL

Part I

1. Create an AWS Role with Administrator access
2. configure dl.cfg with your AWS_ACCESS_ID and AWS_SECRET_KEY
3. Configure the output path in simply_etl.py, etl.py or in emr_etl.py and create_emr_and_start_job.py

Part II

You have three options: -- submit the spark job in your own environment of choosing with: ----> python3 etl.py ------> I wouldn't reccomend this unless your computer is very fast or if you have a spark-hadoop environment setup locally.

-- submit the spark job on your own environment of choosing with: ----> python3 simple_etl.py ------> I would definitely reccomend running this command if you're on local.

These command will run the spark job in your local environment through:

1. reading the data from the udacity's songs_data/ and log_data/ s3 buckets or the data/song_data and data/log_data shortened datasets.
2. creating multiple dimensional tables such as songs and artists through process_song_data() then users and date dimensions through process_log_data
3. Finally process_log_data will get read the songs_dim from the s3 bucket and join said table with date_dim and log_data to build the songsplays fact table.

The third option is to spin up your own EMR instance with the create_emr_and_start_job.py script where it goes through the exact same process as above but the script creates a EMR cluster in AWS at first call. Then runs the emr_etl.py script within a EMR environment, which naturally runs much faster. The create_cluster function within create_emr_and_start_job.py does all the heavy lifting for you. Just provide the the configuration file with the various requested parameters.

Time

EMR instances with the current setup take about 15 minutes to complete the ETL. (python3 create_emr_and_start_job.py)

The simple_etl.py script takes around 20 minutes to run on a local environment

The etl.py script currently takes an undefined amount of time. The internet ingestion and extraction costs are very impactful on performance.

*** Warning *** etl.py takes a while to run in Hadoop Version 2.7.4. I had to run this script in a workspace that isn't able to be upgraded to Hadoop 3.3.1, which is the only way my project doesn't hang with this script.

Thus if you want to run etl.py make sure you have Hadoop 3.3.1 installed in your environment and change line 26 in both etl.py and simple_etl.py from: ".config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:2.7.4")" to ".config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.3.1")" as that'll make the code run a ton faster... like nearly 200%.