Skip to content

Upgrade to Livy 0.7.1 and separate /jars directory #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
stellarshenson wants to merge 8 commits into
base: master
Choose a base branch
from
Open

Upgrade to Livy 0.7.1 and separate /jars directory #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
7e6d684
upgraded livy to 0.7.1
stellarshenson Jun 15, 2021
bc4bede
changed jars directory
stellarshenson Jun 15, 2021
75faf98
upgraded to spark3
stellarshenson Sep 10, 2021
9077bfa
updated versions
stellarshenson Sep 10, 2021
68d6527
updated mirror for livy
stellarshenson Sep 11, 2021
76a6f9d
removed version 3.1.1 from distro build. spark3 is incompatible with …
stellarshenson Sep 12, 2021
25fa051
fixed typo
stellarshenson Sep 12, 2021
9594f8c
updated pushscript
stellarshenson Sep 12, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 13 additions & 6 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,24 +30,31 @@ RUN mkdir /var/apache-spark-binaries/

# binaries
# apache livy
RUN wget http://mirror.23media.de/apache/incubator/livy/0.5.0-incubating/livy-0.5.0-incubating-bin.zip -O /tmp/livy.zip
RUN unzip /tmp/livy.zip -d /opt/
RUN wget https://apache.mirrors.tworzy.net/incubator/livy/0.7.1-incubating/apache-livy-0.7.1-incubating-bin.zip -O /tmp/livy.zip
RUN unzip /tmp/livy.zip -d /opt/ && mv /opt/apache-livy-0.7.1-incubating-bin /opt/livy

# Logging dir
RUN mkdir /opt/livy-0.5.0-incubating-bin/logs
RUN mkdir /opt/livy/logs

# apache spark
RUN wget https://archive.apache.org/dist/spark/spark-2.3.1/spark-2.3.1-bin-hadoop2.7.tgz -O /tmp/spark-2.3.1-bin-hadoop2.7.tgz
RUN tar -xvzf /tmp/spark-2.3.1-bin-hadoop2.7.tgz -C /opt/
#RUN wget https://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz -O /tmp/spark-3.1.1-bin-hadoop3.2.tgz
RUN wget https://archive.apache.org/dist/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz -O /tmp/spark-2.4.8-bin-hadoop2.7.tgz
#RUN tar -xvzf /tmp/spark-3.1.1-bin-hadoop3.2.tgz -C /opt/ && mv /opt/spark-3.1.1-bin-hadoop3.2 /opt/spark
RUN tar -xvzf /tmp/spark-2.4.8-bin-hadoop2.7.tgz -C /opt/ && mv /opt/spark-2.4.8-bin-hadoop2.7 /opt/spark

# set Python3 as default
RUN rm /usr/bin/python
RUN ln -s /usr/bin/python3 /usr/bin/python

# install R
RUN DEBIAN_FRONTEND="noninteractive" apt-get install r-base-dev -y

# make jars directory (from livy.conf)
RUN mkdir /jars

# expose ports
EXPOSE 8998

# start from init folder
WORKDIR /opt/docker-init
ENTRYPOINT ["./entrypoint"]
ENTRYPOINT ["./entrypoint"]
4 changes: 4 additions & 0 deletions build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
#/bin/sh

docker build . -t stellars/livy-server-docker:spark2.4.8
#docker build . -t stellars/livy-server-docker:spark3.1.1
2 changes: 1 addition & 1 deletion conf/livy.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ livy.server.session.state-retain.sec = 1800s
# List of local directories from where files are allowed to be added to user sessions. By
# default it's empty, meaning users can only reference remote URIs when starting their
# sessions.
livy.file.local-dir-whitelist = /tmp/
livy.file.local-dir-whitelist = /jars/

# Whether to enable csrf protection, by default it is false. If it is enabled, client should add
# http-header "X-Requested-By" in request if the http method is POST/DELETE/PUT/PATCH.
Expand Down
6 changes: 3 additions & 3 deletions init/entrypoint
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ function writeConfigOptions(){
export SPARK_MASTER_PORT=$SPARK_MASTER_PORT
export DEPLOY_MODE=$DEPLOY_MODE

cat /opt/docker-conf/livy.conf | envsubst > /opt/livy-0.5.0-incubating-bin/conf/livy.conf
cat /opt/docker-conf/livy.conf | envsubst > /opt/livy/conf/livy.conf
}

function init(){
Expand All @@ -58,9 +58,9 @@ function init(){

function livy_server_service(){

export SPARK_HOME=/opt/spark-2.3.1-bin-hadoop2.7/
export SPARK_HOME=/opt/spark/
echo "starting Livy Server!"
/opt/livy-0.5.0-incubating-bin/bin/livy-server start
/opt/livy/bin/livy-server start

# whatever blocking call
tail -f /dev/null
Expand Down
3 changes: 3 additions & 0 deletions push.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#/bin/sh
#docker push stellars/livy-server-docker:spark3.1.1
docker push stellars/livy-server-docker:spark2.4.8