Skip to content
This repository was archived by the owner on Nov 16, 2023. It is now read-only.

Commit 02d1aaa

Browse files
initmaheshinitmahesh
authored
Merge pull request #100 from NealAnalyticsLLC/mlflow-on-azure-stack
Mlflow on azure stack: Notebook for remote use
2 parents 3542d22 + 4f5f7f8 commit 02d1aaa

File tree

9 files changed

+442
-227
lines changed

9 files changed

+442
-227
lines changed

‎Research/mlflow-on-azure-stack/Readme.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,10 @@ This guide demonstrates how to install a remote MLflow Tracking Server on Kubern
55
**Reference Material:**
66
- [Manual Installation Instructions](./docs/manual_installation.md)
77

8+
**Prerequisite:**
9+
- Will need the k8 cluster ".kubeconfig" file on your local machine to execute commands on the k8 cluster
10+
- Below instructions are not intended to be run from the master node, but from another Linux dev environment
11+
- Clone the github repo at "/home/user/" path
812

913
## Step 1: Install Porter
1014
Make sure you have Porter installed. You can find the installation instructions for your OS at the link provided below.
@@ -20,7 +24,12 @@ First you will need to navigate to porter directory in the repository. For examp
2024
```sh
2125
cd ./research/mlflow-on-azure-stack/porter
2226
```
27+
Change the file permissions
2328

29+
```sh
30+
chmod 777 mlflow.sh
31+
chmod 777 kubeflow.sh
32+
```
2433
Next, you will build the porter CNAB
2534

2635
```sh
@@ -51,3 +60,52 @@ porter install --cred MLServicesInstaller
5160
porter upgrade --cred MLServicesInstaller
5261
porter uninstall --cred MLServicesInstaller
5362
```
63+
### Step 5: Check for pods and services
64+
After the installation each of the services gets installed into its own namespace, try below commands to look for pods and services:
65+
66+
```sh
67+
kubectl get pods -n mlflow
68+
kubectl get pods -n kubeflow
69+
```
70+
### Step 6: Opening Kubeflow dashboard
71+
To access the dashboard using external connection, replace "type: NodePort" with "type: LoadBalancer" using the patch command:
72+
73+
```sh
74+
$ kubectl patch svc/istio-ingressgateway -p '{"spec":{"type": "LoadBalancer"}}' -n istio-system
75+
service/istio-ingressgateway patched
76+
```
77+
Then the EXTERNAL-IP will become available from:
78+
79+
```sh
80+
$ kubectl get -w -n istio-system svc/istio-ingressgateway
81+
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
82+
istio-ingressgateway LoadBalancer 10.0.123.210 12.34.56.78 15020:30397/TCP,80:31380/TCP,.. 7m27s
83+
```
84+
![Kubeflow dashboard](./docs/img/kubeflow_dashboard1.png)
85+
86+
Use external-ip to open it in your browser, and make sure your firewall rules allow HTTP port 80.
87+
88+
You can monitor Kubeflow cluster by looking at the Kubernetes status, you might need to wait to let the pods create containers and start.
89+
90+
For more information see [Installing Kubeflow on Azure](https://www.kubeflow.org/docs/azure/deploy/install-kubeflow/)
91+
92+
### Step 7: Opening MLflow dashboard
93+
To access the dashboard using external connection, first we need to get the external-ip:
94+
95+
```sh
96+
$ kubectl get svc -n mlflow
97+
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
98+
mlflow-service LoadBalancer 10.0.176.78 52.250.47.209 5000:31148/TCP 19m
99+
```
100+
Use external-ip to open it in your browser, and make sure your firewall rules allow HTTP port 5000.
101+
102+
![MLflow dashboard](./docs/img/mlflow_dashboard.png)
103+
104+
### Step 8: Creating a Notebook Server
105+
106+
From the Kubeflow dashboard select "Notebook Servers". Pick the namespace you want to create the server under and select "+ New Server".
107+
108+
Enter the desired specs for your server. Make sure the "Custom Image" checkbox is select and input `naedwebs/jupyter-mlflow` in the text field for this option. Click "Launch".
109+
### Step 9: Upload a Notebook
110+
111+
Once your server is running click "Connect". A Jupyter Notebook landing page should load on a new tab. On the right hand side of this page push the "Upload" button and select the MLflow_Tutorial notebook found in the notebooks folder in this repository and hit open. Click the blue "Upload" button that has just appeard. Select the notebook to run it.

‎Research/mlflow-on-azure-stack/docs/img/kubeflow_dashboard1.png‎

105 KB
Loading

‎Research/mlflow-on-azure-stack/docs/img/mlflow_dashboard.png‎

40.9 KB
Loading

‎Research/mlflow-on-azure-stack/jupyter-entrypoint.sh‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/usr/bin/env bash
2+
set -e
3+
4+
mkdir -p /home/jovyan/.ssh
5+
ssh-keyscan ${SFTP_HOST} > /home/jovyan/.ssh/known_hosts
6+
7+
jupyter notebook \
8+
--notebook-dir=/home/jovyan \
9+
--ip=0.0.0.0 \
10+
--no-browser \
11+
--allow-root \
12+
--port=8888 \
13+
--NotebookApp.token='' \
14+
--NotebookApp.password='' \
15+
--NotebookApp.allow_origin='*' \
16+
--NotebookApp.base_url=${NB_PREFIX}

‎Research/mlflow-on-azure-stack/jupyter.Dockerfile‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
FROM jupyter/datascience-notebook:latest
2+
3+
USER root
4+
5+
RUN apt-get -yq update && \
6+
apt-get -yqq install ssh
7+
8+
USER jovyan
9+
10+
RUN pip install pysftp
11+
12+
ADD ./jupyter-entrypoint.sh .
13+
14+
ENV SFTP_HOST=sftp.mlflow
15+
ENV NB_PREFIX /
16+
17+
ENTRYPOINT [ "./jupyter-entrypoint.sh" ]

‎Research/mlflow-on-azure-stack/mlflow-entrypoint.sh‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#!/usr/bin/env bash
2+
set -e
3+
4+
ssh-keyscan ${SFTP_HOST} > /root/.ssh/known_hosts
5+
6+
mlflow server \
7+
--host 0.0.0.0 \
8+
--port 5000 \
9+
--backend-store-uri sqlite:////app/storage/mlflow-history.db \
10+
--default-artifact-root "sftp://demo:demo@$SFTP_HOST/archive"

‎Research/mlflow-on-azure-stack/mlflow.Dockerfile‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,13 @@ ADD . /app
77
RUN mkdir -p /app/storage/mlruns && touch mlflow-history.db
88

99
RUN pip install mlflow
10+
RUN pip install pysftp
11+
12+
ENV SFTP_HOST=sftp.mlflow
13+
RUN mkdir -p /root/.ssh
1014

1115
EXPOSE 5000
1216

13-
ENTRYPOINT mlflow server --host 0.0.0.0 --port 5000 --backend-store-uri sqlite:////app/storage/mlflow-history.db --default-artifact-root /app/storage/mlruns
17+
RUN chmod +x ./mlflow-entrypoint.sh
18+
19+
ENTRYPOINT [ "./mlflow-entrypoint.sh"]

‎Research/mlflow-on-azure-stack/notebooks/MLflow_Tutorial.ipynb‎

Lines changed: 279 additions & 225 deletions
Large diffs are not rendered by default.

‎Research/mlflow-on-azure-stack/porter/tracking_server.yaml‎

Lines changed: 55 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,4 +51,58 @@ spec:
5151
targetPort: 5000
5252
name: mlflow
5353
type: LoadBalancer
54-
54+
---
55+
kind: PersistentVolumeClaim
56+
apiVersion: v1
57+
metadata:
58+
name: sftp-pvc
59+
spec:
60+
storageClassName: default
61+
accessModes:
62+
- ReadWriteOnce
63+
resources:
64+
requests:
65+
storage: 8Gi
66+
---
67+
apiVersion: apps/v1
68+
kind: Deployment
69+
metadata:
70+
name: sftp-deployment
71+
spec:
72+
replicas: 1
73+
selector:
74+
matchLabels:
75+
app: sftp
76+
template:
77+
metadata:
78+
labels:
79+
app: sftp
80+
spec:
81+
terminationGracePeriodSeconds: 10
82+
containers:
83+
- name: sftp
84+
image: atmoz/sftp:alpine
85+
args: ["demo:demo:::archive"]
86+
ports:
87+
- containerPort: 5000
88+
volumeMounts:
89+
- name: archive
90+
mountPath: /home/demo/
91+
volumes:
92+
- name: archive
93+
persistentVolumeClaim:
94+
claimName: sftp-pvc
95+
---
96+
kind: Service
97+
apiVersion: v1
98+
metadata:
99+
name: sftp
100+
spec:
101+
selector:
102+
app: sftp
103+
ports:
104+
- protocol: TCP
105+
port: 22
106+
targetPort: 22
107+
name: mlflow
108+
type: LoadBalancer

0 commit comments

Comments
 (0)