Merge pull request #91 from panchul/alekp_kubeflow_install

Adding demo of using TensorBoard with PyTorch and TensorFlow on Azure Stack

Author: initmahesh

Research/kubeflow-on-azure-stack/Readme.md

@@ -8,7 +8,6 @@ This module demonstrates how to create and use a Kubeflow cluster on Azure Stack
 - [Prerequisites](#prerequisites)
   - [Installing Kubernetes manually](installing_kubernetes.md)
   - [Kubernetes Dashboard](#kubernetes-dashboard)
-  - [Tensorboard](#tensorboard)
   - [Persistence on AzureStack](#persistence-on-azure-stack)
 - [Install Kubeflow](#install-kubeflow)
 - [Kubeflow dashboard](#preparing-kubeflow-dashboard) (preparing and using)
@@ -114,61 +113,6 @@ you imported them, you should be able to see the Kubernetes Dashboard in a brows
 
 ![pics/kubernetes_dashboard_intro.png](pics/kubernetes_dashboard_intro.png)
 
-## Tensorboard
-
-You can skip this chapter for now. There is another useful tool to monitor some ML applications if
-they support it. We provided a sample file to start it in your Kubernetes cluster, `tensorboard.yaml`.
-You might contact your cloud administrator to help you establish network access, or you can
-use ssh port forwarding to see it via your desktop's `localhost` address and port 6006.
-
-It will look something like this(for a different app, outside the scope of this demo):
-
-![pics/tensorboard_graph.png](pics/tensorboard_graph.png)
-
-Here is how you would connect your Tensorboard with the persistence we discuss next:
-
-    $ cat tb.yaml
-    apiVersion: extensions/v1beta1
-    kind: Deployment
-    metadata:
-      labels:
-        app: tensorboard
-      name: tensorboard
-    spec:
-      replicas: 1
-      selector:
-        matchLabels:
-          app: tensorboard
-      template:
-        metadata:
-          labels:
-            app: tensorboard
-        spec:
-          volumes:
-          - name: samba-share-volume2
-            persistentVolumeClaim:
-              # claimName: azurefile
-              claimName: samba-share-claim
-          containers:
-          - name: tensorboard
-            image: tensorflow/tensorflow:1.10.0
-            imagePullPolicy: Always
-            command:
-             - /usr/local/bin/tensorboard
-            args:
-            - --logdir
-            - /tmp/tensorflow/logs
-            volumeMounts:
-            - mountPath: /tmp/tensorflow
-              #subPath: somedemo55
-              name: samba-share-volume2
-            ports:
-            - containerPort: 6006
-              protocol: TCP
-          dnsPolicy: ClusterFirst
-          restartPolicy: Always
-
-
 ## Persistence on Azure Stack
 
 Most real-life applications need data storage. Azure Stack team actively works on making
@@ -450,6 +394,56 @@ it will look like so:
 
 You can now re-install it if you would like.
 
+## Tensorboard
+
+You can skip this chapter for now. There is another useful tool to monitor some ML applications if
+they support it. We provided a sample file to start it in your Kubernetes cluster, `tensorboard.yaml`.
+You might contact your cloud administrator to help you establish network access, or you can
+use ssh port forwarding to see it via your desktop's `localhost` address and port 6006.
+
+Here is how you would connect your Tensorboard with the persistence we discuss next:
+
+    $ cat tb.yaml
+    apiVersion: extensions/v1beta1
+    kind: Deployment
+    metadata:
+      labels:
+        app: tensorboard
+      name: tensorboard
+    spec:
+      replicas: 1
+      selector:
+        matchLabels:
+          app: tensorboard
+      template:
+        metadata:
+          labels:
+            app: tensorboard
+        spec:
+          volumes:
+          - name: samba-share-volume2
+            persistentVolumeClaim:
+              # claimName: azurefile
+              claimName: samba-share-claim
+          containers:
+          - name: tensorboard
+            image: tensorflow/tensorflow:1.10.0
+            imagePullPolicy: Always
+            command:
+             - /usr/local/bin/tensorboard
+            args:
+            - --logdir
+            - /tmp/tensorflow/logs
+            volumeMounts:
+            - mountPath: /tmp/tensorflow
+              #subPath: somedemo55
+              name: samba-share-volume2
+            ports:
+            - containerPort: 6006
+              protocol: TCP
+          dnsPolicy: ClusterFirst
+          restartPolicy: Always
+
 ## Next Steps
 
 Proceed to [TensorFlow on Kubeflow Tutorial](tensorflow-on-kubeflow/Readme.md#tensorflow-on-kubeflow-on-azure-stack)
@@ -458,6 +452,11 @@ to learn how to execute `TFJob`s on Kubeflow, in the environment that we just cr
 And then run [PyTorch on Kubeflow Tutorial](pytorch-on-kubeflow/Readme.md#pytorch-on-kubeflow-on-azure-stack) tutorial to learn running
 `PyTorchJob`s.
 
+The PyTorch example we run will log data for TensorBoard, you will see something like this:
+
+![pytorch-on-kubeflow/images/tensorboard_scalars.png](pytorch-on-kubeflow/images/tensorboard_scalars.png)
+
+
 # Links
 
 The following resources might help during troubleshooting or modifications:

Research/kubeflow-on-azure-stack/pics/tensorboard_graph.png

@@ -139,7 +139,7 @@ Shortly after, you should see the pods up-and-running:
 
 At standard Kubernetes Dashboard you could see the pods:
 
-![](images/pytorch_cluster_pods.png)
+![images/pytorch_cluster_pods.png](images/pytorch_cluster_pods.png)
 
 
 To get the information about the running PyTorch workload:
@@ -321,6 +321,12 @@ Look at the logs to see the progress:
     accuracy=0.9872
     saving model to 'mnist_cnn.pt'...
 
+
+And, since in our example we used tensorboardX to write summaries, if you connected the `Tensorboard` logs properly,
+you will see your data in the Tensorboard:
+
+![images/tensorboard_scalars.png](images/tensorboard_scalars.png)
+
 See more on PyTorch at [https://github.com/pytorch/pytorch](https://github.com/pytorch/pytorch)
 
 # Links

Research/kubeflow-on-azure-stack/pics/tensorboard_histograms.png

@@ -0,0 +1,6 @@
+#!/bin/sh
+#
+# extracting TensorBoard's pod name and forwarding 6006 to the outside
+#
+export PODNAME=$(kubectl get pod -l app=tensorboard -o jsonpath='{.items[0].metadata.name}')
+kubectl port-forward ${PODNAME} 6006:6006

Research/kubeflow-on-azure-stack/pics/tensorboard_images.png

@@ -257,6 +257,46 @@ And soon after, you should see the serialized model in your folder:
 
 For more tutorials and How-Tos, see TensorFlow's [save_and_load.ipynb](https://github.com/tensorflow/docs/blob/master/site/en/tutorials/distribute/save_and_load.ipynb) example notebook.
 
+## Tensorboard
+
+There is another useful tool to monitor some ML applications if they support it. We provided a sample file to start it in your Kubernetes cluster, `tensorboard.yaml`.
+
+A concrete example of a tensorboard-using script is in folder `mnist-w-tb`. You will need your
+github account to build the image(substitute `rollingstone` for yours) and run:
+
+    $ cd mnist-w-tb
+    $ docker build -t rollingstone/tf-mnist-w-tb:latest .
+    $ docker push rollingstone/tf-mnist-w-tb:latest
+    $ kubectl create -f tb.yaml
+    $ kubectl create -f tfjob.yaml
+
+You might contact your cloud administrator to help you establish network access, or you can
+use ssh port forwarding to see it via your desktop's `localhost` address and port 6006.
+
+    $ export PODNAME=$(kubectl get pod -l app=tensorboard -o jsonpath='{.items[0].metadata.name}')
+    $ kubectl port-forward ${PODNAME} 6006:6006
+
+It will look something like this:
+
+![../pics/tensorboard_graph.png](../pics/tensorboard_graph.png)
+
+Another tab shows the input images (train/test):
+
+![../pics/tensorboard_images.png](../pics/tensorboard_images.png)
+
+There are scalars the model logged:
+
+![../pics/tensorboard_scalars.png](../pics/tensorboard_scalars.png)
+
+And histograms for different parameters:
+
+![../pics/tensorboard_histograms.png](../pics/tensorboard_histograms.png)
+
+There is a projector that animates the points in the NN layer dimensions:
+
+![../pics/tensorboard_projector.png](../pics/tensorboard_projector.png)
+
+
 ## Next Steps
 
 Proceed to [PyTorch on Kubeflow Tutorial](../pytorch-on-kubeflow/Readme.md) tutorial to learn how to run `PyTorchJob`s.
@@ -267,3 +307,4 @@ For further information:
 
 - https://www.kubeflow.org/docs/components/training/tftraining/
 - https://www.tensorflow.org/
+- https://github.com/Azure/kubeflow-labs