Improve torchx/resources README (#505)

Summary:
Pull Request resolved: #505

#470

* Added steps to generate the "torchx-dev-eks.yml"
* Updated "torchx-dev-eks-template.yml"
* Used https://www.kubeflow.org/docs/components/pipelines/installation/standalone-deployment/#deploying-kubeflow-pipelines to update the KFP creation steps
* Fixed typos and removed stale link

Pull Request resolved: #494

Reviewed By: kurman

Differential Revision: D36416809

Pulled By: scottilee

fbshipit-source-id: d7c33d4c672cda1d6e2f23a18564139af15e2e05

Author: scottilee

resources/README.md

@@ -1,15 +1,28 @@
-The readme describes how to create and delete eks cluster and kfp services.
+The readme describes how to create and delete an EKS cluster and KFP services.
 
 #### Creating EKS cluster
 
+    export CLUSTER_NAME="torchx-dev"
+    export EKS_VERSION="1.21"
+    envsubst < torchx-dev-eks-template.yml > torchx-dev-eks.yml
     eksctl create cluster -f torchx-dev-eks.yml
 
+See https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html for the latest EKS version
+
 #### Creating KFP
 
-    kfctl apply -V -f torchx-dev-kfp.yml
+    Source doc: https://www.kubeflow.org/docs/components/pipelines/installation/standalone-deployment/#deploying-kubeflow-pipelines
+
+    export PIPELINE_VERSION=1.8.1
+    kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
+    kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
+    kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=$PIPELINE_VERSION"
+
+See https://github.com/kubeflow/pipelines/releases for the latest KFP version
 
-#### Applying kfp role binding
+#### Applying KFP role binding
 
+    kubectl create namespace torchx-dev
     kubectl apply -f kfp_volcano_role_binding.yaml
 
 #### Creating torchserve
@@ -22,16 +35,6 @@ The readme describes how to create and delete eks cluster and kfp services.
 
     Install `vcctl`
 
-
-#### Installing kfp from source code
-
-    Source doc: https://www.kubeflow.org/docs/components/pipelines/installation/standalone-deployment/
-
-    kubectl apply -k manifests/kustomize/cluster-scoped-resources
-
-    kubectl apply -k manifests/kustomize/env/dev
-
-
 #### Starting etcd service
 
     kubectl apply -f etcd.yaml
@@ -44,21 +47,20 @@ The readme describes how to create and delete eks cluster and kfp services.
 
     eksctl delete -f torch-dev-eks.yml
 
-This command most likely will fail. EKS user cloudformation to create many resources, that
-are hard to remove. If the command fails there needs to be done manual cleanup:
+This command most likely will fail. EKS uses CloudFormation to create many resources that
+are hard to remove. If the command fails there needs to be manual cleanup:
 * Clean up the associated VPC. Go to AWS Console -> VPC -> Press `Delete`. This will
 point you the ENI and NAT that needs to be deleted manually.
-* Clean up the cloudformation temalte. Go to AWS Console -> CNF -> delete corresponding templates.
+* Clean up the CloudFormation template. Go to AWS Console -> CNF -> delete corresponding templates.
 
 ### Gotchas:
 
-* The directory where `torchx-dev-kfp.yml` is located should be the same name
-    as eks cluster
+* The directory where `torchx-dev-kfp.yml` is located should be the same name as eks cluster
 
-* The node groups in eks cluster HAVE to be spread more than a single AZ, otherwise there
+* The node groups in the EKS cluster HAVE to be spread to more than a single AZ, otherwise there
  will be problems with `istio-ingress`
 
-* KFP troubleshooting: https://www.kubeflow.org/docs/distributions/aws/troubleshooting-aws/
+* KFP troubleshooting: https://www.kubeflow.org/docs/components/pipelines/troubleshooting/
 
 * Enable Kubernetes nodes to access AWS account resources: https://stackoverflow.com/a/64617080/1446208
 

resources/torchx-dev-eks-template.yml

@@ -3,7 +3,7 @@
 
 # Running script:
 # export CLUSTER_NAME="torchx-dev"
-# export KFP_VERSION="1.18"
+# export EKS_VERSION="1.21" # See https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html for latest EKS versions
 # envsubst < torchx-dev-eks-template.yml > torchx-dev-eks.yml
 # eksctl create cluster -f torchx-dev-eks.yml
 
@@ -20,8 +20,7 @@ kind: ClusterConfig
 metadata:
   name: ${CLUSTER_NAME}
   region: us-west-2
-  # https://www.kubeflow.org/docs/distributions/aws/deploy/install-kubeflow/
-  version: '${KFP_VERSION}'
+  version: '${EKS_VERSION}'
   tags:
     environment: dev