Update helm charts to reflect HDFS experiment learnings (#8)

* Update helm charts

* Clean up README.md

* Update README

* Address review comments

Author: kimoonkim

charts/README.md

@@ -0,0 +1,9 @@
+Helm charts for launching HDFS in a K8s cluster. They should be launched in
+the following order.
+
+  1. `hdfs-resolv-conf`: Creates a config map containing resolv.conf used by
+     the HDFS daemons. See `hdfs-resolv-conf/README.md` for how to launch.
+  2. `hdfs-namenode-k8s`: Launches the hdfs namenode. See
+     `hdfs-namenode-k8s/README.md` for how to launch.
+  3. `hdfs-datanode-k8s`: Launches the hdfs datanode daemons. See
+     `hdfs-datanode-k8s/README.md` for how to launch.

charts/hdfs-datanode-k8s/README.md

@@ -6,34 +6,24 @@ HDFS `datanodes` running inside a kubernetes cluster. See the other chart for
   Requires Kubernetes version 1.5 and beyond, because `namenode` is using
   `StatefulSet`, which is available only in version 1.5 and later.
 
-  Make sure `namenode` is fully launched using the other chart. `Datanodes` rely
+  Make sure the `hdfs-resolv-conf` chart is launched. Also ensure `namenode` is
+  fully launched using the corresponding chart. `Datanodes` rely
   on DNS to resolve the hostname of the namenode when they start up.
 
 ### Usage
 
-  1. Find the service IP of your `kube-dns` of your k8s cluster.
-     Try the following command and find the IP value in the output.
-     It will be supplied below as the `clusterDnsIP` parameter.
-
-  ```
-  $ kubectl get svc --all-namespaces | grep kube-dns
-  ```
-
-  2. Optionally, find the domain name of your k8s cluster that become part of
+  1. Optionally, find the domain name of your k8s cluster that becomes part of
      pod and service host names. Default is `cluster.local`. See `values.yaml`
      for additional parameters to change. You can add them below in `--set`,
      as comma-separated entries.
 
-  3. Launch this helm chart, `hdfs-datanode-k8s`, while specifying
-     the kube-dns name server IP and other parameters. (You can add multiple
-     of them below in --set as comma-separated entries)
+  2. Launch this helm chart, `hdfs-datanode-k8s`.
 
   ```
-  $ helm install -n my-hdfs-datanode \
-      --set clusterDnsIP=YOUR-KUBE-DNS-IP hdfs-datanode-k8s
+  $ helm install -n my-hdfs-datanode hdfs-datanode-k8s
   ```
 
-  5. Confirm the daemons are launched.
+  3. Confirm the daemons are launched.
 
   ```
   $ kubectl get pods | grep hdfs-datanode-

charts/hdfs-datanode-k8s/templates/datanode-daemonset.yaml

@@ -12,16 +12,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: resolv-conf-datanode
-data:
-  resolv.conf: |
-    search kube-system.svc.{{ .Values.clusterDomain }} svc.{{ .Values.clusterDomain }} {{ .Values.clusterDomain }}
-    nameserver {{ .Values.clusterDnsIP }}
-    options ndots:5
----
+
 # Deleting a daemonset may need some trick. See
 # https://github.com/kubernetes/kubernetes/issues/33245#issuecomment-261250489
 apiVersion: extensions/v1beta1
@@ -40,10 +31,24 @@ spec:
         - name: datanode
           image: uhopper/hadoop-datanode:2.7.2
           env:
+            # The below uses two loops to make sure the last item does not have comma. It uses index 0
+            # for the last item since that is the only special index that helm template gives us.
+            - name: HDFS_CONF_dfs_datanode_data_dir
+              value: |-
+              {{- range $index, $path := .Values.dataNodeHostPath }}
+                {{- if ne $index 0 }}
+                  /hadoop/dfs/data/{{ $index }},
+                {{- end }}
+              {{- end }}
+              {{- range $index, $path := .Values.dataNodeHostPath }}
+                {{- if eq $index 0 }}
+                  /hadoop/dfs/data/{{ $index }}
+                {{- end }}
+              {{- end }}
             # This works only with /etc/resolv.conf mounted from the config map.
             # K8s version 1.6 will fix this, per https://github.com/kubernetes/kubernetes/pull/29378.
             - name: CORE_CONF_fs_defaultFS
-              value: hdfs://hdfs-namenode-0.hdfs-namenode.kube-system.svc.{{ .Values.clusterDomain }}:8020
+              value: hdfs://hdfs-namenode-0.hdfs-namenode.default.svc.{{ .Values.clusterDomain }}:8020
           livenessProbe:
             initialDelaySeconds: 30
             httpGet:
@@ -53,20 +58,24 @@ spec:
           securityContext:
             privileged: true
           volumeMounts:
-            - name: hdfs-data
-              mountPath: /hadoop/dfs/data
+            {{- range $index, $path := .Values.dataNodeHostPath }}
+            - name: hdfs-data-{{ $index }}
+              mountPath: /hadoop/dfs/data/{{ $index }}
+            {{- end }}
             # Use subPath below to mount only a single file.
             # See https://github.com/dshulyak/kubernetes.github.io/commit/d58ba7b075bb4848349a2c920caaa08ff3773d70
             - name: resolv-conf-volume
               mountPath: /etc/resolv.conf
               subPath: resolv.conf
       restartPolicy: Always
       volumes:
-        - name: hdfs-data
+        {{- range $index, $path := .Values.dataNodeHostPath }}
+        - name: hdfs-data-{{ $index }}
           hostPath:
-            path: {{ .Values.dataNodeHostPath }}
+            path: {{ $path }}
+        {{- end }}
         - configMap:
-            name: resolv-conf-datanode
+            name: hdfs-resolv-conf
             items:
             - key: resolv.conf
               path: resolv.conf

charts/hdfs-datanode-k8s/values.yaml

@@ -15,15 +15,13 @@
 
 # Default values for template variables.
 
-# Set this to the IP of your kube-dns name server that resolv POD host names.
-# This is used by datanode daemons when they connect to namenode using
-# the host name.
-clusterDnsIP: 10.96.0.10
-
 # Set this to the domain name of your cluster that become part of POD and service
 # host names.
 clusterDomain: cluster.local
 
-# Path of the local disk directory on cluster nodes that will contain the datanode
-# blocks. This will be mounted to the namenode as a k8s HostPath volume.
-dataNodeHostPath: /hdfs-data
+# A list of the local disk directories on cluster nodes that will contain the datanode
+# blocks. These paths will be mounted to the datanode as K8s HostPath volumes.
+# In a command line, the list should be enclosed in '{' and '}'.
+# e.g. --set "dataNodeHostPath={/hdfs-data,/hdfs-data1}"
+dataNodeHostPath:
+  - /hdfs-data

charts/hdfs-namenode-k8s/README.md

@@ -6,6 +6,8 @@ HDFS `namenode` running inside a kubernetes cluster. See the other chart for
   Requires Kubernetes version 1.5 and beyond, because `namenode` is using
   `StatefulSet`, which is available only in version 1.5 and later.
 
+  Make sure the `hdfs-resolv-conf` chart is launched.
+
 ### Usage
 
   1. Attach a label to one of your k8s cluster host that will run the `namenode`
@@ -20,7 +22,7 @@ HDFS `namenode` running inside a kubernetes cluster. See the other chart for
   2. Launch this helm chart, `hdfs-namenode-k8s`.
 
   ```
-  $ helm install -n my-hdfs-namenode hdfs-k8s
+  $ helm install -n my-hdfs-namenode hdfs-namenode-k8s
   ```
 
   3. Confirm the daemon is launched.

charts/hdfs-namenode-k8s/templates/namenode-statefulset.yaml

@@ -35,7 +35,7 @@ metadata:
 spec:
   serviceName: "hdfs-namenode"
   # Create a size-1 set. The namenode DNS name will be
-  # hdfs-namenode-0.hdfs-namenode.kube-system.svc.YOUR-CLUSTER-DOMAIN
+  # hdfs-namenode-0.hdfs-namenode.default.svc.YOUR-CLUSTER-DOMAIN
   replicas: 1
   template:
     metadata:
@@ -58,6 +58,11 @@ spec:
           volumeMounts:
             - name: hdfs-name
               mountPath: /hadoop/dfs/name
+            # Use subPath below to mount only a single file.
+            # See https://github.com/dshulyak/kubernetes.github.io/commit/d58ba7b075bb4848349a2c920caaa08ff3773d70
+            - name: resolv-conf-volume
+              mountPath: /etc/resolv.conf
+              subPath: resolv.conf
       # Pin the pod to a node. You can label your node like below:
       #   $ kubectl label nodes YOUR-NODE hdfs-namenode-selector=hdfs-namenode-0
       nodeSelector:
@@ -67,3 +72,9 @@ spec:
         - name: hdfs-name
           hostPath:
             path: {{ .Values.nameNodeHostPath }}
+        - configMap:
+            name: hdfs-resolv-conf
+            items:
+            - key: resolv.conf
+              path: resolv.conf
+          name: resolv-conf-volume

charts/hdfs-resolv-conf/Chart.yaml

@@ -0,0 +1,17 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+name: hdfs-resolv-conf
+version: 0.1
+description: ConfigMap entry containing resolv.conf used by HDFS daemons.

charts/hdfs-resolv-conf/README.md

@@ -0,0 +1,40 @@
+ConfigMap entry storing the resolv.conf file that goes inside HDFS `namenode`
+and `datanode` pods.
+
+### Usage
+
+  1. Find the service IP of your `kube-dns` of your k8s cluster.
+     Try the following command and find the IP value in the output.
+     It will be supplied below as the `clusterDnsIP` parameter.
+
+  ```
+  $ kubectl get svc --all-namespaces | grep kube-dns
+  ```
+
+  2. Find the domain name of your cluster that is part of
+     cluster node host names. e.g. MYCOMPANY.COM in kube-n1.MYCOMPANY.COM.
+     Default is "".  This will be supplied below as
+     the `hostNetworkDomains` parameter.  You can find these from the `search`
+     line in the following `kubectl run` output. `hostNetworkDomains` comes
+     after the pod and service domain name such as `cluster.local`.
+
+  ```
+  $ kubectl run -i -t --rm busybox --image=busybox --restart=Never  \
+      --command -- cat /etc/resolv.conf
+  ...
+  search default.svc.cluster.local svc.cluster.local cluster.local MYCOMPANY.COM
+  ...
+  ```
+
+     See `values.yaml`
+     for additional parameters to change.
+
+  3. Launch this helm chart, `hdfs-resolv-conf`, while specifying
+     the kube-dns name server IP and other parameters. (You can add multiple
+     of them below in --set as comma-separated entries)
+
+  ```
+  $ helm install -n my-hdfs-resolv-conf \
+      --set clusterDnsIP=MY-KUBE-DNS-IP,hostNetworkDomains=MYCOMPANY.COM  \
+      hdfs-resolv-conf
+  ```

charts/hdfs-resolv-conf/templates/resolve-conf-configmap.yaml

@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Required for namenode and datanode to access kube-dns. This is required especially for datanode to
+# access namenode using the statefulset name.
+# K8s version 1.6 will fix this, per https://github.com/kubernetes/kubernetes/pull/29378.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: hdfs-resolv-conf
+data:
+  resolv.conf: |
+    search default.svc.{{ .Values.clusterDomain }} svc.{{ .Values.clusterDomain }} {{ .Values.clusterDomain }} {{ .Values.hostNetworkDomains }}
+    nameserver {{ .Values.clusterDnsIP }}
+    options ndots:5

charts/hdfs-resolv-conf/values.yaml

@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Default values for template variables.
+
+# Set this to the IP of your kube-dns name server that resolv POD host names.
+# This is used by datanode daemons when they connect to namenode using
+# the host name.
+clusterDnsIP: 10.96.0.10
+
+# Set this to the domain name of your host network that becomes part of cluster node
+# host names.  e.g. MYCOMPANY.COM in kube-n1.MYCOMPANY.COM.
+# If there are multiple names, then put them inside the quote separted by
+# spaces.
+hostNetworkDomains: ""
+
+# Set this to the domain name of your cluster that becomes part of POD and service
+# host names.
+clusterDomain: cluster.local