Skip to content

Commit e846c00

Browse files
kimoonkimkimoonkim
authored
Adds a uber chart (#47)
* Add krb5-server chart * krb5-server runs * Support multiple test cases * Add cases * Support skip cleaning up * Kerberos test sets up config and keytabs * Fix bugs * Fix typo * Enable kerberos for journal nodes * Fix a bug * Fix a bug * Specify journalnode princiapls in the namenode config * Test client use kerberos * Use nn0 pod for hdfs admin commands * Run test cases back to back * Describe pod when tests fail * Debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * debug travis * Fix typo * debug travis * debug travis * Do mount --make-share for travis * Works end-to-end even in Travis * Document the krb5-server chart * Minor clean up * Clean up * Use -k option * Add a chart for common configmap * Remove vim file * Switch charts to use the new configmap * Fix a minor bug * Fix a typo * Fix a bug * Disable service level RPC authorization * Add group mapping for the hdfs user * Remove redundant license notices * CI clean up the hdfs-config chart * Remove redundant zookeeper param in CI * Test multiple datanode data dirs * Test custom Hadoop config values in CI * Update README.md's * Cover non-HA in CI * Add a uber-chart * zookeeper and journalnode launch * Clean up structure * clean up * journalnode works * clean up * Include namenode in uber-chart * Fix bugs * debugging namenode startup * Work around zkfc bug * Namenodes get ready after retrying a few times * Include datanodes in uber-chart * Restart datanode pod using livenessProbe when failing to register * Fix a bug * Working on CI now * Basic CI passes * Kerberos CI passes * fix a typo * fix a bug * Add a helm repo * Add subcharts CI case * CI fo non-HA namenode * Use zookeeper 1.0.0 * Fix zookeeper options * Fix a bug in test code * Update travis yaml * Simplifies tags * Include kerberos in uber-chart * Fix a bug in test * Use a tag for kerberos * Check helm diff * Clean up * Add new instructions in README.md * Fix a bug in test code * Delete old instructions. * Fix a bug in tests * Update test instructions * Add more instruction for PVs * Support storage class and pv selector * Add more gold files * Clean up README.md * Fix README.md * Fix a template bug * Fix a template bug * Fix a bug from merge * Updated gold files * Fix non-HA instruction * Mount volumes on the parent dir of metadata dir * Use subpath for metadata * Mount name subdir in journalnodes just in case * Update gold files * Cover how to use an existing zookeeper in README.md * Moved all default options to uber-chart * Update gold * Put some default options for datanode * Update test/README.md * Addressed review comments * Fix tests
1 parent d51c7de commit e846c00

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

57 files changed

+6243
-1121
lines changed

‎.travis.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ sudo: required
22

33
env:
44
- CASES=_basic.sh
5+
- CASES=_basic-subcharts.sh
56
- CASES=_kerberos.sh
67
- CASES=_single-namenode.sh
78

‎charts/README.md

Lines changed: 373 additions & 17 deletions
Large diffs are not rendered by default.

‎charts/hdfs-client-k8s/Chart.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
apiVersion: v1
2+
name: hdfs-client-k8s
3+
version: 0.1.0
4+
description: A client for HDFS on Kubernetes.

‎charts/hdfs-client/templates/client-deployment.yaml renamed to ‎charts/hdfs-client-k8s/templates/client-deployment.yaml

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,18 +2,22 @@ apiVersion: apps/v1
22
apiVersion: extensions/v1beta1
33
kind: Deployment
44
metadata:
5-
name: hdfs-client
5+
name: {{ template "hdfs-k8s.client.fullname" . }}
66
labels:
7-
app: hdfs-client
7+
app: {{ template "hdfs-k8s.client.name" . }}
8+
chart: {{ template "hdfs-k8s.subchart" . }}
9+
release: {{ .Release.Name }}
810
spec:
911
replicas: 1
1012
selector:
1113
matchLabels:
12-
app: hdfs-client
14+
app: {{ template "hdfs-k8s.client.name" . }}
15+
release: {{ .Release.Name }}
1316
template:
1417
metadata:
1518
labels:
16-
app: hdfs-client
19+
app: {{ template "hdfs-k8s.client.name" . }}
20+
release: {{ .Release.Name }}
1721
spec:
1822
containers:
1923
- name: hdfs-client
@@ -30,19 +34,19 @@ spec:
3034
- name: hdfs-config
3135
mountPath: /etc/hadoop-custom-conf
3236
readOnly: true
33-
{{- if .Values.kerberosEnabled }}
37+
{{- if .Values.global.kerberosEnabled }}
3438
- name: kerberos-config
3539
mountPath: /etc/krb5.conf
36-
subPath: {{ .Values.kerberosConfigFileName }}
40+
subPath: {{ .Values.global.kerberosConfigFileName }}
3741
readOnly: true
3842
{{- end }}
3943
restartPolicy: Always
4044
volumes:
4145
- name: hdfs-config
4246
configMap:
43-
name: {{ .Values.hdfsConfigMap }}
44-
{{- if .Values.kerberosEnabled }}
47+
name: {{ template "hdfs-k8s.config.fullname" . }}
48+
{{- if .Values.global.kerberosEnabled }}
4549
- name: kerberos-config
4650
configMap:
47-
name: {{ .Values.kerberosConfigMap }}
51+
name: {{ template "krb5-configmap" . }}
4852
{{- end }}

‎charts/hdfs-client/Chart.yaml

Lines changed: 0 additions & 4 deletions
This file was deleted.

‎charts/hdfs-client/values.yaml

Lines changed: 0 additions & 31 deletions
This file was deleted.

‎charts/hdfs-config-k8s/templates/configmap.yaml

Lines changed: 16 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,17 @@
11
apiVersion: v1
22
kind: ConfigMap
33
metadata:
4-
name: {{ template "hdfs-config-k8s.fullname" . }}
4+
name: {{ template "hdfs-k8s.config.fullname" . }}
55
labels:
6-
app: {{ template "hdfs-config-k8s.name" . }}
7-
chart: {{ template "hdfs-config-k8s.chart" . }}
6+
app: {{ template "hdfs-k8s.client.name" . }}
7+
chart: {{ template "hdfs-k8s.subchart" . }}
88
release: {{ .Release.Name }}
9-
heritage: {{ .Release.Service }}
109
data:
1110
core-site.xml: |
1211
<?xml version="1.0"?>
1312
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
1413
<configuration>
15-
{{- if .Values.kerberosEnabled }}
14+
{{- if .Values.global.kerberosEnabled }}
1615
<property>
1716
<name>hadoop.security.authentication</name>
1817
<value>kerberos</value>
@@ -48,27 +47,27 @@ data:
4847
<value>{{ $value }}</value>
4948
</property>
5049
{{- end }}
51-
{{- if .Values.namenodeHAEnabled }}
50+
{{- if .Values.global.namenodeHAEnabled }}
5251
<property>
5352
<name>fs.defaultFS</name>
5453
<value>hdfs://hdfs-k8s</value>
5554
</property>
5655
<property>
5756
<name>ha.zookeeper.quorum</name>
58-
<value>{{ .Values.zookeeperQuorum }}</value>
57+
<value>{{ template "zookeeper-quorum" . }}</value>
5958
</property>
6059
{{- else }}
6160
<property>
6261
<name>fs.defaultFS</name>
63-
<value>hdfs://hdfs-namenode-0.hdfs-namenode.default.svc.cluster.local:8020</value>
62+
<value>hdfs://{{ template "namenode-svc-0" . }}:8020</value>
6463
</property>
6564
{{- end }}
6665
</configuration>
6766
hdfs-site.xml: |
6867
<?xml version="1.0"?>
6968
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
7069
<configuration>
71-
{{- if .Values.kerberosEnabled }}
70+
{{- if .Values.global.kerberosEnabled }}
7271
<property>
7372
<name>dfs.block.access.token.enable</name>
7473
<value>true</value>
@@ -120,7 +119,7 @@ data:
120119
<name>dfs.datanode.keytab.file</name>
121120
<value>/etc/security/hdfs.keytab</value>
122121
</property>
123-
{{- if .Values.jsvcEnabled }}
122+
{{- if .Values.global.jsvcEnabled }}
124123
<property>
125124
<name>dfs.datanode.address</name>
126125
<value>0.0.0.0:1004</value>
@@ -137,7 +136,7 @@ data:
137136
<value>{{ $value }}</value>
138137
</property>
139138
{{- end }}
140-
{{- if .Values.namenodeHAEnabled }}
139+
{{- if .Values.global.namenodeHAEnabled }}
141140
<property>
142141
<name>dfs.nameservices</name>
143142
<value>hdfs-k8s</value>
@@ -148,23 +147,23 @@ data:
148147
</property>
149148
<property>
150149
<name>dfs.namenode.rpc-address.hdfs-k8s.nn0</name>
151-
<value>hdfs-namenode-0.hdfs-namenode.default.svc.cluster.local:8020</value>
150+
<value>{{ template "namenode-svc-0" . }}:8020</value>
152151
</property>
153152
<property>
154153
<name>dfs.namenode.rpc-address.hdfs-k8s.nn1</name>
155-
<value>hdfs-namenode-1.hdfs-namenode.default.svc.cluster.local:8020</value>
154+
<value>{{ template "namenode-svc-1" . }}:8020</value>
156155
</property>
157156
<property>
158157
<name>dfs.namenode.http-address.hdfs-k8s.nn0</name>
159-
<value>hdfs-namenode-0.hdfs-namenode.default.svc.cluster.local:50070</value>
158+
<value>{{ template "namenode-svc-0" . }}:50070</value>
160159
</property>
161160
<property>
162161
<name>dfs.namenode.http-address.hdfs-k8s.nn1</name>
163-
<value>hdfs-namenode-1.hdfs-namenode.default.svc.cluster.local:50070</value>
162+
<value>{{ template "namenode-svc-1" . }}:50070</value>
164163
</property>
165164
<property>
166165
<name>dfs.namenode.shared.edits.dir</name>
167-
<value>qjournal://{{ .Values.journalQuorum }}/hdfs-k8s</value>
166+
<value>qjournal://{{ template "journalnode-quorum" . }}/hdfs-k8s</value>
168167
</property>
169168
<property>
170169
<name>dfs.ha.automatic-failover.enabled</name>
@@ -193,6 +192,6 @@ data:
193192
</property>
194193
<property>
195194
<name>dfs.datanode.data.dir</name>
196-
<value>{{ template "datanode-data-dirs" . }}</value>
195+
<value>{{ join "," .Values.global.dataNodeHostPath }}</value>
197196
</property>
198197
</configuration>

‎charts/hdfs-config-k8s/values.yaml

Lines changed: 0 additions & 43 deletions
This file was deleted.

‎charts/hdfs-datanode-k8s/Chart.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
apiVersion: v1
22
name: hdfs-datanode-k8s
3-
version: 0.2.0
4-
description: Hadoop Distributed File System (HDFS) hosted by Kubernetes.
3+
version: 0.1.0
4+
description: Datanodes for HDFS on Kubernetes.

‎charts/hdfs-datanode-k8s/README.md

Lines changed: 0 additions & 60 deletions
This file was deleted.

0 commit comments

Comments
 (0)