Skip to content

Commit 2bb2fdb

Browse files
ndimidukndimiduk
committed
HBASE-27830 Introduce hdfs overlay
1 parent 1a85dd5 commit 2bb2fdb

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

48 files changed

+2760
-2
lines changed

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,3 +32,7 @@ awscli-exe*
3232
*.key
3333
*.repo
3434
*.jar
35+
36+
# detritus produced by kuttl
37+
kubeconfig*
38+
kuttl-report-*.xml

‎hbase-kubernetes-deployment/base/README.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing, software
13+
distributed under the License is distributed on an "AS IS" BASIS,
14+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
See the License for the specific language governing permissions and
16+
limitations under the License.
17+
-->
18+
19+
# Base
20+
21+
Some values such as SERVICE name, SERVICEACCOUNT name,
22+
and RBAC role are hard-coded in the environment-configmap.yaml
23+
and supplied into the pods as environment variables. Other
24+
hardcodings include the service name ('hadoop') and the
25+
namespace we run in (also 'hadoop').
26+
27+
The hadoop Configuration system can interpolate environment variables
28+
into '\*.xml' file values ONLY. See
29+
[Configuration Javadoc](http://hadoop.apache.org/docs/current/api/org/apache/hadoop/conf/Configuration.html)
30+
31+
...but we can not do interpolation of SERVICE name into '\*.xml' file key names
32+
as is needed when doing HA in hdfs-site.xml... so for now, we have
33+
hard-codings in 'hdfs-site.xml' key names. For example, the property key name
34+
`dfs.ha.namenodes.hadoop` has the SERVICE name ('hadoop') in it or the key
35+
`dfs.namenode.http-address.hadoop` (TODO: Fix/Workaround).
36+
37+
Edit of pod resources or jvm args for a process are
38+
done in place in the yaml files or in kustomization
39+
replacements in overlays.

‎hbase-kubernetes-deployment/base/delete-format-hdfs-configmap-job.yaml‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
# Job to delete the 'format-hdfs' configmap after hdfs has come up
18+
# successfully. The 'format-hdfs' configmap is added by running
19+
# 'kubectl -n hadoop apply -k tools/format-hdfs' (You need the
20+
# '-n hadoop' to apply the configmap to the 'hadoop' namespace).
21+
# Add the configmap if you want hdfs to format the filesystem.
22+
# Do this on initial install only or if you want to clean out
23+
# the current HDFS data.
24+
#
25+
# If the 'format-hdfs' configmap is NOT present, this Job exits/completes.
26+
# Otherwise, it keeps probing until HDFS is up and healthy, and then
27+
# this job removes the 'format-hdfs' configmap. The presence of the
28+
# 'format-hdfs' configmap is checked by all hdfs pods on startup. If
29+
# the configmap is present, they clean out their data directories and run
30+
# format/recreate of their data directories. To install the 'format-hdfs'
31+
# configmap, do it before launch of hdfs. See tools/format-hdfs.
32+
---
33+
apiVersion: batch/v1
34+
kind: Job
35+
metadata:
36+
name: delete-format-hdfs-configmap
37+
spec:
38+
ttlSecondsAfterFinished: 300
39+
template:
40+
spec:
41+
containers:
42+
- image: hadoop
43+
name: delete-format-hdfs-configmap
44+
imagePullPolicy: IfNotPresent
45+
command:
46+
- /bin/bash
47+
- -c
48+
- |-
49+
set -xe
50+
# See if 'format-hdfs' configmap is present.
51+
# If not, then there is nothing for this job to do, complete, exit 0.
52+
/tmp/scripts/exists_configmap.sh format-hdfs || {
53+
echo "No 'format-hdfs' configmap found so no work to do; exiting"
54+
exit 0
55+
}
56+
# The `format-hdfs`` configmap is present. Remove it after HDFS is fully up.
57+
/tmp/scripts/jmxping.sh namenode ${HADOOP_SERVICE}
58+
/tmp/scripts/jmxping.sh datanode ${HADOOP_SERVICE}
59+
# TODO: Should we check if ha and if so, if a NN active... get a report on health?
60+
# HDFS is up. Delete the format-hdfs flag.
61+
/tmp/scripts/delete_configmap.sh format-hdfs
62+
resources:
63+
requests:
64+
cpu: '0.2'
65+
memory: 256Mi
66+
limits:
67+
cpu: '0.5'
68+
memory: 512Mi
69+
envFrom:
70+
- configMapRef:
71+
name: environment
72+
volumeMounts:
73+
- mountPath: /tmp/scripts
74+
name: scripts
75+
# Scratch dir is a location where init containers place items for later use
76+
# by the main containers when they run.
77+
- mountPath: /tmp/scratch
78+
name: scratch
79+
serviceAccountName: hadoop
80+
restartPolicy: Never
81+
volumes:
82+
- configMap:
83+
name: scripts
84+
defaultMode: 0555
85+
name: scripts
86+
# Scratch dir is location where init containers place items for later use
87+
# by the main containers when they run.
88+
- emptyDir: {}
89+
name: scratch

‎hbase-kubernetes-deployment/base/environment-configmap.yaml‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
# Common environment variables shared across pods.
18+
# Include w/ the 'envFrom:' directive.
19+
# We have to be pendantic in here. We cannot have a value
20+
# refer to a define made earlier; the interpolation
21+
# doesn't work.
22+
---
23+
apiVersion: v1
24+
kind: ConfigMap
25+
metadata:
26+
name: environment
27+
data:
28+
DOMAIN: svc.cluster.local
29+
# HADOOP_HOME, HADOOP_HDFS_HOME, etc., and HBASE_HOME are provided by the images.
30+
#
31+
# The headless-service pods in our statefulsets come up in.
32+
# See https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id
33+
# The headless-service is defined in the adjacent rbac.yaml.
34+
# Matches the serviceName we have on our statefulsets.
35+
# Required that we create it according to https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#limitations
36+
HADOOP_SERVICE: hadoop
37+
# dfs.http.policy
38+
# If HTTPS_ONLY or HTTPS_OR_HTTP then we'll depend on https in UI and jmx'ing
39+
# and will adjust schema and ports accordingly. If https, we need to get certificates
40+
# so cert-manager, etc., needs to be instaled.
41+
HTTP_POLICY: HTTP_ONLY
42+
DFS_HTTPS_ENABLE: "false"
43+
HBASE_SSL_ENABLED: "false"
44+
HTTP_AUTH: kerberos
45+
# The insecure port for now.
46+
DATANODE_DATA_DIR: /data00/dn
47+
JOURNALNODE_DATA_DIR: /data00/jn
48+
NAMENODE_DATA_DIR: /data00/nn
49+
HDFS_AUDIT_LOGGER: INFO,RFAAUDIT
50+
HADOOP_DAEMON_ROOT_LOGGER: INFO,RFA,CONSOLE
51+
HADOOP_ROOT_LOGGER: INFO,RFA,CONSOLE
52+
HADOOP_SECURITY_LOGGER: INFO,RFAS
53+
HADOOP_CONF_DIR: /etc/hadoop
54+
HADOOP_LOG_DIR: /var/log/hadoop
55+
HADOOP_SECURE_LOG: /var/log/hadoop
56+
HBASE_ROOT_LOGGER: DEBUG,RFA,console
57+
HBASE_LOG_DIR: /var/log/hbase
58+
HBASE_CONF_DIR: /etc/hbase
59+
# if [ "$HBASE_NO_REDIRECT_LOG" != "" ]; then ... so we are asking for NO redirect of logs.
60+
HBASE_NO_REDIRECT_LOG: "true"
61+
HBASE_MANAGES_ZK: "false"
62+
DFS_REPLICATION: "1"
63+
# What percentage of the container memory to give over to the JVM.
64+
# Be aware that we look at the container resource limit, NOT request: e.g. if
65+
# the resource request memory is set to 8G and the limit is 16G and the
66+
# JVM_HEAP_PERCENTAGE_OF_RESOURCE_LIMIT is 50 as in 50%,
67+
# the heap will be set to 8G: i.e. 1/2 of the 16G limit.
68+
# ip-172-18-132-227.us-west-2.compute.internal
69+
# See https://dzone.com/articles/best-practices-java-memory-arguments-for-container
70+
JVM_HEAP_PERCENTAGE_OF_RESOURCE_LIMIT: "45"

‎hbase-kubernetes-deployment/base/java.security‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
networkaddress.cache.ttl=1
18+
networkaddress.cache.negative.ttl=0

‎hbase-kubernetes-deployment/base/jmxexporter.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
# We run the jmxexporter on most all processes to convert jmx metrics to prometheus.
18+
# This is the config file it uses.
19+
#
20+
# Don't lowercase. Leave the metrics in camelcase. Do this because while
21+
# jmxexport can lowercase metrics names, telegraf can't.
22+
#
23+
#lowercaseOutputName: false
24+
#lowercaseOutputLabelNames: false
25+
# From https://godatadriven.com/blog/monitoring-hbase-with-prometheus/
26+
#rules:
27+
# - pattern: HadoopNamespace_([^\W_]+)_table_([^\W_]+)_region_([^\W_]+)_metric_(\w+)
28+
# name: HBase_metric_$4
29+
# labels:
30+
# namespace: "$1"
31+
# table: "$2"
32+
# region: "$3"

‎hbase-kubernetes-deployment/base/kustomization.yaml‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
---
17+
apiVersion: kustomize.config.k8s.io/v1beta1
18+
kind: Kustomization
19+
20+
configMapGenerator:
21+
- name: hadoop-configuration
22+
# Base set of hadoop configurations. Overlays will add to the set here.
23+
files:
24+
- log4j.properties=log4j.properties.hadoop
25+
- name: scripts
26+
# Useful scripts
27+
files:
28+
- scripts/jmxping.sh
29+
- scripts/apiserver_access.sh
30+
- scripts/get_statefulset_replica_count.sh
31+
- scripts/get_statefulset.sh
32+
- scripts/exists_configmap.sh
33+
- scripts/delete_configmap.sh
34+
- scripts/topology.sh
35+
- scripts/describe_node.sh
36+
- scripts/get_node_name_from_pod_IP.sh
37+
- scripts/get_node_labels.sh
38+
- scripts/get_node_labels_from_pod_IP.sh
39+
- scripts/log.sh
40+
options:
41+
disableNameSuffixHash: true
42+
- name: global-files
43+
# Add files used by most/all processes into a global configuration configmap
44+
# accessible to all processes. The environment-configmap defines env varibles used by
45+
# all processes and pods. This configmap loads files used by each process.
46+
files:
47+
- jmxexporter.yaml
48+
- java.security
49+
- ssl-client.xml
50+
- ssl-server.xml
51+
options:
52+
disableNameSuffixHash: true
53+
54+
secretGenerator:
55+
- name: keystore-password
56+
type: Opaque
57+
options:
58+
disableNameSuffixHash: true
59+
literals:
60+
- password=changeit
61+
62+
resources:
63+
- namespace.yaml
64+
# Global environment variables read in by pods
65+
- environment-configmap.yaml
66+
- rbac.yaml
67+
- delete-format-hdfs-configmap-job.yaml
68+
# These depend on cert-manager being installed.
69+
# See https://cert-manager.io/docs/installation/
70+
#- clusterissuer.yaml
71+
#- certificate.yaml

‎hbase-kubernetes-deployment/base/log4j.properties.hadoop‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
hadoop.console.threshold=LOG
18+
hadoop.log.maxbackupindex=20
19+
hadoop.log.maxfilesize=256MB
20+
hadoop.root.logger=TRACE,CONSOLE
21+
hadoop.security.log.file=SecurityAuth-${user.name}.audit
22+
hadoop.security.log.maxbackupindex=20
23+
hadoop.security.log.maxfilesize=256MB
24+
hadoop.security.logger=INFO,RFAS
25+
hdfs.audit.log.maxbackupindex=20
26+
hdfs.audit.log.maxfilesize=256MB
27+
hdfs.audit.logger=INFO,RFAAUDIT
28+
log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false
29+
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
30+
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
31+
log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n
32+
log4j.appender.CONSOLE.Threshold=${hadoop.console.threshold}
33+
log4j.appender.RFA=org.apache.log4j.RollingFileAppender
34+
log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file}
35+
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
36+
log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n
37+
log4j.appender.RFA.MaxBackupIndex=${hadoop.log.maxbackupindex}
38+
log4j.appender.RFA.MaxFileSize=${hadoop.log.maxfilesize}
39+
log4j.appender.RFAAUDIT=org.apache.log4j.RollingFileAppender
40+
log4j.appender.RFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log
41+
log4j.appender.RFAAUDIT.layout=org.apache.log4j.PatternLayout
42+
log4j.appender.RFAAUDIT.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n
43+
log4j.appender.RFAAUDIT.MaxBackupIndex=${hdfs.audit.log.maxbackupindex}
44+
log4j.appender.RFAAUDIT.MaxFileSize=${hdfs.audit.log.maxfilesize}
45+
log4j.appender.RFAS=org.apache.log4j.RollingFileAppender
46+
log4j.appender.RFAS.File=${hadoop.log.dir}/${hadoop.security.log.file}
47+
log4j.appender.RFAS.layout=org.apache.log4j.PatternLayout
48+
log4j.appender.RFAS.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n
49+
log4j.appender.RFAS.MaxBackupIndex=${hadoop.security.log.maxbackupindex}
50+
log4j.appender.RFAS.MaxFileSize=${hadoop.security.log.maxfilesize}
51+
log4j.category.SecurityLogger=${hadoop.security.logger}
52+
log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger}
53+
log4j.logger.org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy=DEBUG
54+
log4j.logger.org.apache.hadoop.net.NetworkTopology=DEBUG
55+
log4j.rootLogger=${hadoop.root.logger}

0 commit comments

Comments
 (0)