Skip to content

Commit 5b10690

Browse files
carlosadiegosysdigcarlosadiegosysdig
committed
Added Kafka installation files
1 parent d267e2e commit 5b10690

File tree

5 files changed

+454
-0
lines changed

5 files changed

+454
-0
lines changed

‎resources/kafka/INSTALL.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Prerequisites
2+
3+
# Installation of the JMX-Exporter as a sidecar
4+
The JMX-Exporter can be easily installed in two steps.
5+
6+
First deploy the ConfigMap which contains the Kafka JMX configurations. The following example is for a Kafka cluster which exposes the jmx port 9010:
7+
```
8+
helm repo add promcat-charts https://sysdiglabs.github.io/integrations-charts
9+
helm repo update
10+
helm -n kafka install kafka-jmx-exporter promcat-charts/jmx-exporter --set jmx_port=9010 --set integrationType=kafka --set onlyCreateJMXConfigMap=true
11+
```
12+
13+
Then generate a patch file and apply it to your workload (your Kafka Deployment/StatefulSet/Daemonset). The following example is for a Kafka cluster which exposes the jmx port 9010, and is deployed as a StatefulSet called 'kafka-cp-kafka':
14+
```
15+
helm template kafka-jmx-exporter promcat-charts/jmx-exporter --set jmx_port=9010 --set integrationType=kafka --set onlyCreateSidecarPatch=true > sidecar-patch.yaml
16+
kubectl -n kafka patch sts kafka-cp-kafka --patch-file sidecar-patch.yaml
17+
```
18+
19+
# Create Secrets for Authentication for the Kafka-Exporter
20+
Your Kafka cluster external endpoints might be secured by using authentication for the clients that want to connect to it (TLS, SASL+SCARM, SASL+Kerberos).
21+
If you are going to make the Kafka-Exporter (which will be deployed in the next tab) use these secured external endpoints, then you'll need to create Kubernetes Secrets in the following step.
22+
If you prefer using an internal not-secured (plaintext) endpoint for the Kafka-Exporter to connect to the Kafka cluster, then skip this step.
23+
24+
If using TLS, you'll need to create a Secret which contains the CA, the client certificate and the client key. The names of these files must be "ca.crt", "tls.crt" and "tls.key". The name of the secret can be any name that you want. Example:
25+
```
26+
kubectl create secret generic kafka-exporter-certs --from-file=./tls.key --from-file=./tls.crt --from-file=./ca.crt --dry-run=true -o yaml | kubectl apply -f -
27+
```
28+
29+
If using SASL+SCRAM, you'll need to create a Secret which contains the "username" and "password". Example:
30+
```
31+
echo -n 'admin' > username
32+
echo -n '1f2d1e2e67df' > password
33+
kubectl create secret generic kafka-exporter-sasl-scram --from-file=username --from-file=password --dry-run=true -o yaml | kubectl apply -f -
34+
```
35+
36+
If using SASL+Kerberos, you'll need to create a Secret which contains the "kerberos.conf". If the 'Kerberos Auth Type' is 'keytabAuth', it should also contain the "kerberos.keytab". Example:
37+
```
38+
kubectl create secret generic kafka-exporter-sasl-kerberos --from-file=./kerberos.conf --from-file=./kerberos.keytab --dry-run=true -o yaml | kubectl apply -f -
39+
```
40+
41+
# Installation of the Kafka-Exporter
42+
The Kafka-Exporter can be easily installed with one Helm command. The flags will change depending on the authentication used in Kafka. You can find more info about the flags in the [Kafka Exporter chart values.yaml](https://github.com/sysdiglabs/integrations-charts/blob/main/charts/kafka-exporter/values.yaml).
43+
44+
Example of Kafka-Exporter without auth:
45+
```
46+
helm -n kafka install kafka-exporter promcat-charts/kafka-exporter \
47+
--set namespaceName="kafka" \
48+
--set workloadType="statefulset" \
49+
--set workloadName="kafka" \
50+
--set kafkaServer[0]=kafka-cp-kafka:9092
51+
```
52+
53+
Example of Kafka-Exporter with TLS auth:
54+
```
55+
helm -n kafka install kafka-exporter promcat-charts/kafka-exporter \
56+
--set namespaceName="kafka" \
57+
--set workloadType="statefulset" \
58+
--set workloadName="kafka" \
59+
--set kafkaServer[0]=kafka-cp-kafka:9092 \
60+
--set tls.enabled=true \
61+
--set tls.insecureSkipVerify=false \
62+
--set tls.serverName="kafkaServerName" \
63+
--set tls.secretName="kafka-exporter-certs"
64+
```
65+
66+
Example of Kafka-Exporter with SASL+SCRAM auth:
67+
```
68+
helm -n kafka install kafka-exporter promcat-charts/kafka-exporter \
69+
--set namespaceName="kafka" \
70+
--set workloadType="statefulset" \
71+
--set workloadName="kafka" \
72+
--set kafkaServer[0]=kafka-cp-kafka:9092 \
73+
--set sasl.enabled=true \
74+
--set sasl.handshake=true \
75+
--set sasl.scram.enabled=true \
76+
--set sasl.scram.mechanism="plain" \
77+
--set sasl.scram.secretName="kafka-exporter-sasl-scram"
78+
```
79+
80+
Example of Kafka-Exporter with SASL+Kerberos auth:
81+
```
82+
helm -n kafka install kafka-exporter promcat-charts/kafka-exporter \
83+
--set namespaceName="kafka" \
84+
--set workloadType="statefulset" \
85+
--set workloadName="kafka" \
86+
--set kafkaServer[0]=kafka-cp-kafka:9092 \
87+
--set sasl.enabled=true \
88+
--set sasl.handshake=true \
89+
--set sasl.kerberos.enabled=true \
90+
--set sasl.kerberos.serviceName="kerberos-service" \
91+
--set sasl.kerberos.realm="kerberos-realm" \
92+
--set sasl.kerberos.kerberosAuthType="keytabAuth" \
93+
--set sasl.kerberos.secretName="kafka-exporter-sasl-kerberos"
94+
```
95+
96+
You can find below ConfigMap with the JMX configurations for Kafka, a patch for the JMX-exporter as a sidecar, a deployment with the Kafka-Exporter without auth, and the Sysdig Agent ConfigMap with the Prometheus job to scrape both exporters.

‎resources/kafka/include/deployment.yaml‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: kafka-exporter-kafka-kafka-deploy
5+
namespace: kafka
6+
labels:
7+
helm.sh/chart: kafka-exporter-0.0.1
8+
app.kubernetes.io/name: kafka-exporter-kafka-kafka
9+
app.kubernetes.io/instance: my-kafka-exporter-release
10+
app.kubernetes.io/version: "1.4.2"
11+
app.kubernetes.io/managed-by: Helm
12+
spec:
13+
selector:
14+
matchLabels:
15+
app.kubernetes.io/name: kafka-exporter-kafka-kafka
16+
app.kubernetes.io/instance: my-kafka-exporter-release
17+
replicas: 1
18+
template:
19+
metadata:
20+
annotations:
21+
promcat.sysdig.com/port: "9308"
22+
promcat.sysdig.com/target_ns: "kafka"
23+
promcat.sysdig.com/target_workload_type: "statefulset"
24+
promcat.sysdig.com/target_workload_name: "kafka"
25+
promcat.sysdig.com/integration_type: "kafka"
26+
labels:
27+
app.kubernetes.io/name: kafka-exporter-kafka-kafka
28+
app.kubernetes.io/instance: my-kafka-exporter-release
29+
spec:
30+
containers:
31+
- name: kafka-exporter
32+
image: "quay.io/sysdig/kafka-exporter:v1.5.0"
33+
args:
34+
- '--verbosity=0'
35+
- '--kafka.server=kafka-cp-kafka:9092'
36+
env:
37+
ports:
38+
- name: metrics
39+
containerPort: 9308
40+
protocol: TCP
41+
livenessProbe:
42+
failureThreshold: 3
43+
httpGet:
44+
path: /healthz
45+
port: metrics
46+
scheme: HTTP
47+
initialDelaySeconds: 3
48+
periodSeconds: 30
49+
successThreshold: 1
50+
timeoutSeconds: 9
51+
readinessProbe:
52+
failureThreshold: 2
53+
httpGet:
54+
path: /healthz
55+
port: metrics
56+
scheme: HTTP
57+
initialDelaySeconds: 3
58+
periodSeconds: 15
59+
successThreshold: 1
60+
timeoutSeconds: 9
61+
62+
resources:
63+
limits:
64+
cpu: 100m
65+
memory: 128Mi
66+
requests:
67+
cpu: 100m
68+
memory: 128Mi

0 commit comments

Comments
 (0)