Skip to content

Commit 80c67ed

Browse files
pierDipipierDipi
authored
Add Apache Kafka Broker (#2740)
Signed-off-by: Pierangelo Di Pilato <[email protected]>
1 parent b4bd42c commit 80c67ed

File tree

3 files changed

+228
-1
lines changed

3 files changed

+228
-1
lines changed

‎docs/eventing/broker/alternate/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,7 @@ implementations provided by Knative Eventing.
66

77
## GCP Broker
88
The GCP Broker is optimized for running in GCP. For more details, refer to the [doc](https://github.com/google/knative-gcp/blob/master/docs/install/install-gcp-broker.md).
9+
10+
## Apache Kafka Broker
11+
12+
For information about the Apache Kafka broker, see [link](../kafka-broker.md).

‎docs/eventing/broker/kafka-broker.md‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
---
2+
title: "Apache Kafka Broker"
3+
weight: 30
4+
type: "docs"
5+
---
6+
7+
The Apache Kafka Broker is a native Broker implementation, that reduces
8+
network hops, supports any Kafka version, and has a better integration
9+
with Apache Kafka for the Knative Broker and Trigger model.
10+
11+
Notable features are:
12+
13+
- Control plane High Availability
14+
- Horizontally scalable data plane
15+
- [Extensively configurable](#Kafka-Producer-and-Consumer-configurations)
16+
- Ordered delivery of events based on [CloudEvents partitioning extension](https://github.com/cloudevents/spec/blob/master/extensions/partitioning.md)
17+
- Support any Kafka version, see [compatibility matrix](https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix)
18+
19+
## Prerequisites
20+
21+
[Knative Eventing installation](./../../install/any-kubernetes-cluster.md#installing-the-eventing-component).
22+
23+
## Installation
24+
25+
1. Install the Kafka broker by entering the following command:
26+
27+
```bash
28+
kubectl apply --filename {{< artifact org="knative-sandbox" repo="eventing-kafka-broker" file="eventing-kafka-broker.yaml" >}}
29+
```
30+
31+
2. Verify that `kafka-broker-controller`, `kafka-broker-receiver` and `kafka-broker-dispatcher` are running,
32+
by entering the following command:
33+
34+
```bash
35+
kubectl get deployments.apps -n knative-eventing
36+
```
37+
38+
Example output:
39+
40+
```bash
41+
NAME READY UP-TO-DATE AVAILABLE AGE
42+
eventing-controller 1/1 1 1 10s
43+
eventing-webhook 1/1 1 1 9s
44+
kafka-broker-controller 1/1 1 1 3s
45+
kafka-broker-dispatcher 1/1 1 1 4s
46+
kafka-broker-receiver 1/1 1 1 5s
47+
```
48+
49+
## Create a Kafka Broker
50+
51+
A Kafka Broker object looks like this:
52+
53+
```yaml
54+
apiVersion: eventing.knative.dev/v1
55+
kind: Broker
56+
metadata:
57+
annotations:
58+
# case-sensitive
59+
eventing.knative.dev/broker.class: Kafka
60+
name: default
61+
namespace: default
62+
spec:
63+
# Configuration specific to this broker.
64+
config:
65+
apiVersion: v1
66+
kind: ConfigMap
67+
name: kafka-broker-config
68+
namespace: knative-eventing
69+
# Optional dead letter sink, you can specify either:
70+
# - deadLetterSink.ref, which is a reference to a Callable
71+
# - deadLetterSink.uri, which is an absolute URI to a Callable (It can potentially be out of the Kubernetes cluster)
72+
delivery:
73+
deadLetterSink:
74+
ref:
75+
apiVersion: serving.knative.dev/v1
76+
kind: Service
77+
name: dlq-service
78+
```
79+
80+
`spec.config` should reference any `ConfigMap` that looks like the following:
81+
82+
```yaml
83+
apiVersion: v1
84+
kind: ConfigMap
85+
metadata:
86+
name: kafka-broker-config
87+
namespace: knative-eventing
88+
data:
89+
# Number of topic partitions
90+
default.topic.partitions: "10"
91+
# Replication factor of topic messages.
92+
default.topic.replication.factor: "1"
93+
# A comma separated list of bootstrap servers. (It can be in or out the k8s cluster)
94+
bootstrap.servers: "my-cluster-kafka-bootstrap.kafka:9092"
95+
```
96+
97+
The above `ConfigMap` is installed in the cluster. You can edit
98+
the configuration or create a new one with the same values
99+
depending on your needs.
100+
101+
## Set as default broker implementation
102+
103+
To set the Kafka broker as the default implementation for all brokers in the Knative deployment,
104+
you can apply global settings by modifying the `config-br-defaults` ConfigMap in the `knative-eventing` namespace.
105+
106+
This allows you to avoid configuring individual or per-namespace settings for each broker,
107+
such as `metadata.annotations.eventing.knative.dev/broker.class` or `spec.config`.
108+
109+
<<<<<<< HEAD
110+
The following YAML is an example of a config-br-defaults ConfigMap using Kafka broker as the default implementation.
111+
=======
112+
This allows you to avoid configuring individual or per-namespace settings for each broker, such as `metadata.annotations.eventing.knative.dev/broker.class` or
113+
`spec.config`.
114+
>>>>>>> 26ab7c8d... lint
115+
116+
```yaml
117+
apiVersion: v1
118+
kind: ConfigMap
119+
metadata:
120+
name: config-br-defaults
121+
namespace: knative-eventing
122+
data:
123+
default-br-config: |
124+
clusterDefault:
125+
brokerClass: Kafka
126+
apiVersion: v1
127+
kind: ConfigMap
128+
name: kafka-broker-config
129+
namespace: knative-eventing
130+
namespaceDefaults:
131+
namespace1:
132+
brokerClass: Kafka
133+
apiVersion: v1
134+
kind: ConfigMap
135+
name: kafka-broker-config
136+
namespace: knative-eventing
137+
namespace2:
138+
brokerClass: Kafka
139+
apiVersion: v1
140+
kind: ConfigMap
141+
name: kafka-broker-config
142+
namespace: knative-eventing
143+
```
144+
145+
## Kafka Producer and Consumer configurations
146+
147+
Knative exposes all available Kafka producer and consumer configurations that can be modified to suit your workloads.
148+
149+
You can change these configurations by modifying the `config-kafka-broker-data-plane` `ConfigMap` in
150+
the `knative-eventing` namespace.
151+
152+
Documentation for the settings available in this `ConfigMap` is available on the
153+
[Apache Kafka website](https://kafka.apache.org/documentation/),
154+
in particular, [Producer configurations](https://kafka.apache.org/documentation/#producerconfigs)
155+
and [Consumer configurations](https://kafka.apache.org/documentation/#consumerconfigs).
156+
157+
## Enable debug logging for data plane components
158+
159+
The following YAML shows the default logging configuration for data plane components, that is created during the
160+
installation step:
161+
162+
```yaml
163+
apiVersion: v1
164+
kind: ConfigMap
165+
metadata:
166+
name: kafka-broker-config-logging
167+
namespace: knative-eventing
168+
data:
169+
config.xml: |
170+
<configuration>
171+
<appender name="jsonConsoleAppender" class="ch.qos.logback.core.ConsoleAppender">
172+
<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
173+
</appender>
174+
<root level="INFO">
175+
<appender-ref ref="jsonConsoleAppender"/>
176+
</root>
177+
</configuration>
178+
```
179+
180+
To change the logging level to `DEBUG`, you need to:
181+
182+
1. Apply the following `kafka-broker-config-logging` `ConfigMap` or replace `level="INFO"` with `level="DEBUG"` to the
183+
`ConfigMap` `kafka-broker-config-logging`:
184+
185+
```yaml
186+
apiVersion: v1
187+
kind: ConfigMap
188+
metadata:
189+
name: kafka-broker-config-logging
190+
namespace: knative-eventing
191+
data:
192+
config.xml: |
193+
<configuration>
194+
<appender name="jsonConsoleAppender" class="ch.qos.logback.core.ConsoleAppender">
195+
<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
196+
</appender>
197+
<root level="DEBUG">
198+
<appender-ref ref="jsonConsoleAppender"/>
199+
</root>
200+
</configuration>
201+
```
202+
203+
2. Restart the `kafka-broker-receiver` and the `kafka-broker-dispatcher`, by entering the following commands:
204+
205+
```bash
206+
kubectl rollout restart deployment -n knative-eventing kafka-broker-receiver
207+
kubectl rollout restart deployment -n knative-eventing kafka-broker-dispatcher
208+
```
209+
210+
### Additional information
211+
212+
- To report bugs or add feature requests, open an issue in the [eventing-kafka-broker repository](https://github.com/knative-sandbox/eventing-kafka-broker).

‎docs/install/any-kubernetes-cluster.md‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -605,8 +605,19 @@ kubectl apply --filename {{< artifact repo="eventing" file="in-memory-channel.ya
605605
1. Install a Broker (eventing) layer:
606606

607607
<!-- This indentation is important for things to render properly. -->
608-
609608
{{< tabs name="eventing_brokers" default="MT-Channel-based" >}}
609+
{{% tab name="Apache Kafka Broker" %}}
610+
{{< feature-state version="v0.17" state="alpha" >}}
611+
The following command installs the Apache Kafka broker, and runs event routing in a system namespace,
612+
`knative-eventing`, by default.
613+
614+
```bash
615+
kubectl apply --filename {{< artifact org="knative-sandbox" repo="eventing-kafka-broker" file="eventing-kafka-broker.yaml" >}}
616+
```
617+
618+
For more information, see the [Kafka Broker](./../eventing/broker/kafka-broker.md) documentation.
619+
{{< /tab >}}
620+
610621
{{% tab name="MT-Channel-based" %}}
611622
{{< feature-state version="v0.16" state="stable" >}}
612623

0 commit comments

Comments
 (0)