Skip to content

Commit fcfbb39

Browse files
v-regandownerv-regandowner
authored
Merge pull request #277382 from PatAltimore/patricka-kafka-compression
Update kafka connector compression and authtype examples
2 parents 1717ab9 + b1d9eb0 commit fcfbb39

File tree

1 file changed

+133
-42
lines changed

1 file changed

+133
-42
lines changed

‎articles/iot-operations/connect-to-cloud/howto-configure-kafka.md

Lines changed: 133 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
---
2-
title: Send receive messages between Azure IoT MQ and Event Hubs or Kafka
3-
description: Learn how to send and receive messages between Azure IoT MQ and Azure Event Hubs or Kafka.
2+
title: Send and receive messages between Azure IoT MQ and Event Hubs or Kafka
3+
description: Learn how to send and receive messages between Azure IoT MQ and Azure Event Hubs or Kafka using the Kafka connector.
44
author: PatAltimore
55
ms.author: patricka
66
ms.subservice: mq
77
ms.topic: how-to
88
ms.custom:
99
- ignite-2023
10-
ms.date: 04/22/2024
10+
ms.date: 06/06/2024
1111

1212
#CustomerIntent: As an operator, I want to understand how to configure Azure IoT MQ to send and receive messages between Azure IoT MQ and Kafka.
1313
---
@@ -115,7 +115,7 @@ az deployment group create \
115115

116116
The *KafkaConnector* custom resource (CR) allows you to configure a Kafka connector that can communicate a Kafka host and Event Hubs. The Kafka connector can transfer data between MQTT topics and Kafka topics, using the Event Hubs as a Kafka-compatible endpoint.
117117

118-
The following example shows a *KafkaConnector* CR that connects to an Event Hubs endpoint using IoT MQ's Azure identity, it assumes other MQ resources were installed using the quickstart:
118+
The following example shows a *KafkaConnector* CR that connects to an Event Hubs endpoint using different authentication types. It assumes other MQ resources were installed using the quickstart:
119119

120120
```yaml
121121
apiVersion: mq.iotoperations.azure.com/v1beta1
@@ -151,7 +151,10 @@ spec:
151151
kubernetes: {}
152152
```
153153
154-
The following table describes the fields in the KafkaConnector CR:
154+
---
155+
156+
157+
The following table describes the fields in the KafkaConnector custom resource:
155158
156159
| Field | Description | Required |
157160
| ----- | ----------- | -------- |
@@ -194,7 +197,7 @@ The authentication field supports different types of authentication methods, suc
194197
| Field | Description | Required |
195198
| ----- | ----------- | -------- |
196199
| enabled | A boolean value that indicates whether authentication is enabled or not. | Yes |
197-
| authType | A field containing the authentication type used. See [Authentication Type](#authentication-type)
200+
| authType | A field containing the authentication type used. See [Authentication Type](#authentication-type) | Yes |
198201

199202
##### Authentication Type
200203

@@ -206,9 +209,51 @@ The authentication field supports different types of authentication methods, suc
206209

207210
To learn how to use Azure Key Vault and the `keyVault` to manage secrets for Azure IoT MQ instead of Kubernetes secrets, see [Manage secrets using Azure Key Vault or Kubernetes secrets](../manage-mqtt-connectivity/howto-manage-secrets.md).
208211

209-
##### Authenticate to Event Hubs
212+
### Authenticate to Event Hubs
213+
214+
#### [Managed identity](#tab/managed-identity)
210215

211-
To connect to Event Hubs using a connection string and Kubernetes secret, use `plain` SASL type and `$ConnectionString` as the username and the full connection string as the password. First create the Kubernetes secret:
216+
To use managed identity, specify it as the only method under authentication. You also need to assign a role to the managed identity that grants permission to send and receive messages from Event Hubs, such as Azure Event Hubs Data Owner or Azure Event Hubs Data Sender/Receiver. To learn more, see [Authenticate an application with Microsoft Entra ID to access Event Hubs resources](/azure/event-hubs/authenticate-application#built-in-roles-for-azure-event-hubs).
217+
218+
```yaml
219+
apiVersion: mq.iotoperations.azure.com/v1beta1
220+
kind: KafkaConnector
221+
metadata:
222+
name: my-eh-connector
223+
namespace: azure-iot-operations # same as one used for other MQ resources
224+
spec:
225+
image:
226+
pullPolicy: IfNotPresent
227+
repository: mcr.microsoft.com/azureiotoperations/kafka
228+
tag: 0.4.0-preview
229+
instances: 2
230+
clientIdPrefix: my-prefix
231+
kafkaConnection:
232+
# Port 9093 is Event Hubs' Kakfa endpoint
233+
# Plug in your Event Hubs namespace name
234+
endpoint: <NAMESPACE>.servicebus.windows.net:9093
235+
tls:
236+
tlsEnabled: true
237+
authentication:
238+
enabled: true
239+
authType:
240+
systemAssignedManagedIdentity:
241+
# plugin in your Event Hubs namespace name
242+
audience: "https://<NAMESPACE>.servicebus.windows.net"
243+
localBrokerConnection:
244+
endpoint: "aio-mq-dmqtt-frontend:8883"
245+
tls:
246+
tlsEnabled: true
247+
trustedCaCertificateConfigMap: "aio-ca-trust-bundle-test-only"
248+
authentication:
249+
kubernetes: {}
250+
```
251+
252+
#### [SASL](#tab/sasl)
253+
254+
To connect to Event Hubs using a connection string and Kubernetes secret, use `plain` SASL type and `$ConnectionString` as the username and the full connection string as the password.
255+
256+
First create the Kubernetes secret:
212257

213258
```bash
214259
kubectl create secret generic cs-secret -n azure-iot-operations \
@@ -219,16 +264,43 @@ kubectl create secret generic cs-secret -n azure-iot-operations \
219264
Then, reference the secret in the configuration:
220265

221266
```yaml
222-
authentication:
223-
enabled: true
224-
authType:
225-
sasl:
226-
saslType: plain
227-
token:
228-
secretName: cs-secret
267+
apiVersion: mq.iotoperations.azure.com/v1beta1
268+
kind: KafkaConnector
269+
metadata:
270+
name: my-eh-connector
271+
namespace: azure-iot-operations # same as one used for other MQ resources
272+
spec:
273+
image:
274+
pullPolicy: IfNotPresent
275+
repository: mcr.microsoft.com/azureiotoperations/kafka
276+
tag: 0.4.0-preview
277+
instances: 2
278+
clientIdPrefix: my-prefix
279+
kafkaConnection:
280+
# Port 9093 is Event Hubs' Kakfa endpoint
281+
# Plug in your Event Hubs namespace name
282+
endpoint: <NAMESPACE>.servicebus.windows.net:9093
283+
tls:
284+
tlsEnabled: true
285+
authentication:
286+
enabled: true
287+
authType:
288+
sasl:
289+
saslType: plain
290+
token:
291+
secretName: cs-secret
292+
localBrokerConnection:
293+
endpoint: "aio-mq-dmqtt-frontend:8883"
294+
tls:
295+
tlsEnabled: true
296+
trustedCaCertificateConfigMap: "aio-ca-trust-bundle-test-only"
297+
authentication:
298+
kubernetes: {}
229299
```
230300

231-
To use Azure Key Vault instead of Kubernetes secrets, create an Azure Key Vault secret with the connection string `Endpoint=sb://..`, reference it with `vaultSecret`, and specify the username as `"$ConnectionString"` in the configuration.
301+
To use Azure Key Vault instead of Kubernetes secrets, create an Azure Key Vault secret with the connection string `Endpoint=sb://..`, reference it with `vaultSecret`, and specify the username as `"$ConnectionString"` in the configuration.
302+
303+
Use the previous YAML example and change the authentication section to the following:
232304

233305
```yaml
234306
authentication:
@@ -249,17 +321,7 @@ authentication:
249321
# version: 939ecc2...
250322
```
251323

252-
To use managed identity, specify it as the only method under authentication. You also need to assign a role to the managed identity that grants permission to send and receive messages from Event Hubs, such as Azure Event Hubs Data Owner or Azure Event Hubs Data Sender/Receiver. To learn more, see [Authenticate an application with Microsoft Entra ID to access Event Hubs resources](/azure/event-hubs/authenticate-application#built-in-roles-for-azure-event-hubs).
253-
254-
```yaml
255-
authentication:
256-
enabled: true
257-
authType:
258-
systemAssignedManagedIdentity:
259-
audience: https://<NAMESPACE>.servicebus.windows.net
260-
```
261-
262-
##### X.509
324+
#### [X.509](#tab/x509)
263325

264326
For X.509, use Kubernetes TLS secret containing the public certificate and private key.
265327

@@ -272,14 +334,41 @@ kubectl create secret tls my-tls-secret -n azure-iot-operations \
272334
Then specify the `secretName` in configuration.
273335

274336
```yaml
275-
authentication:
276-
enabled: true
277-
authType:
278-
x509:
279-
secretName: my-tls-secret
337+
apiVersion: mq.iotoperations.azure.com/v1beta1
338+
kind: KafkaConnector
339+
metadata:
340+
name: my-eh-connector
341+
namespace: azure-iot-operations # same as one used for other MQ resources
342+
spec:
343+
image:
344+
pullPolicy: IfNotPresent
345+
repository: mcr.microsoft.com/azureiotoperations/kafka
346+
tag: 0.4.0-preview
347+
instances: 2
348+
clientIdPrefix: my-prefix
349+
kafkaConnection:
350+
# Port 9093 is Event Hubs' Kakfa endpoint
351+
# Plug in your Event Hubs namespace name
352+
endpoint: <NAMESPACE>.servicebus.windows.net:9093
353+
tls:
354+
tlsEnabled: true
355+
authentication:
356+
enabled: true
357+
authType:
358+
x509:
359+
secretName: my-tls-secret
360+
localBrokerConnection:
361+
endpoint: "aio-mq-dmqtt-frontend:8883"
362+
tls:
363+
tlsEnabled: true
364+
trustedCaCertificateConfigMap: "aio-ca-trust-bundle-test-only"
365+
authentication:
366+
kubernetes: {}
280367
```
281368

282-
To use Azure Key Vault instead, make sure the [certificate and private key are properly imported](../../key-vault/certificates/tutorial-import-certificate.md) and then specify the reference with `vaultCert`.
369+
To use Azure Key Vault instead, make sure the [certificate and private key are properly imported](../../key-vault/certificates/tutorial-import-certificate.md) and then specify the reference with `vaultCert`.
370+
371+
Use the previous YAML example and change the authentication section to the following:
283372

284373
```yaml
285374
authentication:
@@ -310,6 +399,8 @@ keyVault:
310399
# version: 939ecc2...
311400
```
312401

402+
---
403+
313404
### Manage local broker connection
314405

315406
Like MQTT bridge, the Event Hubs connector acts as a client to the IoT MQ MQTT broker. If you've customized the listener port and/or authentication of your IoT MQ MQTT broker, override the local MQTT connection configuration for the Event Hubs connector as well. To learn more, see [MQTT bridge local broker connection](howto-configure-mqtt-bridge.md).
@@ -326,7 +417,7 @@ metadata:
326417
namespace: <SAME NAMESPACE AS BROKER> # For example "default"
327418
spec:
328419
kafkaConnectorRef: my-eh-connector
329-
compression: snappy
420+
compression: none
330421
batching:
331422
enabled: true
332423
latencyMs: 1000
@@ -369,14 +460,14 @@ The following table describes the fields in the KafkaConnectorTopicMap CR:
369460

370461
### Compression
371462

372-
The compression field enables compression for the messages sent to Kafka topics. Compression helps to reduce the network bandwidth and storage space required for data transfer. However, compression also adds some overhead and latency to the process. The supported compression types are listed in the following table.
463+
The compression field enables compression for the messages sent to Kafka topics. Compression helps to reduce the network bandwidth and storage space required for data transfer. However, compression also adds some overhead and latency to the process. The compression types values and support are listed in the following table.
373464

374-
| Value | Description |
375-
| ----- | ----------- |
376-
| none | No compression or batching is applied. *none* is the default value if no compression is specified. |
377-
| gzip | GZIP compression and batching are applied. GZIP is a general-purpose compression algorithm that offers a good balance between compression ratio and speed. |
378-
| snappy | Snappy compression and batching are applied. Snappy is a fast compression algorithm that offers moderate compression ratio and speed. |
379-
| lz4 | LZ4 compression and batching are applied. LZ4 is a fast compression algorithm that offers low compression ratio and high speed. |
465+
| Value | Description | Supported |
466+
| ----- | ----------- | --------- |
467+
| none | No compression or batching is applied. *none* is the default value if no compression is specified. | Yes |
468+
| gzip | GZIP compression and batching are applied. GZIP is a general-purpose compression algorithm that offers a good balance between compression ratio and speed. | Yes. [Event Hubs Premium](../../event-hubs/event-hubs-premium-overview.md) pricing tier is required for GZIP compression. |
469+
| snappy | Snappy compression and batching are applied. Snappy is a fast compression algorithm that offers moderate compression ratio and speed. | Not supported by [Azure Event Hubs](../../event-hubs/azure-event-hubs-kafka-overview.md#compression). Use [Apache Kafka](https://kafka.apache.org). |
470+
| lz4 | LZ4 compression and batching are applied. LZ4 is a fast compression algorithm that offers low compression ratio and high speed. | Not supported by [Azure Event Hubs](../../event-hubs/azure-event-hubs-kafka-overview.md#compression). Use [Apache Kafka](https://kafka.apache.org). |
380471

381472
### Batching
382473

@@ -476,7 +567,7 @@ kafkaToMqtt:
476567
qos: 0
477568
```
478569

479-
In this example, messages from Kafka topic *sending-event-hub** are published to MQTT topic *heater-commands* with QoS level 0.
570+
In this example, messages from Kafka topic *sending-event-hub* are published to MQTT topic *heater-commands* with QoS level 0.
480571

481572
### Event hub name must match Kafka topic
482573

0 commit comments

Comments
 (0)