Skip to content

Commit 421aa69

Browse files
Merge pull request #278675 from axisc/kafka_compression_docs
adding Kafka compression docs
2 parents a76254e + 624b8ae commit 421aa69

File tree

2 files changed

+10
-12
lines changed

2 files changed

+10
-12
lines changed

articles/event-hubs/apache-kafka-troubleshooting-guide.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,9 @@ You may see consumers not getting any records and constantly rebalancing. In thi
2525
- If your configuration matches those recommended values, and you're still seeing constant rebalancing, feel free to open up an issue (make sure to include your entire configuration in the issue so that we can help debug)!
2626

2727
## Compression/Message format version issue
28-
Kafka supports compression, and Event Hubs for Kafka currently doesn't. Errors that mention a message-format version (for example, `The message format version on the broker does not support the request.`) are caused when a client tries to send compressed Kafka messages to our brokers.
28+
Event Hubs for Kafka currently supports only `gzip` compression algorithm. If any other algorithm is used, client applications will see a message-format version error (for example, `The message format version on the broker does not support the request.`).
2929

30-
If compressed data is necessary, compressing your data before sending it to the brokers and decompressing after receiving is a valid workaround. The message body is just a byte array to the service, so client-side compression/decompression won't cause any issues.
30+
If an unsupported compression algorithm needs to be used, compressing your data with that specific algorithm before sending it to the brokers and decompressing after receiving is a valid workaround. The message body is just a byte array to the service, so client-side compression/decompression won't cause any issues.
3131

3232
## UnknownServerException
3333
You may receive an UnknownServerException from Kafka client libraries similar to the following example:

articles/event-hubs/azure-event-hubs-kafka-overview.md

Lines changed: 8 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,14 @@ Conceptually, Kafka and Event Hubs are very similar. They're both partitioned lo
3131
| Consumer Group | Consumer Group |
3232
| Offset | Offset|
3333

34+
## Apache Kafka features supported on Azure Event Hubs
35+
36+
### Compression
37+
38+
The client-side [compression](https://cwiki.apache.org/confluence/display/KAFKA/Compression) feature in Apache Kafka clients conserves compute resources and bandwidth by compressing a batch of multiple messages into a single message on the producer side and decompressing the batch on the consumer side. The Apache Kafka broker treats the batch as a special message.
39+
40+
Kafka producer application developers can enable message compression by setting the compression.type property. Azure Event Hubs currently supports `gzip` compression.
41+
3442
## Key differences between Apache Kafka and Azure Event Hubs
3543

3644
While [Apache Kafka](https://kafka.apache.org/) is software you typically need to install and operate, Event Hubs is a fully managed, cloud-native service. There are no servers, disks, or networks to manage and monitor and no brokers to consider or configure, ever. You create a namespace, which is an endpoint with a fully qualified domain name, and then you create Event Hubs (topics) within that namespace.
@@ -125,16 +133,6 @@ As explained [above](#is-apache-kafka-the-right-solution-for-your-workload), the
125133

126134
[Azure Service Bus](../service-bus-messaging/service-bus-transactions.md) has robust transaction support that allows receiving and settling messages and sessions while sending outbound messages resulting from message processing to multiple target entities under the consistency protection of a transaction. The feature set not only allows for exactly once processing of each message in a sequence, but also avoids the risk of another consumer inadvertently reprocessing the same messages as it would be the case with Apache Kafka. Service Bus is the recommended service for transactional message workloads.
127135

128-
### Compression
129-
130-
The client-side [compression](https://cwiki.apache.org/confluence/display/KAFKA/Compression) feature of Apache Kafka compresses a batch of multiple messages into a single message on the producer side and decompresses the batch on the consumer side. The Apache Kafka broker treats the batch as a special message.
131-
132-
Kafka producer application developers can enable message compression by setting the compression.type property. In the public preview, the only compression algorithm supported is gzip.
133-
134-
`Compression.type = none | gzip`
135-
136-
The feature is currently only supported for Apache Kafka traffic producer and consumer traffic. AMQP consumer can consume compressed Kafka traffic as decompressed messages. The payload of any Event Hubs event is a byte stream and the content can be compressed with an algorithm of your choosing though in public preview, the only option is gzip. The benefits of using Kafka compression are through smaller message size, increased payload size you can transmit, and lower message broker resource consumption.
137-
138136
### Kafka Streams
139137

140138
Kafka Streams is a client library for stream analytics that is part of the Apache Kafka open-source project, but is separate from the Apache Kafka event broker.

0 commit comments

Comments
 (0)