Skip to content

Zstd compression not working? #248

@BenS1

Description

@BenS1

I'm trying to set my Producer to use Zstd compression, like this:

	// Prepare the configuration
	kafka::Properties props({
		{"bootstrap.servers", {m_BrokerList}}	// The Kafka Brokers
		});

	// The default partitioner is crc_random, but Java decided to use murmur2_random partitioner instead so
	// we need to do the same to be Java compatible
	props.put("partitioner", "murmur2_random");

	// Use Zstd compression for these messages
	props.put("compression.type", "zstd");

	// Intercept logging, so that it logs using our DataDogLogger and not directly to stdout in it's own format
	props.put("log_level", "4");	// Warning level
	props.put("log_cb", KafkaLog);	// Set the logging callback

	// Create a producer instance
	m_KafkaProducer = std::make_shared<kafka::clients::producer::KafkaProducer>(props);

However, this results in the following error in my logs:

Error from Kafka Lib (Level3): KafkaProducer[18f0a03a-ce6be2ba] failed to be initialized with property[compression.type:zstd], result[-1]: Unsupported value "zstd" for configuration property "compression.codec": libzstd not available at build time

It looks like the version of librdkafka used by Modern-cpp-kafka wasn't built with the Zstd library.

Is it possible to enable Zstd compression with Modern-Cpp-Kafka?
I'm guessing I'd need to build librdkafka with Zstd support and use that instead of the version included by Modern-Cpp-Kafka itself?

Thanks
Ben

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions