Since :material-tag: v0.24.0
The Testcontainers module for Kafka.
This module runs Kafka in Kraft mode: Apache Kafka Without ZooKeeper.
Please run the following command to add the Kafka module to your Go dependencies:
go get github.com/testcontainers/testcontainers-go/modules/kafka
- Not available until the next release :material-tag: main
Images apache/kafka, apache/kafka-native (Apache Kafka) are supported by this module in addition to confluentinc/confluent-local (Confluent).
The native container (apache/kafka-native) is based on GraalVM and typically starts several seconds faster than alternatives.
It is recommended to prefer Apache Kafka images over Confluent images, as Confluent has unresolved issue with graceful shutdown.
Apache Kafka Native images are also smallest, however they do not include CLI tools such as kafka-topics.sh.
| Docker Image | Size | Start/stop time | CLI Tools | Graceful Shutdown |
|---|---|---|---|---|
| Apache Kafka Native | 137MB (4.0.1 linux amd) | <1 second | No | OK |
| Apache Kafka | 393MB (4.0.1 linux amd) | ~3-4 seconds | Yes | OK |
| Confluent Kafka | 649MB (7.5.0 linux amd) | ~13-14 seconds | Yes | issue |
!!!info If you use image from custom registry, you might need to override starter script, see Starter script section below.
Apache Kafka Native inside_block:runKafkaContainerApacheNative
Apache Kafka inside_block:runKafkaContainerApacheNotNative
Confluent Kafka inside_block:runKafkaContainerConfluentinc
- Since :material-tag: v0.32.0
!!!info
The RunContainer(ctx, opts...) function is deprecated and will be removed in the next major release of Testcontainers for Go.
The Kafka module exposes one entrypoint function to create the Kafka container, and this function receives three parameters:
func Run(ctx context.Context, img string, opts ...testcontainers.ContainerCustomizer) (*KafkaContainer, error)context.Context, the Go context.string, the Docker image to use.testcontainers.ContainerCustomizer, a variadic argument for passing options.
Use the second argument in the Run function to set a valid Docker image.
In example: Run(context.Background(), "apache/kafka-native:4.0.1").
!!! warning
Module expects that the image in use supports Kraft mode (Kafka without ZooKeeper).
The minimal required version of Confluent images for KRaft mode is confluentinc/confluent-local:7.4.0.
All Apache images support Kraft mode.
The environment variables that are already set by default are:
Environment variables inside_block:envVars
The Kafka container will be started using a custom shell script.
Module would vary the starter script depending on the image in use, using following logic:
- image starts with
apache/kafka: use Apache Kafka starter script. - image starts with
confluentinc/: use Confluent starter script. - otherwise: use Confluent starter script (for backward compatibility).
See also WithApacheFlavor/WithConfluentFlavor and WithStarterScript options to override this behavior.
Apache Kafka starter script inside_block:starterScriptApache
Confluent starter script inside_block:starterScriptConfluentinc
Overriding starter script inside_block:runKafkaContainerWithApacheFlavor
When starting the Kafka container, you can pass options in a variadic way to configure it.
- Not available until the next release :material-tag: main
You can manually specify which flavor of starter script to use with the following options:
With Apache Flavor inside_block:runKafkaContainerWithApacheFlavor
With Confluent Flavor inside_block:runKafkaContainerWithConfluentFlavor
Note that both WithApacheFlavor and WithConfluentFlavor conflict with each other and with WithStarterScript option. An error will be returned if several of those options are provided.
- Not available until the next release :material-tag: main
This allows to provide a completely custom starter script for the Kafka container. Be careful when using this option, as compatibility with any image and module version cannot be guaranteed.
Note that WithStarterScript conflicts with WithApacheFlavor and WithConfluentFlavor options. An error will be returned if several of those options are provided.
{% include "../features/common_functional_options_list.md" %}
The Kafka container exposes the following methods:
- Since :material-tag: v0.24.0
The Brokers(ctx) method returns the Kafka brokers as a string slice, containing the host and the random port defined by Kafka's public port (9093/tcp).
Get Kafka brokers inside_block:getBrokers
- Not available until the next release :material-tag: main
Kafka container would by default be configured with localhost:9095 as one of advertised listeners. This can be used when you need to run CLI commands inside the container, for example with custom wait strategies or to prepare test data.
Here is an example that uses custom wait strategy that checks if listing topics works:
Custom wait strategy inside_block:runKafkaContainerAndUseLocalhostListener
Note: this will not work with apache/kafka-native images, as they do not include CLI tools.