feat(kafka): use new apache/kafka-native docker image#2683
feat(kafka): use new apache/kafka-native docker image#2683CTrando wants to merge 5 commits intotestcontainers:mainfrom
Conversation
✅ Deploy Preview for testcontainers-go ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
|
||
| // validateKRaftVersion validates if the image version is compatible with KRaft mode, | ||
| // which is available since version 7.0.0. | ||
| func validateKRaftVersion(fqName string) error { |
There was a problem hiding this comment.
every apache/kafka-native image supports kraft
modules/kafka/kafka_test.go
Outdated
|
|
||
| hostname := inspect.Config.Hostname | ||
|
|
||
| code, r, err := container.Exec(context.Background(), []string{"cat", "/usr/sbin/testcontainers_start.sh"}) |
There was a problem hiding this comment.
this script doesn't exist anymore in the kafka image - also I didn't think it was necessary to check advertisted listeners, considering if it's set incorrectly, the other tests would fail because kafka wouldn't work in the first place
| ctx := context.Background() | ||
|
|
||
| kafkaContainer, err := kafka.Run(ctx, "confluentinc/confluent-local:7.5.0", kafka.WithClusterID("kraftCluster")) | ||
| kafkaContainer, err := kafka.Run(ctx, "apache/kafka-native:3.8.0", kafka.WithClusterID("kraftCluster")) |
There was a problem hiding this comment.
TODO: need some more tests on the client side to demonstrate this works + add tests for setting host port
* main: (302 commits) chore(deps): bump mkdocs-include-markdown-plugin from 6.2.2 to 7.1.5 (testcontainers#3137) chore(deps): bump github.com/shirou/gopsutil/v4 from 4.25.1 to 4.25.4 (testcontainers#3133) chore(deps): bump github.com/docker/docker from 28.0.1+incompatible to 28.1.1+incompatible (testcontainers#3152) feat(memcached): add memcached module (testcontainers#3132) fix(etcd): single node etcd cluster access (testcontainers#3149) feat(valkey): add TLS support for Valkey (testcontainers#3131) fix(dockermodelrunner): wait for the model to be pulled (testcontainers#3125) fix(localstack): remove checksum before parsing version (testcontainers#3130) fix(dockermodelrunner): dependency with socat chore: prepare for next minor development cycle (0.38.0) chore: use new version (v0.37.0) in modules and examples fix: handle stopped containers more gracefully when reuse is enabled (testcontainers#3062) feat(gcloud): add option to run firestore in datastore mode (testcontainers#3009) feat: support for mounting images (testcontainers#3044) chore(ci): close PR if it was sent from main (testcontainers#3123) feat: add `WithReuseByName` for modifying Generic Container Requests (testcontainers#3064) chore(deps): bump github/codeql-action from 3.28.15 to 3.28.16 (testcontainers#3120) chore(deps): bump mkdocs-include-markdown-plugin from 6.2.2 to 7.1.5 (testcontainers#3119) chore(deps): bump github.com/magiconair/properties from 1.8.9 to 1.8.10 (testcontainers#3118) chore(ci): exclude more files for a full-blown build (testcontainers#3122) ...
|
@CTrando thanks for opening this PR, and sorry for getting to it that late. As I'm not a Kafka expert, I wonder if we want to provide a different flavour of kafka for the native image to avoid changing the existing behaviours. @eddumelendez wdyt? |
mdelapenya
changed the title
|
Hi, I believe this pull request can be modified to match Kafka Module of Testcontainers for Java (which supports apache/kafka-native image):
Thank you. |
|
Hi @CTrando, FYI, this can be helpful / connected:
Thank you. |
|
Hi, FYI, support of both bitnami/kafka:4.0 and apache/kafka-native:3.9.1 images is implemented in mabrarov/testcontainers-go-kafka-2206#2. Feel free to use that PR in this one if find it useful. Thank you. |
3 participants
What does this PR do?
Uses the new apache/kafka-native docker image instead of the confluent local docker image. The kafka native docker image is much faster (startup time is maybe a second once the image is downloaded).
Trying to see if there is a desire from the community to use this. I can also add support for authentication mechanisms like SASL_PLAINTEXT if desired.
A current limitation of this however is that the desired host port must be known at container creation time - unlike before where we could dynamically pick a port at runtime. The way it was done before was a bit strange, it was running a kafka instance after the container had already started up I believe, which allowed it to pick the right port to use. Here, when we start up the kafka container it must know the port already.
We can still support picking random ports with some tricks like this, but there's no guarantee there won't be collisions.
Also I realize some of these changes may cause unavoidable breaking changes, so I'm happy to make a v2 module.
Why is it important?
Related issues