Default to Apache Kafka instead of Redpanda

[nscuro]

We chose to use Redpanda over Apache Kafka (AK) for our testing as it has a noticeably lower footprint.

However, Redpanda is licensed under BUSL which can be problematic for some organizations. Further, as Dependency-Track is an OSS project under a truly open foundation (OWASP), we should not depend on commercial solutions per default. Instead, we should default to Apache Kafka, which is Apache-2.0 licensed, just like DT itself.

Further, we may want to replace Redpanda Console with something like https://github.com/provectus/kafka-ui (for the same licensing reason).

In summary, the following needs to be done:

• The default docker-compose.yml should use Apache Kafka
  • We can use the bitnami/kafka image which supports KRaft and does not depend on Zookeeper
  • Should use official apache/kafka image, see Default to Apache Kafka instead of Redpanda #854 (comment)
  • Should consider using apache/kafka-native variant, see Default to Apache Kafka instead of Redpanda #854 (comment)
• e2e tests should either:
  • Use AK instead of Redpanda
  • Run against both AK and Redpanda
• Continuing to use Redpanda in unit and integration tests is fine
  • We should still test how AK behaves here, perhaps it's not as bad as we think
  • AK-powered dev services are available through Strimzi

[nscuro]

As of v3.7.0, Kafka has an official Docker image:

• https://hub.docker.com/r/apache/kafka
• https://github.com/apache/kafka/blob/trunk/docker/examples/README.md
• https://github.com/apache/kafka/blob/trunk/docker/examples/jvm/single-node/plaintext/docker-compose.yml

[nscuro]

As per KIP-974, Apache Kafka will ship a native image version with v3.8: https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker

That will be useful for demo setups. The KIP does say explicitly though that production usage is not yet recommended:

The GraalVM native-image based container is recommended only for development, and testing and not for production workloads.