Connecting Druid to Kafka

[anmoldukaan]

Hi I'm trying to connect druid with kafka over 9092 with tls disabled and authentication type scram-sha-512, when i provide the password and user name for the kafka user to druid, i'm getting the following error org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-kafka-supervisor-abailjnc-14, groupId=kafka-supervisor-abailjnc] Connection to node -1 (my-cluster-kafka-bootstrap.kafka.svc.cluster.local/10.102.106.33:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue

Any idea how to proceed

[anmoldukaan]

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 3.3.1
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
authentication:
type: scram-sha-512
- name: tls
port: 9093
type: internal
tls: true
authentication:
type: scram-sha-512
authorization:
type: simple
superUsers:
- my-user
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
default.replication.factor: 3
min.insync.replicas: 2
inter.broker.protocol.version: "3.3"
ssl.endpoint.identification.algorithm: " "
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 100Gi
deleteClaim: false
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 100Gi
deleteClaim: false
entityOperator:
template:
topicOperatorContainer:
env:
- name: STRIMZI_USE_ZOOKEEPER_TOPIC_STORE
value: "true"
topicOperator: {}
userOperator: {}

[scholzj]

If you properly format the error and the Kafka CR as code, you would make it readable for people. You should probably also share the full client log including the configuration so that one can see how the client is configured.

[anmoldukaan]

Got it resolved was error on my part, is there a provision for plaintext coming anytime soon? all my consumers and producers are internal to my network and maintaining too mamy certs is an headache

[scholzj]

I guess you need to clarify what provision for plaintext means.

[anmoldukaan]

currently strimzi supports SASL_PLAINTEXT but few third party apps such as Airbyte for instance need a tls key as well even for SASL_PLAINTEXT, native kafka have PLAIN auth mechanism EXTERNAL:PLAINTEXT is there a way i can achieve the same functionality with strimzi thanks for the response

[scholzj]

I think you are mixing several different things ... such as SASL PLAIN authentication mechanism or SASL_PLAINTEXT / PLAINTEXT security protocol. So it is not really clear what is it you want. Strimzi supports listeners with all of SASL_PLAINTEXT, PLAINTEXT, SSL or SASL_SSL security protocols. We do not support SASL PLAIN authentication mechanisms which is insecure and basically unmanageable.