Unable to run kafka in localhost

[jaya-rama]

Hi
Below is my YAML file to deploy kafka, and all the pods are running fine when I deploy kafka cluster but when I use 127.0.0.1:32100 as bootstap server then I'm getting the following error

WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:32100) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

But when I use the same configuration without advertisedHost then kafka is running perfectly fine but it's not running in localhost. Can someone help me in running kafka in localhost

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 2.8.0
    replicas: 3
    listeners:
    - name: external
      port: 9094
      type: nodeport
      tls: false
      configuration:
        bootstrap:
          nodePort: 32100
        brokers:
        - broker: 0
          advertisedHost: 127.0.0.1
          advertisedPort: 12340
          nodePort: 32000
        - broker: 1
          advertisedHost: 127.0.0.1
          advertisedPort: 12341
          nodePort: 32001
        - broker: 2
          advertisedHost: 127.0.0.1
          advertisedPort: 12342
          nodePort: 32002
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.5"
    storage: 
      type: persistent-claim 
      size: 128Mi
  zookeeper:
    replicas: 3
    storage: 
      type: persistent-claim 
      size: 128Mi 
  entityOperator:
    topicOperator: {}
    userOperator: {}

[scholzj]

What does running in localhost mean? The error Connection to node -1 (/127.0.0.1:32100) means that it failed to connect to the bootstrap server - most probably because it cannot connect. Are you sure 127.0.0.1 is really where your Kubernetes cluster is running?

[scholzj]

But is the broker node port listening on the same localhost as your client is running? This would be very rare, this would mean that you are running the client exactly on the Kubernetes worker node where all the Kafka pods are running as well.

[scholzj]

Try something like telnet 127.0.0.1 32100 ... if you do not connect anywhere, it means the brokers are not listening on the localhost where you are connecting.

[scholzj]

PS: On another topic ... 128Mi storage is super small ... you will most probably run out of disk space before you manage to do anything even if you connect. You should give it at least several GBs.

[jaya-rama]

One question generally kafka runs with node ip right? how can I change that and make kafka run on 127.0.0.1 ( so that I can access bootstarp sever using localhost:<node-port> )
My main problem is to run bootstrap on localhost:<node-port> instead of nodeip:<nodeport>

[scholzj]

Maybe in order to make more sense you should start by explaining your use case and your environment (you OS, how you run Kubernetes etc.). Because right now it makes only little sense. In particular you need to explain which local host you want ti to listen to.

• When Strimzi configures the Kafka brokers, they are always configured to listen also on localhost. But this is the Kafka pod localhost.
• When you expose the broker using the node ports services, Kubernetes open a port on the work node(s). TBH I do not know if this port is also opened for the localhost / 127.0.0.1. That is up to Kubernetes. But in any case, this localhost is completely different from the previous localhost.
• Last, when you run the Kafka client it connect to its own localhosts. In some rare cases, you might be running it from the Kubernetes worker node in which case it would be the same localhost as in the pervious point. But that would not be usual - in most cases this would be again comepltely different localhost.

When you configure the advertisedHost field, you will configure the brokers to tell the clients to connect to localhost. But it will not create any infrastructure to do so. So you will need to make sure your self that the connectivity between the localhost` you have in mind and the broker pods actually works.