Request for Support of Custom Readiness Probe

[m2khosravi]

Hi Team,

I've observed that the current readinessProbe functionality in Streamzi Operator only permits basic TCP socket checks for broker liveness by confirming the existence of a file managed by kafka-agent.

readinessProbe:
  exec:
    command:
    - test
    - -f
    - /var/opt/kafka/kafka-ready

However, there's a possibility that the broker process is running without the ability to handle requests effectively. I'm considering solutions, such as enabling an HTTP custom endpoint for readiness checks to determine whether Kafka brokers are genuinely ready to process requests, This would give a better understanding of whether Kafka brokers are really prepared to handle requests.

appreciated your feedback on this matter.

Best,
Mahdi

[scholzj]

The file that the probe checks is created based on the internal Kafka states. If you have some ideas how to improve it, you can share them. But the operator needs to understand how the readiness works and you cannot customize it. I'm not sure I understand what you say about HTTP. Kafka is not HTTP-based, so you cannot probe it with HTTP client. And checking it using Kafka client would be very expensive and might not tell you much more anyway.

[m2khosravi]

Thank you for the feedback. You're right that Kafka is not HTTP based, so checking HTTP endpoints wouldn't be relevant.

My goal is to be able to validate if brokers can actually service client requests before considering pods ready, beyond just a basic process check. I agree checking via Kafka clients on every probe would be expensive.

one idea is that to allow custom exec script for readiness that can run broker CLI tools or check metrics to see if broker work probably, but as you said 'the operator needs to understand how the readiness works and you cannot customize it' it 'd be challenging.

The motivation is to give application more knobs to validate readiness. But happy to brainstorm approaches that don't burden the broker.

[scholzj]

But that is how the agent creates the file today -> it looks into Kafka metrics and based on them it manages the file accordingly. We actually used to use a broker CLI tool in the past and it did not work well because as I said, it creates a lot of load and is very expensive. So as a result it is not very reliable in finding out how well / if the broker works.

[m2khosravi]

so kafka-agent determines broker readiness by looking internal Kafka metrics and server state and relying on Kafka's native signals rather than custom logic makes sense better based on your explanation.