Skip to content

Using separated roles, controller nodes advertise localhost in advertised.listeners, causing broker reconnection failures #200

@see-quick

Description

@see-quick

When using dedicated roles, controller nodes set their CONTROLLER advertised.listeners using getBootstrapControllers(), which returns CONTROLLER://localhost:mappedPort (i.e., an address meant for external host access). Inside a broker container, localhost resolves to the brokers own loopback, not the controller. This for sure works for combined mode but doesn't for separated roles.

Initial connections work because controller.quorum.voters uses correct Docker network aliases (i.e., broker-0:9094). But there are some edge cases where brokers need to re-connect (i.e., some broker might go down because of memory/cpu limits within CI) and when they try to re-connect:

...
Connection to node 0 (localhost/127.0.0.1:33487) could not be established
...

We need to improve our buildListenersConfig() that would use network alias (i.e., CONTROLLER://broker-{nodeId}:9094). Combined-mode can keep using localhost but we should be consistent I think so maybe for both we can use just network alias. Also we need to consider these two methods getBootstrapControllers() and getNetworkBootstrapServers into account if we do not break anything with such change.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions