Cruise control in maintenance status when deployed after upgrade from current stable

[deusebio]

I'm currently testing the upgrade scenario from current stable (kafka rev 195 and zookeeper rev 149) to current beta (kafka rev 202 and zookeeper rev 156), and after upgrade to add cruise-control to do rebalancing of the cluster.

However after deploying cruise-control and relating, the cruise-control charms stays in maintenance mode with the message missing broker data.

When deployed from scratch, Kafka and ZooKeeper from beta, CC works fine.

Steps to reproduce

1. Deploy bundle

juju deploy ./bundle.yaml

2. Upgrade

juju run zookeeper/leader pre-upgrade-check --format yaml
juju refresh zookeeper --switch ch:zookeeper --channel 3/beta
juju run kafka/leader pre-upgrade-check --format yaml
juju refresh kafka --switch ch:kafka --channel 3/beta

3. Deploy Cruise-control

juju deploy kafka cruise-control --channel 3/beta --config roles=balancer --config profile=testing
juju relate cruise-control tls
juju relate cruise-control:certificates tls
juju relate cruise-control:peer-cluster kafka:peer-cluster-orchestrator

Expected behavior

Cruise-control gets into active/idle state

Actual behavior

Cruise-control charms stays in maintenance mode with the message missing broker data.

Juju status output is the following:

Model  Controller  Cloud/Region         Version  SLA          Timestamp
kafka  overlord    localhost/localhost  3.6.4    unsupported  17:06:23Z

App              Version  Status       Scale  Charm                     Channel        Rev  Exposed  Mess
age
consumer                  active           1  kafka-test-app            latest/edge     11  no       Please relate with Kafka or use the action to run the application without the relation.
cruise-control            maintenance      1  kafka                     3/beta         202  no       missing broker data
data-integrator           active           1  data-integrator           latest/stable   79  no
kafka            3.9.0    active           3  kafka                     3/beta         202  no       machine system settings are not optimal - see logs for info
producer                  active           1  kafka-test-app            latest/edge     11  no       Please relate with Kafka or use the action to run the application without the relation.
tls                       active           1  self-signed-certificates  latest/stable  155  no
zookeeper        3.9.2    active           3  zookeeper                 3/beta         156  no

Unit                Workload     Agent  Machine  Public address  Ports           Message
consumer/0*         active       idle   0        10.194.169.214                  Please relate with Kafka or use the action to run the application without the relation.
cruise-control/0*   maintenance  idle   10       10.194.169.131                  missing broker data
data-integrator/0*  active       idle   1        10.194.169.106
kafka/0*            active       idle   2        10.194.169.182  9093,19093/tcp  machine system settings are not optimal - see logs for info
kafka/1             active       idle   3        10.194.169.11   9093,19093/tcp  machine system settings are not optimal - see logs for info
kafka/2             active       idle   4        10.194.169.63   9093,19093/tcp  machine system settings are not optimal - see logs for info
producer/0*         active       idle   5        10.194.169.231                  Please relate with Kafka or use the action to run the application without the relation.
tls/0*              active       idle   6        10.194.169.149
zookeeper/0*        active       idle   7        10.194.169.39
zookeeper/1         active       idle   8        10.194.169.96
zookeeper/2         active       idle   9        10.194.169.36

Versions

Operating system: 22.04

Juju CLI: 3.6/stable

Juju agent: 3.6.4

LXD: 5.0.4

Log output

Juju debug log:

Additional context

The bundle used is the first deploy is

default-base: ubuntu@22.04/stable
applications:
  consumer:
    charm: kafka-test-app
    channel: latest/edge
    revision: 11
    num_units: 1
    options:
      consumer_group_prefix: my-cg
      role: consumer
      topic_name: test-topic
    constraints: arch=amd64
  data-integrator:
    charm: data-integrator
    channel: latest/stable
    revision: 79
    num_units: 1
    options:
      consumer-group-prefix: admin-cg
      extra-user-roles: admin
      topic-name: admin-topic
    constraints: arch=amd64
  kafka:
    charm: kafka
    channel: 3/stable
    revision: 195
    num_units: 3
    options:
      profile: testing
    constraints: arch=amd64 instance-type=m1.medium
    storage:
      data: rootfs,1,10240M
  producer:
    charm: kafka-test-app
    channel: latest/edge
    revision: 11
    num_units: 1
    options:
      num_messages: 10000000
      partitions: 2
      replication_factor: 3
      role: producer
      topic_name: test-topic
    constraints: arch=amd64
  tls:
    charm: self-signed-certificates
    channel: latest/stable
    revision: 155
    num_units: 1
    options:
      ca-common-name: Canonical
    constraints: arch=amd64
  zookeeper:
    charm: zookeeper
    channel: 3/stable
    revision: 149
    num_units: 3
    constraints: arch=amd64
    storage:
      data: rootfs,1,1024M
relations:
- - kafka:certificates
  - tls:certificates
- - zookeeper:certificates
  - tls:certificates
- - consumer:certificates
  - tls:certificates
- - producer:certificates
  - tls:certificates
- - kafka:zookeeper
  - zookeeper:zookeeper
- - data-integrator:kafka
  - kafka:kafka-client

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-6859.

This message was autogenerated