Kafka scaling and failover

[egorpoltorackiy]

Hello
I've deployed kafka cluster with 1 replica and then scaled it up to 3 replicas
Updated replication factor for existing topics with kafka-reassign-partitions.sh tool
But when I try to reassign topic "__consumer-offsets" I see error:
Error: org.apache.kafka.common.errors.InvalidTopicException: The request attempted to perform an operation on an invalid topic. java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidTopicException: The request attempted to perform an operation on an invalid topic. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165) at kafka.admin.ReassignPartitionsCommand$.$anonfun$describeTopics$1(ReassignPartitionsCommand.scala:622) at scala.collection.StrictOptimizedMapOps.map(StrictOptimizedMapOps.scala:28) at scala.collection.StrictOptimizedMapOps.map$(StrictOptimizedMapOps.scala:27) at scala.collection.convert.JavaCollectionWrappers$AbstractJMapWrapper.map(JavaCollectionWrappers.scala:313) at kafka.admin.ReassignPartitionsCommand$.describeTopics(ReassignPartitionsCommand.scala:621) at kafka.admin.ReassignPartitionsCommand$.getReplicaAssignmentForPartitions(ReassignPartitionsCommand.scala:660) at kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:760) at kafka.admin.ReassignPartitionsCommand$.handleAction(ReassignPartitionsCommand.scala:238) at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:206) at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala) Caused by: org.apache.kafka.common.errors.InvalidTopicException: The request attempted to perform an operation on an invalid topic.
main goal: to test kafka behavior when one of cluster nodes will go down
So, after reassigning partitions all topics have replication factor 3, except " __consumer_offsets"
./kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic __consumer_offsets Topic: __consumer_offsets TopicId: 0g26EfzLQ7icIaoVWrL0Lw PartitionCount: 50 ReplicationFactor: 1 Configs: compression.type=producer,cleanup.policy=compact,segment.bytes=104857600,message.format.version=3.0-IV1 Topic: __consumer_offsets Partition: 0 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 1 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 2 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 3 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 4 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 5 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 6 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 7 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 8 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 9 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 10 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 11 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 12 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 13 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 14 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 15 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 16 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 17 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 18 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 19 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 20 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 21 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 22 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 23 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 24 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 25 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 26 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 27 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 28 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 29 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 30 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 31 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 32 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 33 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 34 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 35 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 36 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 37 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 38 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 39 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 40 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 41 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 42 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 43 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 44 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 45 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 46 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 47 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 48 Leader: 0 Replicas: 0 Isr: 0 Topic: __consumer_offsets Partition: 49 Leader: 0 Replicas: 0 Isr: 0
new kafka config:
kafka: config: auto.create.topics.enable: false default.replication.factor: 3 offsets.topic.replication.factor: 3 transaction.state.log.min.isr: 2 transaction.state.log.replication.factor: 3
So, when I disable kubernetes node 1 where broker-0 is located , I have issues with kafka cluster
My questions:

1. Should all topics be replicated among all brokers for failover?
2. Is it better to use kafkaRebalance with Cruise Control (not manual kafka-reassign-partitions.sh )?
3. How to change replication factor for existing "__consumers_offset" topic

[egorpoltorackiy]

Sorry
I have to close topic
There was the mistake in my command, that's why I've got error
Error: org.apache.kafka.common.errors.InvalidTopicException: The request attempted to perform an operation on an invalid topic.

[egorpoltorackiy]

Not relevant
Typo in command