java.lang.IllegalStateException: Replica [Topic=__org_kroxylicious_testing_consistencyTest,Partition=0,Replica=2] should be in the OfflineReplica,ReplicaDeletionStarted states before moving to ReplicaDeletionIneligible state. Instead it is in ReplicaDeletionSuccessful state during CI run

[k-wall]

Describe the bug

I noticed an ERROR being logged during a CI run during TemplateTest$Tuples.

2024-03-18 05:33:09 ERROR state.change.logger:76 - [Controller id=0 epoch=1] Controller 0 epoch 1 initiated state change of replica 2 for partition __org_kroxylicious_testing_consistencyTest-0 from ReplicaDeletionSuccessful to ReplicaDeletionIneligible failed
java.lang.IllegalStateException: Replica [Topic=__org_kroxylicious_testing_consistencyTest,Partition=0,Replica=2] should be in the OfflineReplica,ReplicaDeletionStarted states before moving to ReplicaDeletionIneligible state. Instead it is in ReplicaDeletionSuccessful state
	at kafka.controller.ZkReplicaStateMachine.logInvalidTransition(ReplicaStateMachine.scala:442) ~[kafka_2.13-3.7.0.jar:?]
	at kafka.controller.ZkReplicaStateMachine.$anonfun$doHandleStateChanges$2(ReplicaStateMachine.scala:164) ~[kafka_2.13-3.7.0.jar:?]
	at kafka.controller.ZkReplicaStateMachine.$anonfun$doHandleStateChanges$2$adapted(ReplicaStateMachine.scala:164) ~[kafka_2.13-3.7.0.jar:?]
	at scala.collection.immutable.List.foreach(List.scala:333) ~[scala-library-2.13.12.jar:?]
	at kafka.controller.ZkReplicaStateMachine.doHandleStateChanges(ReplicaStateMachine.scala:164) ~[kafka_2.13-3.7.0.jar:?]
	at kafka.controller.ZkReplicaStateMachine.$anonfun$handleStateChanges$2(ReplicaStateMachine.scala:112) ~[kafka_2.13-3.7.0.jar:?]
	at kafka.controller.ZkReplicaStateMachine.$anonfun$handleStateChanges$2$adapted(ReplicaStateMachine.scala:111) ~[kafka_2.13-3.7.0.jar:?]
	at kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62) ~[kafka_2.13-3.7.0.jar:?]
	at scala.collection.immutable.HashMap.foreachEntry(HashMap.scala:1131) ~[scala-library-2.13.12.jar:?]
	at kafka.controller.ZkReplicaStateMachine.handleStateChanges(ReplicaStateMachine.scala:111) ~[kafka_2.13-3.7.0.jar:?]
	at kafka.controller.TopicDeletionManager.failReplicaDeletion(TopicDeletionManager.scala:157) ~[kafka_2.13-3.7.0.jar:?]
	at kafka.controller.KafkaController.onReplicasBecomeOffline(KafkaController.scala:682) ~[kafka_2.13-3.7.0.jar:?]
	at kafka.controller.KafkaController.onBrokerFailure(KafkaController.scala:643) ~[kafka_2.13-3.7.0.jar:?]
	at kafka.controller.KafkaController.processBrokerChange(KafkaController.scala:1660) ~[kafka_2.13-3.7.0.jar:?]
	at kafka.controller.KafkaController.process(KafkaController.scala:2620) ~[kafka_2.13-3.7.0.jar:?]
	at kafka.controller.QueuedEvent.process(ControllerEventManager.scala:54) ~[kafka_2.13-3.7.0.jar:?]
	at kafka.controller.ControllerEventManager$ControllerEventThread.process$1(ControllerEventManager.scala:138) ~[kafka_2.13-3.7.0.jar:?]
	at kafka.controller.ControllerEventManager$ControllerEventThread.$anonfun$doWork$1(ControllerEventManager.scala:141) ~[kafka_2.13-3.7.0.jar:?]
	at kafka.controller.ControllerEventManager$ControllerEventThread.$anonfun$doWork$1$adapted(ControllerEventManager.scala:141) ~[kafka_2.13-3.7.0.jar:?]
	at com.yammer.metrics.core.Timer.time(Timer.java:91) [metrics-core-2.2.0.jar:?]
	at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:141) [kafka_2.13-3.7.0.jar:?]
	at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:131) [kafka-server-common-3.7.0.jar:?]

It appears similar to https://issues.apache.org/jira/browse/KAFKA-13906

To Reproduce
Steps to reproduce the behavior:

1. Use kroxylicious-junit5-extension like this ...
2. Run command '....'
3. See error

Expected behavior
A clear and concise description of what you expected to happen.

Logs

Attach or copy and paste relevant logs.

Additional context
Add any other context about the problem here.

[k-wall]

Not sure if there is anything actionable. As this is Zookeeper, maybe we just won't need to care for long.

logs_21818831997.zip

[showuon]

Checking the logs. Will update if I found anything.