Quorum health check in KRaft mode

[katheris]

As part of proposal 91 which discusses how the KafkaRoller should act on a KRaft cluster, we are proposing to only roll a controller node if it will not affect the quorum health to minimise the risk of data loss. There are a few different approaches we could use to work out whether we consider the quorum to be "healthy". To have a more focused discussion we thought it would be useful to open a discussion thread here and then we can put the result of the discussion into the proposal.

Current proposal approach

The current approach described in the proposal is:

1. Create admin client connection to the brokers and call describeMetadataQuorum API
2. From the quorum info returned from the admin API, fetch lastCaughtUpTimestamp for every controller node. lastCaughtUpTimestamp is the last millisecond timestamp at which a replica controller was known to be caught up with the quorum leader
3. Check the quorum leader id using the quorum info and identify the lastCaughtUpTimestamp of the quorum leader
4. Retrieve the current value of Kafka configuration controller.quorum.fetch.timeout.ms
5. Mark a node as caught up if leaderLastCaughtUpTimestamp - replicaLastCaughtUpTimestamp < controllerQuorumFetchTimeoutMs
6. Count each controller node that is caught up, including the leader (numOfCaughtUpControllers)
7. Can roll if: numOfCaughtUpControllers > ceil((double) totalNumOfControllers / 2)

In order to perform quorum check for controller, KafkaRoller creates admin client connection to the brokers. This is required until KIP-919 is implemented.

Problems with proposed approach

We identified some potential problems with the proposed approach:

• The value of controller.quorum.fetch.timeout.ms that matters is the one on the current active controller, it's not clear how we get that specific value if we can't call that controller directly (due to 919 being missing). In the current KafkaRoller code all we can see is the desired value for the configuration
• Does it actually make sense to use controller.quorum.fetch.timeout.ms to verify that a follower is caught up? Given the description of this configuration in the docs, a controller could maybe be making fetch requests but hasn't been able to catch up to the log end offset. Should we instead use some hardcoded time for the comparison?
  • The controller.quorum.fetch.timeout.ms is defined as

  maximum time without a successful fetch from the current leader before becoming a candidate and triggering an election for voters; Maximum time without receiving fetch from a majority of the quorum before asking around to see if there's a new epoch for leader".

• This approach doesn't consider the lag or log end offset as described in the Kafka docs where it details the checks to do before formatting a controller disk

Previous discussions/alternative approaches

This was previously discussed under proposal #82. Some of the other approaches discussed there were:

• Checking expected controller nodes are in the voters list
  • This was rejected because the voters list is static and always includes all nodes
• Requiring that all controller nodes have a lastCaughtUpTimestamp that is close to the leaders' lastCaughtUpTimestamp
  • This was rejected in favour of only requiring a majority to be caught up (excluding the node that is about to be rolled)
• If leader's end offset is not increasing, wait until Lag is 0 for majority
  • This was rejected because there is a periodic heartbeat commit by default, meaning the log end offset will always be increasing
• Pick the latest leader log end offset and wait until all or a majority of followers have reached this offset
  • This could be a good check, but needs some thought around when we choose the log end offset, how long we wait for followers to catch up and what happens of the reconciliation times out and we start again - it could lead to the rolling never finishing

@showuon @fvaleri @tombentley @mimaison What are your thoughts here? If we can find an approach we're happy with that only involves calling the describeMetadataQuorum API and doesn't require knowledge of other configuration that would make things simpler as we can do the same check regardless of whether some nodes have been rolled and some haven't.

The response from describeMetadataQuorum looks like:

NodeId	LogEndOffset	Lag	LastFetchTimestamp	LastCaughtUpTimestamp	Status  	
0     	35250       	0  	1695122352698     	1695122352698        	Leader  	
1     	35250       	0  	1695122352590     	1695122352590        	Follower	
2     	35250       	0  	1695122352593     	1695122352593        	Follower

[tinaselenge]

After discussing this offline, we agreed that the current proposed mechanism for quorum health check is the most reasonably accurate. The operator should determine the health of quorum based on the Kafka's behaviour, controlled by the controller.quorum.fetch.timeout.ms because that indicates how quickly we expect quorum followers to replicate from the leader.

Since we cannot easily get controller's configuration from the controller itself without KIP-919, we have to use the value from the desired configuration, not the current configuration. This is not 100% reliable since we could end up checking the quorum based on configuration value that's not applied to the quorum leader yet. However, this could only happen if user happen to change this specific configuration (controller.quorum.fetch.timeout.ms), therefore we think this is a small risk we can accept.

Also, the quorum health is not specific to Strimzi so it is something that should be discussed more in Kafka upstream. Therefore probably can close this discussion, unless anyone else has an objection.

[showuon]

I think using controller.quorum.fetch.timeout.ms makes sense to me. If this value is set as a small number and causing the quorum cannot be formed, then the quorum health check will fail, which is what we expected.

I also verified that, when the leader node doesn't append no-op messages by setting metadata.max.idle.interval.ms=0, and then, all other 2 nodes crashed. In this case, even though there is no more records appended to the active controller (leader), the LastCaughtUpTimestamp will also be increased each time we query it. Like this:

First time we query it (note: node 6 and 7 are down)

NodeId	LogEndOffset	Lag	LastFetchTimestamp	LastCaughtUpTimestamp	Status  	
1     	702         	0  	1695281445782     	1695281445782        	Leader  	
6     	702         	0  	1695281433705     	1695281433705        	Follower	
7     	702         	0  	1695281434717     	1695281434717        	Follower

2nd time we query it (note: node 6 and 7 are down)
Even though the log end offset is the same (no new records appended), but the LastCaughtUpTimestamp is increasing, while the other 2 nodes are the same numbers. So we under this case, we can successfully identify the quorum is unhealthy.

NodeId	LogEndOffset	Lag	LastFetchTimestamp	LastCaughtUpTimestamp	Status  	
1     	702         	0  	1695281462130     	1695281462130        	Leader  	
6     	702         	0  	1695281433705     	1695281433705        	Follower	
7     	702         	0  	1695281434717     	1695281434717        	Follower	

[fvaleri]

Sorry, I'm late to this party. In Kafka there is no way to enforce the same controller.quorum.fetch.timeout.ms configuration for all controllers, but we can do that in Strimzi right? This would remove the reliability issue mentioned by @tinaselenge. Wdyt?