Performance issue in 5.1.3.RELEASE for clustered redis.

[arun-oyo]

I am using version 5.1.3.RELEASE with clustered redis on azure redis for cache.

Cluster configuration: 5 shards with each shard having one master and one replica.

ReadFrom setting is NEAREST.

Operations per second on the cluster : 50K rps having combination of sync as well as async apis.

Problem: When the replica node is restarted in one of the shard, my entire application comes down -> the throughput falls to zero for at least 30 minutes, and I have to restart my application to recover my system.
Error: cannot determine a partition to read for slot

I upgraded the lettuce to version 6.2.4.RELEASE, and again when the replica node was restarted, the system auto-recovered in 2-3 minutes.

What is going wrong in version 5.1.3.RELEASE?

[mp911de]

Do you have topology refresh enabled? Sounds like the topology refresh isn't active if you need to restart to recover.

[arun-oyo]

@mp911de below is the configuration,

ClusterTopologyRefreshOptions clusterTopologyRefreshOptions = ClusterTopologyRefreshOptions.builder()
.enablePeriodicRefresh(Duration.ofSeconds(30))
.enableAllAdaptiveRefreshTriggers()
.build();

ClusterClientOptions clusterClientOptions = ClusterClientOptions.builder()
.topologyRefreshOptions(clusterTopologyRefreshOptions)
.validateClusterNodeMembership(false)
.autoReconnect(true)
.build();

[mp911de]

It would be good to monitor what is actually happening on the client level, what exceptions you experience and to dump the cluster topology view of the client (RedisClusterClient.getPartitions()) vs the cluster topology reported by your cluster nodes.