AWS Elasticache: Lettuce ROLE to master node fails and connection to primary never gets established.

[amaan75]

Hi, I am using lettuce with spring data Redis
LettuceCore : 6.1.4.RELEASE
spring-data-redis: 2.5.4
AWS elasticache for Redis 6.X (non clustered mode)
while connecting with Redis we supply the client 2 endpoints 1 reader endpoint and a primary endpoint.
we recently encountered an issue where while connecting to this cluster the role command to primary failed the first time because of some transient network issue and then even though the network became stable our set and get commands both continued to failed because we had set ReadFromUpstream as a read preference.

Is there any plans to change this behaviour?
Ideally I would like my container to retry figuring out if the master is available even if it wasn't able to do so the first time.

This is easily reproducible too, I setup a Redis with 1 master and 2 replicas via docker and keep the master down during the time my service tries to connect to Redis the first time and post that it stays that way.

My understanding is that the issue comes from the fact that once this masterReplicaConnectionProvider has the knownNodes set the first time and never changes afterwords.

/**
 * Channel writer/dispatcher that dispatches commands based on the intent to different connections.
 *
 * @author Mark Paluch
 */
class MasterReplicaChannelWriter implements RedisChannelWriter {

    private MasterReplicaConnectionProvider<?, ?> masterReplicaConnectionProvider;

[tishun]

Hey @amaan75 ,

Can you give some more context to the issue,I am afraid I do not fully understand it.

... while connecting with Redis we supply the client 2 endpoints 1 reader endpoint and a primary endpoint.

Could you give us a sample of the code that makes the connection?

... while connecting to this cluster the role command to primary failed the first time because of some transient network issue

I assume you are using at-most-once command execution reliability? Does the at-least-once not work for you?

... then even though the network became stable our set and get commands both continued to failed because we had set ReadFromUpstream as a read preference

Can you tell me what means "continued to failed" in this case, was there an exception, are you using Async APIs, etc.

This is easily reproducible too, I setup a Redis with 1 master and 2 replicas via docker and keep the master down during the time my service tries to connect to Redis the first time and post that it stays that way.

Could you create a minimum reproducible example where we can see the exact problem? This would be extremely helpful.

[amaan75]

hi @tishun,
sorry for the delayed response, but i have tried to create a MRE as you suggested
Lettuce-Test

I assume you are using at-most-once command execution reliability? Does the at-least-once not work for you?

I do have auto reconnect enabled, as can be seen in the MRE.

Can you tell me what means "continued to failed" in this case, was there an exception, are you using Async APIs, etc.

This is the exception that i was referring to
org.springframework.data.redis.RedisSystemException: Redis exception; nested exception is io.lettuce.core.RedisException: Cannot determine a node to read (Known nodes: [RedisMasterReplicaNode [redisURI=redis://localhost:6479?clientName=test-app-bug&timeout=3s, role=REPLICA], RedisMasterReplicaNode [redisURI=redis://localhost:6579?clientName=test-app-bug&timeout=3s, role=REPLICA]]) with setting io.lettuce.core.ReadFromImpl$ReadFromUpstream@595ada3d] with root cause io.lettuce.core.RedisException: Cannot determine a node to read (Known nodes: [RedisMasterReplicaNode [redisURI=redis://localhost:6479?clientName=test-app-bug&timeout=3s, role=REPLICA], RedisMasterReplicaNode [redisURI=redis://localhost:6579?clientName=test-app-bug&timeout=3s, role=REPLICA]]) with setting io.lettuce.core.ReadFromImpl$ReadFromUpstream@595ada3d

There is also a possible fix that i can do from my end, which is to ensure that i set the shareNativeConnection to false on spring-data-redis, the downside of that however is that spring will then potentially only fire of 1 request/connection.

Please let me know if you are able to access the MRE, or if you'd like me to add anything else.

[amaan75]

@tishun i would be happy to submit a PR for this, can you guide me to possible fix here?

[tishun]

Hey, apologies for the delay, please let me take a look at the MRE tomorrow and I will let you know what I think.

[amaan75]

@tishun any luck on guiding me towards a fix, or perhaps, is there a newer version that already fixes this?

[tishun]

Hey @amaan75, sorry, no bandwidth for that in the moment.
I will add this issue to the list so - as soon as I have some time - we can take a look at the example you created.