In redis cluster, lettuce reconnects to different port

[rethab]

Hi,

looking at the logs in our application, I see these messages:

Reconnecting, last destination was my-redis-host.redis.cache.windows.net/10.1.2.3:15004
Reconnected to my-redis-host.redis.cache.windows.net/<unresolved>:15002
Reconnecting, last destination was my-redis-host.redis.cache.windows.net/10.1.2.3:15002
Reconnected to my-redis-host.redis.cache.windows.net/<unresolved>:15005
Reconnecting, last destination was my-redis-host.redis.cache.windows.net/10.1.2.3:15005

Lettuce decides that the connection pointing to 15004 needs to be reconnected. But then it connects to 15002. One minute later, it decides to reconnect 15002 and connects to 15005.

This continues forever and I don't know what's happening. Note that in the meantime the application works fine, I only find these logs confusing :)

Please note that:

• we are using redis cluster with periodic refresh enabled. The period refresh is set to 60 seconds (default unchanged)
• we did not configure any keepalive options
• we are using azure redis
• we are using lettuce 6.2.6.RELEASE

I read in anther issue that these constant reconnects could point to an infrastructure issue. Perhaps there is something, but that's another problem ;-).

My question is why does the port change? I am confident that there is no cluster topology change going on, so lettuce saying it reconnects to a different port seems odd to me.

Secondly, these messages appear in 1-minute intervals. As I said the periodic topology refresh is set to 60s, but if this was what is happening, why does it only reconnect one connection? We have five shards, so I would expect it to do the same for all five.

[TerjeDahlstrom]

Redis has a configuration timeout in redis.conf. If that is set to 60, idle connections will be disconnected after 60 seconds, and lettuce will react to this by reconnecting. This might be what you are seeing.
Like any configuration in redis, timeout can also be set as a command-line argument to run-redis
Regarding the fact that a new connection does not go to the same cluster node compared to what node was disconnected;
Reconnects do not know why they happen. So a reconnect goes to the node with the fewest current connections. In light of this; what you are seeing is not surprising. For details, look at RedisClusterClient#getSocketAddressSupplier

[patrykwierzbilo]

hi @rethab , were you able to solve this issue? i'm seeing the same thing. in my case the reconnect interval is 10 minutes because azure cache for redis is set up that way to drop idle connection after 10 minutes. any workaround or fix would help. thanks.

[tishun]

Folks, see #3441 and more specifically the solution in #2078