-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Description
We're in the process of testing MemoryDB and it's failure cases. We've run into an issue when upgrading a cluster while our application is using ioredis.
Environment
- Server: AWS MemoryDB
- Cluster: 1 master 2 slaves
- ioredis: 5.3.1
Problem
When changing the instance type of a MemoryDB cluster, an in-place upgrade happens where new nodes are deployed with the same hostnames as the old ones. The old ones start returning MOVED
to all queries, and after a while they are removed.
The way ioredis reacts to this is it ends up in a loop of retrying commands and receiving MOVED
. This loop is only broken once the old nodes are removed and the individual connections time out.
Cause
The MOVED
responses cause ioredis to refresh it's slots, which ends up resetting the connectionpool with the new slots:
Line 902 in 96a9f41
this.connectionPool.reset(nodes); |
But because the hostname and port are identical to the old nodes1, no new connections are made and the old ones are left intact. 2
Solution
I believe this could be fixed by including the nodeId (the third return value from CLUSTER SLOTS
3) in the node keys, so that a new id will trigger a new connection.