Skip to content

Redis Cluster not recognizing new cluster nodes with identical hostnames #1778

@annervisser

Description

@annervisser

We're in the process of testing MemoryDB and it's failure cases. We've run into an issue when upgrading a cluster while our application is using ioredis.

Environment

  • Server: AWS MemoryDB
  • Cluster: 1 master 2 slaves
  • ioredis: 5.3.1

Problem

When changing the instance type of a MemoryDB cluster, an in-place upgrade happens where new nodes are deployed with the same hostnames as the old ones. The old ones start returning MOVED to all queries, and after a while they are removed.

The way ioredis reacts to this is it ends up in a loop of retrying commands and receiving MOVED. This loop is only broken once the old nodes are removed and the individual connections time out.

Cause

The MOVED responses cause ioredis to refresh it's slots, which ends up resetting the connectionpool with the new slots:

this.connectionPool.reset(nodes);

But because the hostname and port are identical to the old nodes1, no new connections are made and the old ones are left intact. 2

Solution

I believe this could be fixed by including the nodeId (the third return value from CLUSTER SLOTS3) in the node keys, so that a new id will trigger a new connection.

Footnotes

  1. https://github.com/redis/ioredis/blob/d1d9dba9eafc574a9d9041fd4bc7cd04f1584159/lib/cluster/util.ts#L25-L29

  2. https://github.com/redis/ioredis/blob/06782e681500eae6f3ceafcc6385b9be4fdaf4e3/lib/cluster/ConnectionPool.ts#L125-L135

  3. https://redis.io/commands/cluster-slots/

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions