Handling MOVED with Redis Sentinel

[jcustenborder]

Hello there!

I'm trying to add support for Sentinel for one of my open source projects. I know I'm doing something wrong but I'm getting a RedisCommandExecutionException every time I try to write. My project basically bulk writes data to Redis. What am I missing? Thank you in advance!

I used this as my guide

First attempt

RedisURI redisUri = RedisURI.Builder.sentinel("127.0.0.1", 48000)
    .withSentinel("127.0.0.1", 48001)
    .withSentinel("127.0.0.1", 48002)
    .withSentinelMasterId("sentinel50000")
    .build();

RedisClient client = RedisClient.create(redisUri);
StatefulRedisConnection<String, String> connection = client.connect();
RedisAsyncCommands<String, String> commands = connection.async();
RedisFuture<String> future = commands.set("test", "value");
future.get();

Second Attempt

    RedisURI redisUri = RedisURI.Builder.sentinel("127.0.0.1", 48000)
        .withSentinel("127.0.0.1", 48001)
        .withSentinel("127.0.0.1", 48002)
        .withSentinelMasterId("sentinel50000")
        .build();

    RedisClient client = RedisClient.create();
    client.setOptions(
        ClusterClientOptions.builder()
            .topologyRefreshOptions(
                ClusterTopologyRefreshOptions.builder()
                    .enableAllAdaptiveRefreshTriggers()
                    .build()
            )
            .build()
    );

    StatefulRedisMasterReplicaConnection<String, String> connection = MasterReplica.connect(client, new StringCodec(), redisUri);
    connection.setReadFrom(ReadFrom.MASTER);
    RedisAsyncCommands<String, String> commands = connection.async();
    RedisFuture<String> future = commands.set("test", "value");
    future.get();
  }

Exception thrown.

io.lettuce.core.RedisCommandExecutionException: MOVED 6918 127.0.0.1:50001
	at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:147)
	at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:116)
	at io.lettuce.core.protocol.AsyncCommand.completeResult(AsyncCommand.java:120)
	at io.lettuce.core.protocol.AsyncCommand.complete(AsyncCommand.java:111)
	at io.lettuce.core.protocol.CommandHandler.complete(CommandHandler.java:747)
	at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:682)
	at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:599)

logs.txt

[mp911de]

MOVED replies are only used with Redis Cluster. Your cluster nodes report also that they are using Cluster mode:

$4
mode
$7
cluster

Please use RedisClusterClient as the use of Redis Sentinel with Cluster doesn't make any sense.

[jcustenborder]

Thank you! Ok that makes sense. I'm not familiar with Sentinel or Cluster. Basically you're saying that it's either Cluster or Sentinel but not both at the same time. The container I am using for testing generated a config that is not realistic in the real world. I'm the maintainer of kafka-connect-redis and was trying to get integration tests working with Sentinel. Can you recommend a local docker environment to test against? Thank you so much for your response. It's greatly appreciated.

For sake of completeness this is the container I was using. It has support for cluster which seems to work fine.

version: "3"
services:
  redis:
    image: grokzen/redis-cluster:6.0.16
    environment:
      IP: "0.0.0.0"
      INITIAL_PORT: "50000"
      SENTINEL: "true"
    ports:
     - "48000:48000"
     - "48001:48001"
     - "48002:48002"
     - "50000:50000"
     - "50001:50001"
     - "50002:50002"
     - "50003:50003"
     - "50004:50004"
     - "50005:50005"

It generated these configs

sentinel configs

::::::::::::::
sentinel-50000.conf
::::::::::::::
port 48000
sentinel myid 0c54e269a4ee8b6255a45787cdfa3df5210f3ac0
sentinel deny-scripts-reconfig yes
sentinel monitor sentinel50000 127.0.0.1 50000 2
sentinel down-after-milliseconds sentinel50000 5000
# Generated by CONFIG REWRITE
user default on nopass ~* +@all
dir "/data"
sentinel failover-timeout sentinel50000 60000
sentinel config-epoch sentinel50000 0
sentinel leader-epoch sentinel50000 0
sentinel known-replica sentinel50000 127.0.0.1 50004
sentinel current-epoch 0
::::::::::::::
sentinel-50001.conf
::::::::::::::
port 48001
sentinel myid d2d279944768e8cd21102cd445c6932fe4303d87
sentinel deny-scripts-reconfig yes
sentinel monitor sentinel50001 127.0.0.1 50001 2
sentinel down-after-milliseconds sentinel50001 5000
# Generated by CONFIG REWRITE
user default on nopass ~* +@all
dir "/data"
sentinel failover-timeout sentinel50001 60000
sentinel config-epoch sentinel50001 0
sentinel leader-epoch sentinel50001 0
sentinel known-replica sentinel50001 127.0.0.1 50005
sentinel current-epoch 0
::::::::::::::
sentinel-50002.conf
::::::::::::::
port 48002
sentinel myid 733c9bd1fe6f2e92869f5f0bb90e7dfde5ce9c64
sentinel deny-scripts-reconfig yes
sentinel monitor sentinel50002 127.0.0.1 50002 2
sentinel down-after-milliseconds sentinel50002 5000
# Generated by CONFIG REWRITE
user default on nopass ~* +@all
dir "/data"
sentinel failover-timeout sentinel50002 60000
sentinel config-epoch sentinel50002 0
sentinel leader-epoch sentinel50002 0
sentinel known-replica sentinel50002 127.0.0.1 50003
sentinel current-epoch 0

Redis configs

::::::::::::::
50000/redis.conf
::::::::::::::
bind 0.0.0.0
port 50000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
dir /redis-data/50000
::::::::::::::
50001/redis.conf
::::::::::::::
bind 0.0.0.0
port 50001
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
dir /redis-data/50001
::::::::::::::
50002/redis.conf
::::::::::::::
bind 0.0.0.0
port 50002
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
dir /redis-data/50002
::::::::::::::
50003/redis.conf
::::::::::::::
bind 0.0.0.0
port 50003
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
dir /redis-data/50003
::::::::::::::
50004/redis.conf
::::::::::::::
bind 0.0.0.0
port 50004
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
dir /redis-data/50004
::::::::::::::
50005/redis.conf
::::::::::::::
bind 0.0.0.0
port 50005
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
dir /redis-data/50005

[mp911de]

I'm not familiar with Redis docker images, in Lettuce I use a Makefile that spins up various setups tailored to our testing needs.

[jcustenborder]

Awesome! I’ll look there! Thank you!

…

On Thu, Jan 26, 2023 at 1:47 AM Mark Paluch ***@***.***> wrote: I'm not familiar with Redis docker images, in Lettuce I use a Makefile that spins up various setups tailored to our testing needs. — Reply to this email directly, view it on GitHub <#2304 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABTHAC6DVNPS5I75DEKDH3WUITX5ANCNFSM6AAAAAAUFK2J6U> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>