Add redis cluster config (#5799)

# Support Redis Cluster
We are using a redis cluster setup, but the current code uses
redis.from_url which only works for Redis Standalone. See:
https://github.com/redis/redis-py/blob/ceb12fe8b52f488bd1e0a42b9dc53161e77514d0/redis/utils.py#L33

So I added a config-option to support Redis Cluster which should not
break the current default behavior only extend it.

**Type of change**
- New feature (non-breaking change which adds functionality)
- Documentation update

**How Has This Been Tested**
<!-- Please add some reference about how your feature has been tested.
-->
AWS Redis cluster deployment

**Checklist**
- I added relevant documentation
- I followed the style guidelines of this project
- I did a self-review of my code
- I made corresponding changes to the documentation
- I confirm My changes generate no new warnings
- I have added tests that prove my fix is effective or that my feature
works
- TODO - I have added relevant notes to the CHANGELOG.md file (See
https://keepachangelog.com/)

---------

Co-authored-by: David Berenstein <[email protected]>

Author: paulbauriegel

argilla-server/CHANGELOG.md

@@ -16,6 +16,10 @@ These are the section headers that we use:
 
 ## [Unreleased]()
 
+## Added
+
+- Add configuration switch to use Redis cluster vs Redis standalone via ARGILLA_REDIS_USE_CLUSTER ([#5799](https://github.com/argilla-io/argilla/pull/5799))
+
 ### Fixed
 
 - Fixed error when computing user progress with PostgreSQL database. ([#5795](https://github.com/argilla-io/argilla/pull/5795))

argilla-server/src/argilla_server/jobs/queues.py

@@ -13,12 +13,16 @@
 #  limitations under the License.
 
 import redis
+from redis.cluster import RedisCluster
 
 from rq import Queue
 
 from argilla_server.settings import settings
 
-REDIS_CONNECTION = redis.from_url(settings.redis_url)
+if settings.redis_use_cluster:
+    REDIS_CONNECTION = RedisCluster.from_url(settings.redis_url)
+else:
+    REDIS_CONNECTION = redis.from_url(settings.redis_url)
 
 DEFAULT_QUEUE = Queue("default", connection=REDIS_CONNECTION)
 HIGH_QUEUE = Queue("high", connection=REDIS_CONNECTION)

argilla-server/src/argilla_server/settings.py

@@ -114,6 +114,7 @@ class Settings(BaseSettings):
     cors_origins: List[str] = ["*"]
 
     redis_url: str = "redis://localhost:6379/0"
+    redis_use_cluster: bool = False
 
     docs_enabled: bool = True
 

argilla/docs/reference/argilla-server/configuration.md

@@ -96,6 +96,7 @@ The following environment variables are useful only when PostgreSQL is used:
 Redis is used by Argilla to store information about jobs to be processed on background. The following environment variables are useful to config how Argilla connects to Redis:
 
 - `ARGILLA_REDIS_URL`: A URL string that contains the necessary information to connect to a Redis instance (Default: `redis://localhost:6379/0`).
+- `ARGILLA_REDIS_USE_CLUSTER`: If "True" tries the connection with the URL to a  Redis Cluster instead of a Redis Standalone instance.
 
 ### Datasets
 

examples/deployments/k8s/argilla-chart/templates/deployment.yaml

@@ -37,7 +37,9 @@ spec:
             - name: ARGILLA_AUTH_SECRET_KEY
               value: {{ .Values.argilla.authSecretKey | quote }}
             - name: ARGILLA_REDIS_URL
-              value: redis://{{ .Release.Name }}-redis-master:6379/0
+              value: {{if .Values.externalRedis.enabled}}"{{ .Values.externalRedis.url }}"{{else}}"redis://{{ .Release.Name }}-redis-master:6379/0"{{end}}
+            - name: ARGILLA_REDIS_USE_CLUSTER
+              value: {{ if and .Values.externalRedis.enabled .Values.externalRedis.is_redis_cluster }} "True" {{ else }} "False" {{ end }}
             - name: USERNAME
               value: {{ .Values.argilla.auth.username | quote }}
             - name: PASSWORD

examples/deployments/k8s/argilla-chart/values.yaml

@@ -80,6 +80,12 @@ redis:
         limits:
           cpu: 1
           memory: 2Gi
+
+externalRedis:
+  enabled: false
+  url: "redis://localhost:6379/0"
+  is_redis_cluster: false
+
 serviceAccount:
   create: false