Skip to content

Commit 732fdf1

Browse files
iBotPeachesiBotPeaches
authored
Blog: Redis Cluster with TLS in Laravel (#12)
* feat: wip of redis cluster blog * wip * chore: progression on cluster guide * chore: finish blog * chore: reword
1 parent da1b781 commit 732fdf1

File tree

1 file changed

+384
-0
lines changed
  • blog/2024/2024-08-28-redis-cluster-tls-laravel

1 file changed

+384
-0
lines changed

‎blog/2024/2024-08-28-redis-cluster-tls-laravel/index.mdx‎

Lines changed: 384 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,384 @@
1+
---
2+
slug: 2024/redis-cluster-with-tls-in-laravel
3+
title: Redis Cluster with TLS in Laravel
4+
authors: [justinw, connort]
5+
tags: [laravel, redis, redis-cluster, tls]
6+
---
7+
8+
As a project we built evolved with websockets becoming a far more important feature we wanted to move off of a regular implementation of Redis onto a Redis Cluster to scale for larger usage while giving us the safety of failover and redundancy.
9+
10+
{/* truncate */}
11+
12+
In order to test this we took our `docker-compose.yml` file and set up a few nodes to replicate a cluster that we might have configured in AWS.
13+
14+
```
15+
valkey-cluster-1:
16+
image: bitnami/valkey-cluster:7.2
17+
environment:
18+
- ALLOW_EMPTY_PASSWORD=yes
19+
- VALKEY_CLUSTER_REPLICAS=0
20+
- VALKEY_NODES=valkey-cluster-1 valkey-cluster-2 valkey-cluster-3
21+
valkey-cluster-2:
22+
image: bitnami/valkey-cluster:7.2
23+
environment:
24+
- ALLOW_EMPTY_PASSWORD=yes
25+
- VALKEY_CLUSTER_REPLICAS=0
26+
- VALKEY_NODES=valkey-cluster-1 valkey-cluster-2 valkey-cluster-3
27+
valkey-cluster-3:
28+
image: bitnami/valkey-cluster:7.2
29+
environment:
30+
- ALLOW_EMPTY_PASSWORD=yes
31+
- VALKEY_CLUSTER_REPLICAS=0
32+
- VALKEY_NODES=valkey-cluster-1 valkey-cluster-2 valkey-cluster-3
33+
- VALKEY_CLUSTER_CREATOR=yes
34+
```
35+
36+
:::note
37+
38+
While Valkey is not Redis we wanted to take this chance to explore compatibility with the alternative Redis fork.
39+
40+
:::
41+
42+
After these containers booted up we found a [little test php script](https://gist.github.com/michael-grunder/ec1cd54b321c454d63864091ff288401) that can confirm your PhpRedis is working great.
43+
44+
```text
45+
7ed6ecafd4a4:/code# php cluster-quick-check.php --host sourcetoad_valkey_cluster_1 --port 6379
46+
Checking general cluster INFO: OK
47+
Checking [0:5460] (172.18.0.28:6379): OK
48+
Checking [5461:10922] (172.18.0.29:6379): OK
49+
Checking [10923:16383] (172.18.0.30:6379): OK
50+
Attempting to set key 'phpredis-cluster-key:0'
51+
Success setting 'phpredis-cluster-key:0'
52+
Attempting to set key 'phpredis-cluster-key:1'
53+
Success setting 'phpredis-cluster-key:1'
54+
Attempting to set key 'phpredis-cluster-key:9'
55+
Redirected to '172.18.0.28:6379'
56+
Redirected to '172.18.0.30:6379'
57+
Success setting 'phpredis-cluster-key:9'
58+
Cluster seems OK
59+
7ed6ecafd4a4:/code#
60+
```
61+
62+
Now we had the confidence of a working cluster and re-configured our Laravel installation to point to that cluster. With a single change to our `.env` we refreshed and were met with some crashes.
63+
64+
* `MOVED 15031 172.18.0.30:6379`
65+
66+
Of course that would be expected. We haven't changed anything yet, so off to the Laravel Docs we went to the [Redis Clusters section](https://laravel.com/docs/12.x/redis#clusters). The docs guided you on introducing a `clusters.default` array into your existing `config/database.php` file.
67+
68+
At the time of a base Laravel 12 install. The file would look like this:
69+
70+
```php
71+
/*
72+
|--------------------------------------------------------------------------
73+
| Redis Databases
74+
|--------------------------------------------------------------------------
75+
|
76+
| Redis is an open source, fast, and advanced key-value store that also
77+
| provides a richer body of commands than a typical key-value system
78+
| such as Memcached. You may define your connection settings here.
79+
|
80+
*/
81+
82+
'redis' => [
83+
'client' => env('REDIS_CLIENT', 'phpredis'),
84+
'options' => [
85+
'cluster' => env('REDIS_CLUSTER', 'redis'),
86+
'prefix' => env('REDIS_PREFIX', Str::slug(env('APP_NAME', 'laravel'), '_').'_database_'),
87+
'persistent' => env('REDIS_PERSISTENT', false),
88+
],
89+
'default' => [
90+
'url' => env('REDIS_URL'),
91+
'host' => env('REDIS_HOST', '127.0.0.1'),
92+
'username' => env('REDIS_USERNAME'),
93+
'password' => env('REDIS_PASSWORD'),
94+
'port' => env('REDIS_PORT', '6379'),
95+
'database' => env('REDIS_DB', '0'),
96+
],
97+
'cache' => [
98+
'url' => env('REDIS_URL'),
99+
'host' => env('REDIS_HOST', '127.0.0.1'),
100+
'username' => env('REDIS_USERNAME'),
101+
'password' => env('REDIS_PASSWORD'),
102+
'port' => env('REDIS_PORT', '6379'),
103+
'database' => env('REDIS_CACHE_DB', '1'),
104+
],
105+
],
106+
```
107+
108+
So we worked to add in a new section as described like this:
109+
110+
```php
111+
'clusters' => [
112+
'default' => [
113+
[
114+
'url' => env('REDIS_URL'),
115+
'host' => env('REDIS_HOST', '127.0.0.1'),
116+
'username' => env('REDIS_USERNAME'),
117+
'password' => env('REDIS_PASSWORD'),
118+
'port' => env('REDIS_PORT', '6379'),
119+
'database' => env('REDIS_DB', '0'),
120+
],
121+
]
122+
]
123+
```
124+
125+
Very quickly we learned this level of configuration was not for us due to the way Laravel operates. If we look at the pseudocode of how Laravel loads a Redis connection we'd see this:
126+
127+
```php
128+
$name = $name ?: 'default';
129+
$options = $this->config['options'] ?? [];
130+
131+
if (isset($this->config[$name])) {
132+
return $this->resolveConnection($this->config[$name]);
133+
}
134+
135+
if (isset($this->config['clusters'][$name])) {
136+
return $this->resolveCluster($name);
137+
}
138+
139+
// https://github.com/laravel/framework/blob/12.x/src/Illuminate/Redis/RedisManager.php#L104
140+
```
141+
142+
Since we had a `redis.default` block as well as a `redis.clusters.default` block our non-cluster connection was always loaded. The code in this resolve method has not changed in 7 years, but we are thinking that perhaps the loading of the cluster should go ahead of the normal connection. That would mean once you add the optional `clusters` block with connection name - it would win loading if both a cluster and non-cluster connection had the same name.
143+
144+
However, that is also not preferred as that change would mean the instant a code change introduced a `redis.clusters.default` block - it would win. This might explain why this code hasn't changed in almost a decade.
145+
146+
So we rewrote our configuration block slightly to this:
147+
148+
```php
149+
'clusters' => [
150+
'aws' => [
151+
[
152+
'url' => env('REDIS_CLUSTER_URL'),
153+
'host' => env('REDIS_CLUSTER_HOST', '127.0.0.1'),
154+
'username' => env('REDIS_CLUSTER_USERNAME'),
155+
'password' => env('REDIS_CLUSTER_PASSWORD'),
156+
'port' => env('REDIS_CLUSTER_PORT', '6379'),
157+
'database' => env('REDIS_CLUSTER_DB', '0'),
158+
],
159+
],
160+
],
161+
```
162+
163+
This gave us 2 major advantages:
164+
165+
1. We could deploy this change without configuring the cluster with no change.
166+
2. We could leverage a different env for cluster and non-cluster in case we had to revert quickly.
167+
168+
The downside to this is our connection was no longer named `default`. So in order to switch to our cluster connection, we invoked the connection name of `aws` which was shorthand for our ElastiCache Valkey instance in AWS.
169+
170+
Our `.env` to swap to clusters then was roughly:
171+
172+
```text
173+
QUEUE_CONNECTION=redis
174+
175+
SESSION_DRIVER=redis
176+
SESSION_CONNECTION=aws
177+
178+
CACHE_STORE=redis
179+
CACHE_PREFIX=alpha_
180+
181+
REDIS_CLUSTER_HOST=clustercfg.project-redis-cluster.xxxxxx.use1.cache.amazonaws.com
182+
REDIS_CLUSTER_PORT=6379
183+
REDIS_PERSISTENCE=true
184+
REDIS_PREFIX=alpha_
185+
REDIS_CACHE_CONNECTION=aws
186+
REDIS_CACHE_LOCK_CONNECTION=aws
187+
REDIS_QUEUE_CONNECTION=aws
188+
```
189+
190+
So lets break this down:
191+
192+
* Anything `_DRIVER`, `_STORE` or `_CONNECTION` is simply pointing that feature of Laravel to our new Redis Cluster `aws` connection.
193+
* Anything `REDIS_CLUSTER_` is for configuration of our Redis Cluster.
194+
* Anything `_PREFIX` is because Redis Clusters does NOT support multiple databases. So we prefix items to prevent collisions on a shared server.
195+
* `REDIS_PERSISTENCE` keeps Laravel using the same connection instead of opening a connection on each Redis usage.
196+
197+
With all of that we booted up our system and clicked around. We had a few crashes that became apparent when utilizing the queues. These errors were:
198+
199+
* `Cannot use 'DEL' with redis-cluster`
200+
* `Cannot use 'EVAL' with redis-cluster`
201+
202+
So before taking to Google we first checked the [Queue documentation](https://laravel.com/docs/12.x/queues#redis) on Laravel and found a call-out.
203+
204+
:::warning
205+
206+
If your Redis queue connection uses a Redis Cluster, your queue names must contain a key hash tag. This is required in order to ensure all the Redis keys for a given queue are placed into the same hash slot:
207+
208+
```php
209+
'redis' => [
210+
'queue' => env('REDIS_QUEUE', '{default}'),
211+
],
212+
```
213+
214+
:::
215+
216+
So we understood the problem, but with our complex queue system and multiple lower environments on the same cluster we had to get creative to implement this properly. We introduced a new custom `QueueServiceProvider`
217+
218+
```php
219+
<?php
220+
declare(strict_types=1);
221+
222+
namespace App\Support\RedisCluster;
223+
224+
class QueueServiceProvider extends \Illuminate\Queue\QueueServiceProvider
225+
{
226+
protected function registerRedisConnector($manager): void
227+
{
228+
$manager->addConnector('redis', function () {
229+
// @phpstan-ignore-next-line
230+
return new RedisClusterConnector($this->app['redis']);
231+
});
232+
}
233+
}
234+
```
235+
236+
This loaded our custom `RedisClusterConnector`, which was basically a shell to our key override the queue class.
237+
238+
```php
239+
<?php
240+
declare(strict_types=1);
241+
242+
namespace App\Support\RedisCluster;
243+
244+
use Illuminate\Queue\Connectors\RedisConnector;
245+
246+
class RedisClusterConnector extends RedisConnector
247+
{
248+
public function connect(array $config): RedisClusterQueue
249+
{
250+
return new RedisClusterQueue(
251+
redis: $this->redis,
252+
default: $config['queue'],
253+
connection: $config['connection'] ?? $this->connection,
254+
retryAfter: $config['retry_after'] ?? 60,
255+
blockFor: $config['block_for'] ?? null,
256+
dispatchAfterCommit: $config['after_commit'] ?? null,
257+
migrationBatchSize: $config['migration_batch_size'] ?? -1
258+
);
259+
}
260+
}
261+
```
262+
263+
Now our custom `RedisClusterQueue` could be loaded.
264+
265+
```php
266+
<?php
267+
declare(strict_types=1);
268+
269+
namespace App\Support\RedisCluster;
270+
271+
use Illuminate\Queue\RedisQueue;
272+
use Illuminate\Support\Facades\App;
273+
274+
class RedisClusterQueue extends RedisQueue
275+
{
276+
public function getQueue($queue): string
277+
{
278+
$isCluster = config('queue.connections.redis.connection') === 'aws';
279+
280+
if ($isCluster) {
281+
$queueName = ($queue ?: $this->default);
282+
return sprintf('{queues:%s}', $queueName);
283+
}
284+
285+
return parent::getQueue($queue);
286+
}
287+
}
288+
```
289+
290+
All we had to do was remove the stock loader and load our own.
291+
292+
```php
293+
// Illuminate\Queue\QueueServiceProvider::class,
294+
App\Support\RedisCluster\QueueServiceProvider::class,
295+
```
296+
297+
This code would automatically build a key hash tag based on the queue name. Sure enough with successful queue tests we found the `keys *` command breaking down our naming pattern.
298+
299+
```text
300+
127.0.0.1:6379> keys *
301+
1) "local_{queues:default}:notify"
302+
2) "local_{queues:default}"
303+
3) "local_{queues:openai}"
304+
4) "local_{queues:openai}:notify"
305+
5) "local_{queues:openai-moderation}"
306+
6) "local_{queues:openai-moderation}:notify"
307+
127.0.0.1:6379>
308+
```
309+
310+
:::info
311+
312+
The important aspect of a key hash tag is to ensure all keys for a given queue are placed into the same hash slot.
313+
314+
:::
315+
316+
This was working great and allowed us to leverage a cluster connection without having to remember to change any specific value. Any usage of our queue system would automatically produce a key that was safe for cluster usage. Now we were ready to ramp it up with TLS support. At this point with our confidence working with a local cluster we were ready to move to AWS and test our cluster with TLS.
317+
318+
We spun up an ElastiCache instance and made some configurations:
319+
320+
* Multi-AZ: Enabled
321+
* Encrypted at Rest: Enabled
322+
* Encrypted in Transit: Enabled
323+
* Transit Encryption Mode: Required
324+
325+
These tests didn't work too well with errors like:
326+
327+
* "Can't communicate with any node in the cluster"
328+
* "Couldn't map cluster keyspace using any provided seed"
329+
330+
This took us a bit, but it seemed PhpRedis was not configured to use TLS. We found digging through [PhpRedis documentation](https://github.com/phpredis/phpredis/blob/develop/cluster.md#declaring-a-cluster-with-an-array-of-seeds) that we needed to set `verify_peer` in our config. So we just needed to know how to pass a value to the `new RedisCluster` execution.
331+
332+
Fortunately, we can peek [Laravel sourcecode](https://github.com/laravel/framework/blob/12.x/src/Illuminate/Redis/Connectors/PhpRedisConnector.php#L199-L203) to see how it builds the RedisCluster, which was via the `context` array element.
333+
334+
```php {8-10}
335+
'redis' => [
336+
'client' => env('REDIS_CLIENT', 'phpredis'),
337+
'options' => [
338+
'cluster' => env('REDIS_CLUSTER', 'redis'),
339+
'prefix' => env('REDIS_PREFIX', Str::slug(env('APP_NAME', 'laravel'), '_').'_database_'),
340+
'persistent' => env('REDIS_PERSISTENCE', true),
341+
'timeout' => env('REDIS_TIMEOUT', 5),
342+
'context' => [
343+
'verify_peer' => env('REDIS_SSL_VERIFY_PEER', false),
344+
],
345+
],
346+
],
347+
```
348+
So we added a new `context` block to our `config/database.php` file including `verify_peer`.
349+
350+
We had a successful Redis Cluster connection with TLS! Our application was working great with a Redis Cluster communicating over TLS. However, for local usage this change forced TLS which was not preferred. We had to get creative reading the connection name and array merging the context block conditionally.
351+
352+
To recap our journey:
353+
354+
* We set up our `docker-compose.yml` to run a Redis Cluster.
355+
* We configured our `.env` to point to the cluster.
356+
* We added a `clusters` block to our `config/database.php` file using a non-default name of `aws`.
357+
* We created a custom `QueueServiceProvider` to override the default Redis connection to use a key hash tag.
358+
* We changed the `context` block to include `verify_peer` to allow TLS connections on the base Redis options.
359+
360+
Now we tested out a few fault scenarios with failover to ensure our cluster was working as expected. As long as our timeout was set to a reasonable value (5 seconds) the node failover worked as expected and recovered.
361+
362+
---
363+
364+
#### Common Errors
365+
366+
##### `Cannot use 'EVAL' with redis-cluster`
367+
* You are missing a key hash tag `{example}` in your Redis key.
368+
369+
##### `MOVED 15031 172.18.0.30:6379`
370+
371+
* You are sending cluster commands, but your connection (PhpRedis) is not in cluster mode.
372+
* Ensure your `.env` / `config/database.php` is configured properly.
373+
374+
##### `Can't communicate with any node in the cluster`
375+
376+
* Your cluster server is unreachable (or requiring SSL) and you aren't providing it.
377+
378+
##### `Couldn't map cluster keyspace using any provided seed`
379+
380+
* Your cluster server is unreachable, generally because its requiring/preferring TLS and you aren't sending it.
381+
382+
##### Laravel Horizon won't work with a Cluster
383+
384+
* As of April 28, 2025 - Horizon [does not officially support](https://github.com/laravel/horizon/issues/274#issuecomment-457218217) Redis Clusters.

0 commit comments

Comments
 (0)