Redis "tagged cache" key-reference lists grow indefinitely, consuming available keyspace slowly until dead

[zanderwar]

This seems like an oversight, but wanted to discuss it here before raising an issue regarding the illuminate/redis package...

• Using illuminate/redis:9.52.0
• Using Redis 6 (on Redis Enterprise)

Description:

Hello,

The SET keys created by illuminate/redis grow indefinitely, never expire, and eat away at keyspace until a flush is needed.

The graphs below show the past month where memory usage has remained the same, but the keys that can fit into memory continue to decline, because these "sets" within them continue to grow

We have eviction policy allkeys-lfu which means we are more likely to be left with these sets, than we are to be left with the cache they are meant to reference.

We cache based on filter/sort query parameters present in the URL (sorted by key) as an example:

modio:fb5e7d6dbecb5e0a6bdc17042aef511f866dd0a7:?x-platform=ps5&_limit=9&_offset=18&_sort=downloads&game_id=306&tags-in%5b0%5d=4x4&tags-not-in%5b0%5d=manual{3}

[00.07%] Biggest set found so far '"modio:63d3a7b87e6d6643428850:standard_ref{0}"' with 1228 members
[00.13%] Biggest set found so far '"modio:63e84f9d9f10c924068019:standard_ref{0}"' with 1286 members
[00.61%] Biggest set found so far '"modio:63f278e0357b6997490067:standard_ref{0}"' with 4809 members

 > MEMORY USAGE modio:63f278e0357b6997490067:standard_ref{0}
(integer) 1000597 // 1MB~

 > TTL modio:63f278e0357b6997490067:standard_ref{0}
(integer) -1

We have so many thousands of these as we have millions of users where cache relevant to any particular individual needs to be grouped with them.

A vast majority of the keys within this set however, have expired and don't even exist anymore, they will unfortunately stay there forever.

Steps To Reproduce:

for ($i = 0; $i < 10000; $i++) {
    $cacheKey = 'my-dynamic-cache-key-'.substr(md5(microtime(true) . rand(10000,50000)), 0, 4);

    Cache::tags('my-tagged-cache')->put($cacheKey, 'Test', ttl: 1);
}

1. Flush your test redis instance with FLUSHALL or FLUSHDB
2. Run the script above, simulating the possible permutations one may have for sorting and filtering and the caching of those responses

After waiting for the 1 second TTL to expire:

1. In redis-cli use KEYS *
2. In redis-cli use SCARD <the-standard-ref-key-you-found-in-the-above // should see 10,000
3. In redis-cli use MEMORY USAGE <the-same-key> // should see the memory wasted to hold 10,000 expired keys

Questions

1. Why is there no cleanup of these lists?
2. How do I clean them up
3. If needed, how can I change my approach to not have this problem

[henzeb]

This issue seems partially fixed in 10.0.0.

Could you try it out with laravel 10

[henzeb]

Have you tried it?

[zanderwar]

Have you tried it?

Yes, still an issue, all keys within the SET have expired.

Repro repo: https://github.com/zanderwar/illuminate-redis-congestion

[zanderwar]

I'm not sure what you would mean by partially fixed though sorry, could you please point me towards commits that would imply anything was done to prevent or manage this?

[henzeb]

I usually don't see the point of sharing a link to code changes if you can easily spin up a new instance with laravel 10 en run your code in there to see if it is any better. But sure, here's the pull request that should have enhanced the redis tagged cache behavior.

You could try the new added command to see if that purges stale tags.

#45690

[zanderwar]

Thank you for that, looks like Laravel 10 will resolve this. It would seem the best approach would be to ->flushStale() immediately after adding something to tagged cache

[JimmyBastos]

I successfully extended Laravel's default cache driver to support tag expiration by creating custom TagSet.

This implementation addresses a critical incident in a high-volume transaction application where the default behavior of Laravel’s cache tags, without automatic tag expiration, led to a production outage. The system encountered failures due to excessively growing cache tag keys, which necessitated implementing tag expiration to mitigate future risks.

TaggedCacheProvider

<?php

namespace App\Providers;

use App\Cache\RedisStore;
use Illuminate\Foundation\Application;
use Illuminate\Support\Facades\Cache;
use Illuminate\Support\ServiceProvider;

class TaggedCacheProvider extends ServiceProvider
{
    /**
     * Register services.
     *
     * @return void
     */
    public function register()
    {
        $this->app->booting(function () {
            Cache::extend('redis', function (Application $app, $config) {
                return Cache::repository(
                    new RedisStore(
                        $app['redis'],
                        $config['prefix'] ?? '',
                        $config['connection'] ?? null
                    )
                );
            });
        });
    }

    /**
     * Bootstrap services.
     *
     * @return void
     */
    public function boot()
    {
        // No additional boot logic required.
    }
}

RedisStore

<?php

namespace App\Cache;

use Illuminate\Cache\RedisStore as BaseRedisStore;
use Illuminate\Cache\RedisTaggedCache;
use Illuminate\Contracts\Redis\Factory as Redis;

class RedisStore extends BaseRedisStore
{
    public function __construct(Redis $redis, $prefix = '', $connection = 'default')
    {
        parent::__construct($redis, $prefix, $connection);
    }

    public function tags($names)
    {
        return new RedisTaggedCache(
            $this,
            new RedisTagSet($this, is_array($names) ? $names : func_get_args())
        );
    }
}

RedisTagSet

<?php

namespace App\Cache;

use Illuminate\Cache\RedisTagSet as BaseRedisTagSet;

class RedisTagSet extends BaseRedisTagSet
{
    public function addEntry(string $key, ?int $ttl = 0, $updateWhen = null): void
    {
        parent::addEntry($key, $ttl, $updateWhen);

        // Calculate tag TTL as original TTL plus 24 hours (86400 seconds).
        $tagsTtl = $ttl + 86400;

        foreach ($this->tagIds() as $tagKey) {
            $this->store->connection()->expire(
                $this->store->getPrefix() . $tagKey,
                $tagsTtl
            );
        }
    }
}

Explanation

In TaggedCacheProvider, we extend the 'redis' cache driver to use RedisStore, which supports tagging with custom expiration. The RedisStore class extends BaseRedisStore and introduces tagging through RedisTaggedCache, while RedisTagSet manages tag keys and their expiration by adding a fixed 24-hour TTL (86400 seconds) to each tag key after setting an entry.

This approach enhances Laravel's caching capabilities, ensuring efficient management of tagged cache expiration.