Cache stampede with multiple nodes

[phatcher]

Does anyone have some sample code or ideas that would help with avoiding a cache stampede with multiple nodes?

We have multiple nodes for scalability/resiliency and we call some external services which are expensive and not idempotent and we get charged per request.

What I was thinking might work was using the L2 cache to record who is the master node for a particular cache key - this code is called before the factory and then blocks if the current node is not master, but I don't know how likely/easy this is.

BTW I did a diagram for my team showing the different caches that we have which you might find interesting...

[phatcher]

I think this is basically a distributed lock problem, so then we would acquire a lock on the cache key before invoking the factory, so using DistributedLock.Redis as RedLock.Net doesn't seem to be maintained anymore we get something like this, though the semantics are a bit off..

var int = 42;
var key = $"product:{id}";

/*
This bit hidden behind a IDistributedLockFactory
var connection = await ConnectionMultiplexer.ConnectAsync(connectionString); // uses StackExchange.Redis

// Might need to prefix the passed key with "lock:" to differentiate from our actual key
var @lock = new RedisDistributedLock(key, connection.GetDatabase());
*/

var @lock = distributedLockFactory.Create(key);
await using (var handle = await @lock.TryAcquireAsync()))
{
    if (handle != null)
    {
        cache.GetOrSet<Product>(
            key,
            _ => GetProductFromDb(id),
            TimeSpan.FromSeconds(30)
        );
    }
}

// If we get here, someone else had the lock and should have populated the L2 cache
cache.GetOrSet<Product>(
    key,
    _ => GetProductFromDb(id),
    TimeSpan.FromSeconds(30)
);

So we have the usual critical section double lock problem, ideally we want this after the L2 check but before the call to the factory.

Issues:

• What's the relationship should be between the lock timings and the factory timings and the cache expiry.
• Can we be correct, i.e. guarantee that we never issue more than one request or does the timing of the L2 lookup after the critical section break us

[jodydonetti]

Hi all, it's something in my backlog, see this comment for more details, including why it's not so immediate to do.

Hope this helps.

[phatcher]

Sorry missed that when I did my search, think I have working solution for now