Batch support for GetOrSet

[rzepinskip]

Hi,

I am curious whether you have explicitly decided not to support batch methods (if so what are the reasons - implementation burden / other patterns to use / no need etc.) or is it planned for the future?

BTW. Neat project ;)

[jodydonetti]

Hi,

I am curious whether you have explicitly decided not to support batch methods (if so what are the reasons - implementation burden / other patterns to use / no need etc.) or is it planned for the future?

Good question!

The project was born as an abstraction layer on top of the standard IMemoryCache and IDistributedCache interfaces, so the operations available are the ones available on those interfaces + the ones I felt added value (like the atomic GetOrSet, the GetOrDefault, etc) that - while not available there - maintained the same access pattern (eg: one entry at a time).

Do you already have an idea about how those methods would be modeled?
Like I can imagine instead of a single string key it would accept an IEnumerable<string> keys, but what would be the shape of the factory? And what would the method return, a Dictionary<string,TValue> or something else? Also we should think about a factory failing for a single entry while for the others succeed (in case the impl is a series of single GetOrDefault because the underlying cache can't handle multi-get), how should we handle that?
If you have any suggestion in this regard I'd be happy to know.

As of today the need for multi-key methods did not came up, but I'm open to explore that and put it in my backlog, which by now contains adding support for events #14 , a plugin system #15 (which combined will hopefully lead, with the kind help of @JoeShook , to the support of metrics #9 with multiple providers) and a distributed backplane #11 .

BTW. Neat project ;)

Thanks 😁 !

Any suggestion about any aspect (docs, etc) is more than welcome.

[rzepinskip]

It is mainly a performance improvement when underlying service (endpoint) supports batch requests. GetOrSet and factory method could follow the same interface: accept IReadOnlyCollection<string> and return IReadOnlyDictionary<string , TValue>. Failure for a single entry when cache does not support batch request may be signaled by AggregateException but using such cache for batch requests probably should be discouraged.

[Pikabanga]

I am also interested in this feature. At my previous job I built a cache with first class support for "IEnumerable" caches.

This enabled us to load all entities (either lazily or eagerly), which we then would use for full text search, to feed dropdowns with incremental loading, etc.

It would also be useful if this concept supported adding, updating and deleting single entities from the list.

[percramer]

I was searching for exactly the same thing. Feed dropdowns with data that is used a lot. I have got multiple dropdowns which show a list of relations and it would be nice if those would come from the cache instead of the database each time

[Place1]

I'll also add that this feature would be very useful to reduce round-trip times to redis when fetching batches of keys using MGET.

[jodydonetti]

Hi @Place1 , as stated:

The project was born as an abstraction layer on top of the standard IMemoryCacheand IDistributedCache interfaces, so the operations available are the ones available on those interfaces + the ones I felt added value (like the atomic GetOrSet, the GetOrDefault, etc) that - while not available there - maintained the same access pattern (eg: one entry at a time).

So basically I'm "limited by technology of our time" 😅

We also recently talked about multi get/set in the issue about the new caching abstractions in .net , I suggest to take a look at that since there are a lot of useful informations.

Hope this helps.

[onionhammer]

I'm also interested in the same thing; heres an example scenario:

    /// <summary>
    /// Get a product by its ID
    /// </summary>
    ValueTask<Product?> GetByIdAsync(Guid id, CancellationToken cancellationToken);

    /// <summary>
    /// Get products by their IDs
    /// </summary>
    Task<IReadOnlyList<Product>> GetByIdsAsync(IList<Guid> ids, CancellationToken cancellationToken);

Ideally, each item returned from GetByIds is also cached/retrievable for GetById, but the result of GetByIds shouldn't really be cached.

Here's an implementation

        // Attempt to get all products from cache
        List<Product> products = [];
        List<Guid> productsToLookup = [];

        foreach (var id in ids)
        {
            var product = await cache.TryGetAsync<Product>($"products:{id}", token: cancellationToken);

            if (product.HasValue)
                products.Add(product.Value);
            else
                productsToLookup.Add(id);
        }

        // Get the remaining products from the database
        if (productsToLookup.Count > 0)
        {
            var productsFromDb = await dbContext.Products
                .Where(p => productsToLookup.Contains(p.Id))
                .Include(p => p.Partner)
                .ToListAsync(cancellationToken);

            products.AddRange(productsFromDb);

            // Cache the products
            foreach (var product in productsFromDb)
                await cache.SetAsync($"products:{product.Id}", product, TimeSpan.FromHours(1), cancellationToken);
        }

        return products;

Imagine if it was

        var products = await cache.GetOrSetManyAsync(
            keyFactory: id => $"products:{id}",
            valueFactory: async (ids, ct) =>
            {
                var products = await dbContext.Products
                    .Where(p => ids.Contains(p.Id))
                    .Include(p => p.Partner)
                    .ToDictionaryAsync(p => p.Id, ct);

                return products;
            });

where the valueFactory was only called for ids which weren't in the cache, and potentially there's room for optimizing caching operations by storing multiple new items in the cache at once

[ZUOXIANGE]

Here's my simple implementation

  public static async ValueTask<Dictionary<string, T>> GetOrSetManyAsync<T>(this IFusionCache cache, IEnumerable<string> keys,
      TimeSpan duration, Func<IEnumerable<string>, Task<Dictionary<string, T>>> getDataAsync, CancellationToken cancellationToken = new())
  {
      keys = keys.Distinct().ToList();
      var dic = new Dictionary<string, T>();
      List<string> keysToLookup = [];

      foreach (var key in keys)
      {
          //MGET?
          var product = await cache.TryGetAsync<T>(key, token: cancellationToken);

          if (product.HasValue)
          {
              dic.Add(key, product.Value);
          }
          else
          {
              keysToLookup.Add(key);
          }
      }

      if (keysToLookup.Count > 0)
      {
          //Get data from the database
          var data = await getDataAsync(keysToLookup);
          foreach (var item in data)
          {
              await cache.SetAsync(item.Key, item.Value, duration, cancellationToken);
              dic.Add(item.Key, item.Value);
          }
      }

      return dic;
  }

[juantxorena]

I'm also very interested in this feature. I ended up implementing something similar to @onionhammer solution, but then the question is, how does this plays with cool things like AllowTimedOutFactoryBackgroundCompletion?

[onionhammer]

@jodydonetti have you given this one any more thought? I think the shape is relatively straightforward - and not uncommon; this is basically how HotChocolate's dataloaders are shaped.

I'm not sure if it's possible to access, or if it's too abstracted within IDistributedCache, but if you could do something like this (this isnt the GetOrSetMany version, obviously, just the GetMany)

    public static async Task<IReadOnlyDictionary<string, MaybeValue<T>>> TryGetManyAsync<T>(
        this IFusionCache cache,
        IReadOnlyCollection<string> keys,
        FusionCacheEntryOptions? options = null,
        CancellationToken cancellationToken = default
    )
    {
        options ??= new FusionCacheEntryOptions();
        var syncOptions = options.Duplicate().SetSkipDistributedCache(true, true);

        // Get all the keys which are already cached in L1
        Dictionary<string, MaybeValue<T>> cachedValues = new(capacity: keys.Count);
        foreach (var key in keys)
        {
            if (cache.TryGet<T>(key, syncOptions, cancellationToken) is { HasValue: true } value)
            {
                cachedValues.Add(key, value);
            }
        }

        // If the cache is redis-backed, utilize the distributed cache to fetch missing keys
        if (cache.HasDistributedCache && cache.DistributedCache is RedisCache distributed)
        {
            // TODO: Use MGET to fill in more missing keys
        }

        return cachedValues;
    }

[invalidoperation]

I also created an implementation of a batching function, which updates stale cache values in the background and uses GetOrSetAsync for stampede protection in case of multiple batches with the same keys, by hashing the missing and stale keys to a single key beforehand:

https://gist.github.com/invalidoperation/2d48e05c44cf66c9a188d480239fee2b

[onionhammer]

I would recommend trying to set in parallel rather than sequentially; you're awaiting each value in a loop sequentially

[invalidoperation]

Thank you for the suggestion, I updated the gist accordingly!

[ne4ta]

Hi,

I was looking for the same feature, but I couldn't find any library that provides this behavior. That's why I created my own library, called Cachula. I was inspired by FusionCache. Many thanks to @jodydonetti for the best documentation I've ever seen. It's been extremely helpful.

Some words about Cachula. For now it has the following features:

• Multi-layer caching (memory, Redis).
• Bulk operations: native support for efficient batch get/set/remove operations for multiple keys.
• Cache stampede protection: prevents multiple concurrent loads for the same key(s) using single-flight logic.
• Null/miss handling: distinguishes between cache misses and null values.

It's still an early beta version, but I'm looking forward to improving it. So if you like the idea, feel free to ask for features or report issues.