FailSafe - Not Working As Expected

[bendursley]

Morning!

I've been looking at FusionCache for some work I'm doing, and so far it's been great! But I'm a bit stumped as to what's happening when we turn on FailSafe.

I've set up a unit test with a fully initiated version of FusionCache, and have a test which will get from an inner method (successfully), then wait some time, then try again (after the cache entity has been moved from active to expired, but not removed). Then I throw an exception from the inner method... but I get the exception bubbled up out of the method I'm testing, rather than the expired entity back, which is what I thought would happen according to the docs?

public class CacheAccessor : IAccessor

{
        private readonly IFusionCache fusionCache;
        private readonly IAccessor inner;
        private readonly CacheOptions configurationOptions;

        public CacheAccessor(
            IFusionCacheProvider fusionCacheProvider,
            IOptions<CacheOptions> options,
            IAccessor inner)
        {
            configurationOptions = options.Value;

            fusionCache = fusionCacheProvider.GetCache(CacheConstants.CacheName);
            this.inner = inner;
        }

        Task<T> IAccessor.GetConfiguration<T>(int customerId, CancellationToken token)
        {
            var typeName = typeof(T).FullName;

            return fusionCache.GetOrSet(
                key: $"CustomerConfigSingle-{typeName}-{customerId}",
                factory: _ => inner.GetConfiguration<T>(customerId, token),
                setupAction: options => options
                    .SetDuration(configurationOptions.CacheDuration)
                    .SetFailSafe(
                        isEnabled: true,
                        maxDuration: configurationOptions.ExpiredCacheDuration,
                        throttleDuration: configurationOptions.FailSafeThrottleDuration),
                token: token);
        }
}

public class CacheAccessorTests
{
    private Mock<IAccessor> Inner { get; } = new();
    private IAccessor CacheAccessor { get; }

    public CacheAccessorTests()
    {
        var factory = LoggerFactory.Create(b => b
            .SetMinimumLevel(LogLevel.Trace)
            .AddSimpleConsole(options => options.IncludeScopes = true)
        );

        var logger = factory.CreateLogger<FusionCache>();
        var provider = new Mock<IFusionCacheProvider>();
        Inner = new Mock<IAccessor>();

        var fusionCache = new FusionCache(
            new FusionCacheOptions { FailSafeActivationLogLevel = LogLevel.Trace }, logger: logger);

        provider
            .Setup(x => x.GetCache(It.IsAny<string>()))
            .Returns(fusionCache);

        var mockOptions = Options.Create(new CacheOptions
        {
            CacheDuration = TimeSpan.FromSeconds(3),
            ExpiredCacheDuration = TimeSpan.FromMinutes(2),
            FailSafeThrottleDuration = TimeSpan.FromSeconds(5)
        });

        CacheAccessor = new CacheAccessor(
            provider.Object,
            mockOptions,
            Inner.Object);
    }

    [Fact]
    public async Task GetConfiguration_ByCustomerId_AndFails_ShouldReturnCachedValue()
    {
        var customerId = 123;
        var expectedConfig = new TestConfiguration { Name = "TestCustomer" };
        Inner
            .Setup(x => x.GetConfiguration<TestConfiguration>(customerId, It.IsAny<CancellationToken>()))
            .ReturnsAsync(expectedConfig);

        var result1 = await CacheAccessor.GetConfiguration<TestConfiguration>(
            customerId, CancellationToken.None);

        result1.Should().BeEquivalentTo(expectedConfig);

        await Task.Delay(TimeSpan.FromSeconds(4));

        var exception = new InvalidOperationException("Test exception");

        Inner
            .Setup(x => x.GetConfiguration<TestConfiguration>(customerId, It.IsAny<CancellationToken>()))
            .ThrowsAsync(exception);

        var result2 = await CacheAccessor.GetConfiguration<TestConfiguration>(
            customerId, CancellationToken.None);
        result2.Should().BeEquivalentTo(expectedConfig);

        Inner
            .Verify(
                x => x.GetConfiguration<TestConfiguration>(
                    customerId,
                    It.IsAny<CancellationToken>()),
                Times.Exactly(2));
    }

    private class TestConfiguration
    {
        public string Name { get; set; }
    }
}

[jodydonetti]

Hi @bendursley , on which versions of FusionCache/.NET/OS/etc are you?

[jodydonetti]

Anyway, as an MRE you can try this:

var fc = new FusionCache(new FusionCacheOptions());

var v1 = fc.GetOrSet<int>(
	"foo",
	_ => 123,
	options => options
		.SetDuration(TimeSpan.FromSeconds(3))
		.SetFailSafe(true, TimeSpan.FromMinutes(2), TimeSpan.FromSeconds(5))
);

Console.WriteLine($"v1: {v1}");

await Task.Delay(TimeSpan.FromSeconds(4));

var v2 = fc.GetOrSet<int>(
	"foo",
	_ => throw new Exception("BOOM!"),
	options => options
		.SetDuration(TimeSpan.FromSeconds(3))
		.SetFailSafe(true, TimeSpan.FromMinutes(2), TimeSpan.FromSeconds(5))
);

Console.WriteLine($"v2: {v2}");

For which you can see that the output is:

v1: 123
v2: 123

As soon as you disable fail-safe on the 2nd call, you'll get this output:

v1: 123
Unhandled exception. System.Exception: BOOM!
   at Program.<>c.<<Main>$>b__0_2(CancellationToken _) in C:\...\Program.cs:line 20
   at ZiggyCreatures.Caching.Fusion.FusionCacheExtMethods.<>c__DisplayClass53_0`1.<GetOrSet>b__0(FusionCacheFactoryExecutionContext`1 _, CancellationToken ct) in /_/src/ZiggyCreatures.FusionCache/FusionCacheExtMethods_FactoryNoCtx_Sync.cs:line 205
   at ZiggyCreatures.Caching.Fusion.FusionCache.GetOrSetEntryInternal[TValue](String operationId, String key, String[] tags, Func`3 factory, Boolean isRealFactory, MaybeValue`1 failSafeDefaultValue, FusionCacheEntryOptions options, Activity activity, CancellationToken token) in /_/src/ZiggyCreatures.FusionCache/FusionCache_Sync.cs:line 250
   at ZiggyCreatures.Caching.Fusion.FusionCache.GetOrSet[TValue](String key, Func`3 factory, MaybeValue`1 failSafeDefaultValue, FusionCacheEntryOptions options, IEnumerable`1 tags, CancellationToken token) in /_/src/ZiggyCreatures.FusionCache/FusionCache_Sync.cs:line 399
   at ZiggyCreatures.Caching.Fusion.FusionCacheExtMethods.GetOrSet[TValue](IFusionCache cache, String key, Func`2 factory, Action`1 setupAction, IEnumerable`1 tags, CancellationToken token) in /_/src/ZiggyCreatures.FusionCache/FusionCacheExtMethods_FactoryNoCtx_Sync.cs:line 205
   at Program.<Main>$(String[] args) in C:\...\Program.cs:line 18
   at Program.<Main>(String[] args)

Maybe it's something with the mocking part?

[bendursley]

It was indeed the mocking in my test!

[bendursley]

        Inner
            .Setup(x => x.GetConfiguration<TestConfiguration>(customerId, It.IsAny<CancellationToken>()))
            .ThrowsAsync(exception);

Should of been

        Inner
            .Setup(x => x.GetConfiguration<TestConfiguration>(customerId, It.IsAny<CancellationToken>()))
            .Throws(exception);

As my class just returns a task, and does not use async \ await.

[bendursley]

Interesting, I'll have a look at this today, thanks for responding @jodydonetti !

We're on the latest version of FusionCache, and .NET 8. We're also using Moq for our mocking framework

[bendursley]

@jodydonetti It was indeed the mock, made a mistake and a fresh set of eyes caught it. Thanks for the help!

[jodydonetti]

Great, thanks for the update!