aspire compatible facade for cosmos global distributed lock

[secheledelaruse]

this follows up an earlier idea of an aspire facade for the cosmos db global distributed lock design pattern listed here

execution summary
the canonical distributed lock strategy is amenable to abstraction with dependency injection

my back of the napkin analysis suggests that independent client consumption of the api surface requires only each client to maintain its own lock ownership state, which leverages global atomicity via the underlying cosmos lock lease with ownership information updatable and queryable by any client

to do

• show whether consuming clients should specify a cosmos resource in their app.host and consume this resource via a client integration. this would permit the cloud native opinions in app host like observability, resiliance etc. perhaps this resource cculd expose the target cosmos container
• prove correctness
• shape the repo like the community toolkit
• read further in the documentation

accomplished so far in this repo

abstracted design pattern guidance example code with dependency injection friendly interfaces and api surface

first cut of aspire custom client integration with simple settings that require a url and registers the ecosystem of dependent services in di

working on integrating the canonical azure durable functions say hello orchestration with a timer trigger interlocked with the lease duration for distribued locks

if this interests you

• i don't yet know how to reference a cosmos resource registered in apphost from a custom client integration, or whether there should be a custom resource that wraps a cosmos resource with extra config, and if the latter is even possible

regards

[davidfowl]

i don't yet know how to reference a cosmos resource registered in apphost from a custom client integration, or whether there should be a custom resource that wraps a cosmos resource with extra config, and if the latter is even possible

hosting integrations pass environment variables to the application, if you can read them then you are golden.

[secheledelaruse]

execution summary

• the lock api surface is amenable to Task<TLeaseRequestStatus?> AcquireLeaseAsync(TLeaseRequest leaseRequest, Func<TLeaseRequestStatus?, Task>? onSuccessAsync = null) semantics
• fence tokens really seem to disconnect threads from ownership when the fence token is monotonically incremented by the (cosmos) storage api

example artifact

bearing in mind

testbed ascii model
``` timer trigger --entrypoint client caller model -> entrypoint orchestration -> entrypoint lease acquisition activity -> `gated suborchestration --acquired lease model --> gated suborchestration activity``

as abridged per

        [Function("GatedLauncherTimer")]
        public async Task GatedLauncherTimer(
        [TimerTrigger("*/3 * * * * *")] TimerInfo timerInfo,
        [DurableClient] DurableTaskClient client,
        FunctionContext context)
        {
            ILogger logger = context.GetLogger("TimerTriggerExample");
            logger.LogInformation("Timer trigger executed at: {time}", DateTime.UtcNow);
            LeaseRequest leaseRequest = new
            (
                context.InvocationId.ToString(), // a dynamic client id that will try to re/acquire a lease
                20, // lease duration in seconds
                0, // on entry the fence token idea calls for zero here
                 nameof(GatedOrchestrationLauncher), // this is part of a stable name of the lock
                context.FunctionId // this is part of a stable name of the lock 
            );

            string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(
                nameof(GatedOrchestrationLauncher), leaseRequest);
            // Add your logic here
            logger.LogInformation($"schedule attempt with {JsonSerializer.Serialize(leaseRequest)}");
        }

a gateway that tries to satisfy the lease request

        [Function(nameof(GatedOrchestrationLauncher))]
        public async Task GatedOrchestrationLauncher([OrchestrationTrigger] TaskOrchestrationContext context)
        {
            ILogger logger = context.CreateReplaySafeLogger(nameof(GatedOrchestration));
            var leaseRequest = context.GetInput<LeaseRequest>();
            logger.LogInformation($"GatedOrchestrationLauncher started with {JsonSerializer.Serialize(leaseRequest)}");
            // call the gating activity that returns lease request status messages, necessary for further operations against the same lease
            var gatedActivityResult = await context.CallActivityAsync<LeaseRequestStatus>(nameof(GatedLauncherActivity), leaseRequest);

            if(gatedActivityResult != null && gatedActivityResult.isAcquired)
            {
                var model = new GatedOrchestrationModel()
                {
                    LeaseRequest = leaseRequest,
                    LeaseRequestStatus = gatedActivityResult
                };

                // call the gated suborchestration
                var gatedSubOrchestrationResult = await context.CallSubOrchestratorAsync<GatedOrchestrationModel>(nameof(GatedOrchestration), model);

                logger.LogInformation($"GatedOrchestrationLauncher completed with {JsonSerializer.Serialize(gatedSubOrchestrationResult)}");
            }
            else
            {
                logger.LogInformation($"GatedOrchestrationLauncher failed with {JsonSerializer.Serialize(gatedActivityResult)}");
            }
        }

and the associated lock orchestration api operation surface

       [Function(nameof(GatedLauncherActivity))]
       public async Task<LeaseRequestStatus> GatedLauncherActivity([ActivityTrigger] LeaseRequest leaseRequest, FunctionContext context)
       {
           ILogger logger = context.GetLogger("GatedActivity");
           // try to acquire an existing leaseRequest
           // construct the updated parameters
           logger.LogInformation($"Lease Request resourceName: {leaseRequest.gatedResourceName} will try to claim its leaseRequest");
           var updatedStatus = await _lockManager.AcquireLeaseAsync(leaseRequest, onSuccessAsync: async (status) =>
           {
               // this is the success path
               // we can start the orchestration
               logger.LogInformation($"Lease Request resourceName: {leaseRequest.gatedResourceName} claimed its leaseRequest");
               await Task.CompletedTask;

           }, onFailureAsync: async (status) =>
           {
               // this is the failure path
               // we can retry or fail the orchestration
               logger.LogInformation($"Lease Request resourceName: {leaseRequest.gatedResourceName} could not claim its leaseRequest");
               await Task.CompletedTask;
           });

           return updatedStatus;
       }

regards

[secheledelaruse]

operation summary

work on proving the behavior of the aspire compatible cosmos distributed lock client has converged on

• testing correctness by triggering azure durable orchestrations that consume the lock with
  • a cosmosdb changefeed azure functions trigger that
  • observes the container where the lease schedule is deployed and
  • triggers an orchestration for each lease request that
  • waits until the scheduled time, then
  • calls the lock acquisition activity
  • and records lease acquired / not / exception
  • with a variation that delivers the schedule via scheduled servicebus messages, which would permit collision recovery scenario testing for 100% delivery at most once

• generating the lease acquisition schedule with a negative binomial distribution salted with hyperparameters interpretable as
  • the number of requests and the minimal number of first time successful lease acquisitions desired
  • number of lock contenders
  • number of locks that can be acquired by the contenders and a range of lock hold times
  • probability of acquiring a lock for a given combination of contender and lock
  • a model timedomain scaled to accommodate the (mean or average or minimum or maximum) lease hold time for the number of requests

the model generates a nicely annoying distribution across the timedomain with multiple collision surfaces

the testbed makes sumptuous use of containers, and the cosmos vnext simulator

regards

[secheledelaruse]

on security

a global lock with its state persisted in cosmosdb exposes implementation details that provoke the following follow ons

• not all consomers of the lock may be in the same administrative domain, and some consumers may be competitors
• the owner of the storage accounts involved may not be part of the same administrative entitiy as the consumers
• the ability to perform operations against the lock may not correspond with the current 'right' to do so

as such the obvious encryption layer required

• should restrict access to the keys required
• should transparently apply key management operations against the keys in use
• should probably permit the use of individual keys per consuming entity, and bring your own key scenarios

example suspect persisted artifact

              var leaseOperationEvent = new LeaseOperationEvent
              {
                  LeaseRequest = new[] { leaseRequest },
                  LeaseRequestStatus = new[] { requestStatus },
                  EventUTCTime = DateTime.UtcNow,
                  EntityName = nameof(LeaseOperationEvent).HashToGuid(),
                  BatchId = request.BatchId,
                  Id = Guid.NewGuid(),
                  ObjectId = Guid.NewGuid(),
              };

regards