Handling a potentially unbounded set of external events in an orchestration

[olitomlinson]

Context : An open Registration system whereby anyone can text in with an SMS to register for participation in a real-life event. The registration has a closing deadline. Registrations received after the deadline are dropped.

[FunctionName(nameof(RegistrationPhase))]
        public async Task<List<string>> RegistrationPhase(
            [OrchestrationTrigger] IDurableOrchestrationContext context)
        {
            var phoneNumbers = new List<string>();

            Task registrationClosed = context.WaitForExternalEvent(Orchestrations.CLOSE_REGISTRATION);
            context.SetCustomStatus($"Accepting registrations, no registrations so far...");

            while (true)
            {
                var registration = context.WaitForExternalEvent<string>(Orchestrations.REGISTER_ATTENDEE);

                var @event = await Task.WhenAny(registrationClosed, registration);

                if (@event == registration)
                {
                    var phoneNumber = ((Task<string>)@event).Result;
                    
                    if (!phoneNumbers.Contains(phoneNumber)) 
                    {
                        phoneNumbers.Add(phoneNumber);
  
                        var message = new RichContentMessage()
                            .WithAccountReference("XXXXXX")
                            .WithRecipient(new Recipient().WithPhonenumber(phoneNumber))
                            .WithTextMessage("Thank you for registering. You will be contacted closer to the event taking place.")
                            .UseRcs()
                            .UseSms();

                        await context.CallActivityAsync(nameof(Activities.SendMessage), message);

                        context.SetCustomStatus($"Accepting registrations, {phoneNumbers.Count} registrations made so far...");
                    }

                    continue;
                }
                if (@event == registrationClosed)
                {
                    context.SetCustomStatus($"Registration Closed. {phoneNumbers.Count} registrations made");

                    return phoneNumbers.ToList();
                }
            }
        }

Given the above code, lets assume that hundreds, even thousands, of people registered before the deadline.

However, hypothetically, lets assume that thousands of people registered in the same moment of time. This would essentially flood the Orchestration with many external events (that will be matched and realised via context.WaitForExternalEvent<string>(Orchestrations.REGISTER_ATTENDEE);)

I understand that the runtime maintains an internal queue of external events that it attempts to match to counterpart code such as context.WaitForExternalEvent.... so this pattern should work in theory.

My question is how long is this internal queue of unmatched events allowed to be, before older or newer events are dropped? 10s, 100s, 1000s?

Also, are there any other rules with how this matching occurs?

For example, is there any time criticality to it? i.e. if an external event isn't matched within 10 seconds of its arrival it is dropped?

Or does iterations of the orchestration matter? i.e. if the orchestration has advanced through N async/await cycles and the event hasn't been matched, it is dropped etc?

Thanks in advance!

P.s. I know the above would be better expressed as an Eternal Orchestration to stop the history growing crazy.

[cgillum]

This is a good scenario and definitely worth talking about.

My question is how long is this internal queue of unmatched events allowed to be, before older or newer events are dropped? 10s, 100s, 1000s?

An orchestration won't drop any events as long as it remains in a running (or pending) status. All of them will be stored in the orchestration's history, and in-this case, they'll stick around in-memory until they are consumed. I think this might be the answer to several of your questions.

Also, are there any other rules with how this matching occurs?

The matching is first by name and then by the order in which an event is received. It happens to be that we'll surface these events in a last-in, first-out (e.g. stack) order, but I don't recommend taking any dependency on this behavior. It may change to FIFO in future versions.

Also, you didn't ask this question explicitly, but I thought it would be worth mentioning: any event received before the "deadline" event should get processed by your orchestration. This is because each event is stored in the orchestration history in the order it was received. Note however that we can't make this guarantee for messages that are still waiting in the queue since Azure Queues are not ordered. It's therefore possible that if a submission was received a few seconds or even a few minutes before the deadline, but if the orchestration hasn't yet picked up those messages, it's possible that some kind of processing failure could cause those messages to "miss" the deadline.

As an aside, it's also worth mentioning that you could alternatively model the deadline using a Durable Timer that expires at a specific time. This is a slightly simpler design since timers are self-triggering. However, I'll acknowledge that they are less flexible since you can't change the expiration time of a durable timer after it has been scheduled. :)

[olitomlinson]

Thanks so much!

The matching is first by name and then by the order in which an event is received. It happens to be that we'll surface these events in a last-in, first-out (e.g. stack) order, but I don't recommend taking any dependency on this behavior. It may change to FIFO in future versions.

This bit confuses me, and I'm interpreting it as a contradiction?

If matching observes the time, this would imply that older events will be matched first? But then you mention that newer events will get popped as its actually a stack? Sorry...?

---

any event received before the "deadline" event should get processed by your orchestration. This is because each event is stored in the orchestration history in the order it was received.

This is an important nuance that I did not know. So just so I get this straight, if both Tasks are complete (event is received for registration, and deadline has been reached) --- the oldest event will be handled as the 'winner' in Task.WhenAny ? So there could be many registrations successfully processed, until the deadline Task becomes next in the timeline of events?

---

it's also worth mentioning that you could alternatively model the deadline using a Durable Timer

Yeah, I did initially run with a DurableTimer but adjusted the design because the deadline in this scenario could be brought forward or pushed back.

[cgillum]

If matching observes the time, this would imply that older events will be matched first? But then you mention that newer events will get popped as its actually a stack? Sorry...?

I don't recall saying that matching observes the time - and I'm actually not sure what this means. We care about order, but not about time per-se. My statement is simply that events with the same name will be popped up in LIFO order, currently.

if both Tasks are complete (event is received for registration, and deadline has been reached) --- the oldest event will be handled as the 'winner' in Task.WhenAny ?

No, the two events in your Task.WhenAny example have different names (Orchestrations.CLOSE_REGISTRATION and Orchestrations.REGISTER_ATTENDEE), so whichever event gets saved into the orchestration history first is the winner. The LIFO behavior is only for events that have the same name that were buffered in the orchestration history. For your scenario it simply means attendees that registered more recently may get their SMS confirmation delivered sooner than those who may have registered a few seconds/minutes sooner. It's a bit counter-intuitive, which is why I want to change it to a FIFO model in a future version. however, it doesn't change the fact that all of these users will get their notifications of a successful registration.

[olitomlinson]

Sorry I took this to be about time :

The matching is first by name and then by the order in which an event is received <-- i.e. the time it is received.

But yes, I get what you're saying now. Its a stack not a queue.

---

For your scenario it simply means attendees that registered more recently may get their SMS confirmation delivered sooner than those who may have registered a few seconds/minutes sooner. It's a bit counter-intuitive, which is why I want to change it to a FIFO model in a future version. however, it doesn't change the fact that all of these users will get their notifications of a successful registration.

Got it 100% - Thanks for taking the time to explain!

Is changing this to FIFO being tracked as an issue anywhere? I guess its a breaking change so v3 milestone?

[cgillum]

Is changing this to FIFO being tracked as an issue anywhere? I guess it's a breaking change so v3 milestone?

Yes, more-or-less. I'm looking at using Durable Functions for .NET Isolated as the opportunity for doing a "v3", since pretty much everything about moving from .NET in-proc to .NET Isolated is a breaking change already. :) Future SDKs, like Java, will also use the more intuitive FIFO model.