Adds Nats-Schedule-Rollup to schedules

[ripienaar]

Relates to nats-io/nats-server#7557

[ripienaar]

A bit non ideal since we can't detect this is happening on older servers, so maybe we should also add a thing where we check Nats-Schedule-* on the schedule messages and ensure we do not get any headers the current server dont support, that sets us up to expand this in future and inform users if they use a too old version, wdyt @MauriceVanVeen

[MauriceVanVeen]

LGTM

[MauriceVanVeen]

A bit non ideal since we can't detect this is happening on older servers, so maybe we should also add a thing where we check Nats-Schedule-* on the schedule messages and ensure we do not get any headers the current server dont support, that sets us up to expand this in future and inform users if they use a too old version, wdyt @MauriceVanVeen

We do already strip Nats-Schedule-* headers when publishing the delayed message, so at least this new header doesn't end up there. Only thing that will happen now is that a server would not recognize this, and silently ignore. Although not the best, I think this is fine for now and we should look at this from a broader perspective when we get to the feature flags work. This could then do proper acknowledging that all servers support a feature and error otherwise, etc.

[ripienaar]

Although not the best, I think this is fine for now and we should look at this from a broader perspective when we get to the feature flags work. This could then do proper acknowledging that all servers support a feature and error otherwise, etc.

Ok so we just silently fail for now? We have no idea how long feature flags work will take, and then we would have to check them anyway right?

Getting the check into 2.12.x might help?

[MauriceVanVeen]

Getting the check into 2.12.x might help?

We could add detection of unknown NATS headers in nats-io/nats-server#7454. There we scan through all the headers and could error if we see an unknown suffix. But that's currently also reserved for 2.14, although we could perhaps backport if we'd like.

[ripienaar]

Getting the check into 2.12.x might help?

We could add detection of unknown NATS headers in nats-io/nats-server#7454. There we scan through all the headers and could error if we see an unknown suffix. But that's currently also reserved for 2.14, although we could perhaps backport if we'd like.

not sure we can back port all the things there but yeah looking at the schedule code I can see how handling it there is a pain, will think

[MauriceVanVeen]

Should we also add support for a Nats-Schedule-Msg-Id to add deduplication. So a delayed message can't be duplicated if the user wants to publish the message earlier and have the schedule not duplicate that message when it triggers? (otherwise there's a race condition here)

[ripienaar]

Should we also add support for a Nats-Schedule-Msg-Id to add deduplication. So a delayed message can't be duplicated if the user wants to publish the message earlier and have the schedule not duplicate that message when it triggers? (otherwise there's a race condition here)

Yes we could do that, but it would be quite limited right since dupe windows tend to be short - I mean for the use case where the user would want to publish earlier it would be quite limiting to be only able to do so in the dupe window, and this might insentivise users to make very long dupe windows which might not be an ideal outcome

[MauriceVanVeen]

Thinking a bit more on that.. the use case of being able to 'end' a schedule early is valuable. Especially in the case where you want to ensure that ending the schedule early still guarantees no duplicate messages on the target subject. However, I agree now solving this with deduplication isn't the right way. It becomes confusing even for an @every 1s schedule with deduplication of 2 minutes.. that would then not schedule every second but be limited by deduplication.

Instead was thinking combining these two already existing headers to achieve this use case:

• Using Nats-Expected-Last-Subject-Sequence-Subject: <schedule>, Nats-Expected-Last-Subject-Sequence: <scheduleSeq>. This ensures the schedule you want to 'end' early still exists, and guards against concurrent modification.
• Using Nats-Scheduler: <schedule>, Nats-Schedule-Next: purge headers. The server can then block if concurrent messages with these headers were sent.

The combination of those, since schedules are purged based on the Nats-Schedule-Next: purge. Would ensure that:

• If you publish later (slightly or way later) the schedule would already be gone for single-delayed schedules, so your publish can't duplicate.
• If your publish came in slightly earlier than the schedule, the schedule's message would be blocked since the server would notice another purge is already being proposed through.

What are your thoughts around formalizing that behavior?
The nice thing is that this doesn't rely on deduplication, and would not only allow you to reliably 'end' single-delayed schedules, it could also guarantee that repeating interval/cron schedules finish based on this 'ending message'.