ADR-60: JetStream reliable stream sourcing/mirroring on WQ/Interest streams

[MauriceVanVeen]

Stream sourcing/mirroring on WorkQueue or Interest streams is not reliable due to the use of an ephemeral AckNone consumer. This ADR proposes a design for durable stream sourcing/mirroring that works reliably on all stream retention types.

Signed-off-by: Maurice van Veen github@mauricevanveen.com

[neilalexander]

@roeschter Please add comments inline using the review function, otherwise it's extremely difficult to thread and track replies to individual comments.

[roeschter]

Missing specifications?
-Will the sourcing side verify that the consumer has the right AckPolicy of AckFlowControl?
-What will happen to sourcing from a workqueue is NOT consumer is set? WE shouldn't fail (compatibility), but should we at least print a warning

[jnmoyne]

LGTM

[MauriceVanVeen]

Opened a draft PR on the server with a working version for this. I'll update this ADR with some refinements soon.

[MauriceVanVeen]

This ADR is now also updated to reflect the initial implemention. Requiring to specify MaxAckPending as upper bound for pending messages, unset AckWait/BackOff, and infinite MaxDeliver.

[ripienaar]

LGTM, feels like we need a proper ok from @derekcollison also maybe?

[roeschter]

I vaguely remember there is a reasons for moving the ack floor. In the next section we talk about flow control ack being done according to window size. So this one is for very slow moevign streams?

But if the source/mirror is idle (low message rate) the Heartbeat will also trigger a flow control message to move the acknowledgement floor up even if the window size has not been reached.

[MauriceVanVeen]

Indeed, if a stream receives small messages very slowly, it will not trigger the flow control messages often. So, if the stream hasn't received messages for Heartbeat amount of time, then a flow control message will also be sent. This ensures that a WorkQueue/Interest stream can remove messages after ack in such a slow/idle stream case.

[MauriceVanVeen]

Updated the ADR to include a "hybrid option" between ephemeral consumer and a pre-created durable consumer that needs to be fully managed by the user. This hybrid is a "server-managed" durable consumer. The server creates a durable instead of ephemeral consumer, but otherwise functions in exactly the same way. Importantly, it allows for easy migration from ephemeral to durable sourcing, as well as not requiring too much additional work from the user when not necessary.

[MauriceVanVeen]

Updated the ADR and server PR to automatically "upgrade" the ephemeral sourcing consumer to be durable if sourcing from a WQ/Interest stream. This ensures this type of sourcing will work reliably after performing the server upgrade. The "bring your own consumer" approach is also kept, but is an opt-in still.