How do I delete a durable JetStream push subscription?

[SkinnySackboy]

Hi,

Today we were running a few experiments with a JetStream cluster of 3, individually turning off servers one at a time (but no more than 1 each time), to ensure we understand the behaviour precisely before we go into production. Our subscription types are async push subscriptions.

What we noticed (which might be expected behaviour) is that when randomly killing servers, any clients that were directly connected to the stopped server would correctly reconnect to another server, but their subscriptions would not resume. We presume this is by design as it is an ephemeral push subscription, because we resolved the problem by using durable subscriptoins.

What we had to do though was pass in a Guid or some unique durable name, to ensure that every time a client restarted it started from zero. So far so good, but we now likely have a problem where over time the server will accumulate stale durable subscription names. Is there a programmatic way to clean up/remove a durable subscription name once we are done with it?

Thanks in advance.

[scottf]

@SkinnySackboy converted this to a discussion.

[scottf]

So yes ephemerals disappear when there is a disconnect. But since you want to restart the client from zero, why do you need a durable, why not start a new ephemeral? The concept of durable is for exactly the situation where you do NOT want to start from zero, you want to start from where you left off.

Regarding consumer management, you can clean up durable consumers easily. You can accomplish this task both programmatically and using the cli. Since we are in the .NET discussion, programmatically, you would use the JetStreamManagement context. You can use

IList<string> GetConsumerNames(string streamName);
IList<ConsumerInfo> GetConsumers(string streamName);

to enumerate your consumers and you can use

bool DeleteConsumer(string streamName, string consumer);

to delete a consumer.

[SkinnySackboy]

Thanks @scottf

Our use case is one where the client's subscription must be durable only for the lifetime of the client. If the client is connected but for some reason the connection drops or the server dies, that client must continue (and not restart from zero) as soon as it manages to reconnect to another server in the cluster. The only case we want to start from zero is if the client process itself is restarted or a new client started for the first time. Many of these clients are started by users sporadically, possibly multiple on the same machine - so I cannot know in advance how many there will be.

Initially we assumed ephemeral connections where what we were after, so we created a JetStream cluster consisting of nodes A, B and C, with a stream created with replicas=3. As we were disconnecting servers (to test automatic client recovery), if for example an ephemeral push subscriber was connected to node B, terminating node B would successfully result in the client connecting to another node, but the ephemeral subscription at this point is dead. The client therefore never resumes consuming data from the ephemeral push subscription, even though the NATS connection has recovered. This is something we can always reproduce.

I'm assuming this is a feature and not a bug as it seems ephemeral subscriptions are tracked only by the server you are connected to, even though there is a cluster. The way we solved this problem is by switching to durable async push subscribers (each new client instance sets a unique string, like a Guid), and now the clients are resilient to random server failures, even if the server failure happened to be the one they were connected to. If a client is now connected to node B and node B fails, it takes a brief moment to connect to another server in the cluster, but the client stream subscription now resumes perfectly where it left off. The reason we use a unique string such as a Guid is because I cannot know how many clients are going to be running - machines may have a dynamic number of independent clients connected.

The only drawback I could think of now is if the client dies/crashes and doesn't terminate cleanly, it will forever leave an orphaned durable subscription name + consumer on the server. Probably no big deal, but this was the reason I asked about how to clean them up - to periodically check for orphaned subscriptions resulting from clients who didn't neatly unsubscribe to avoid leaving stale data on the server.

From the explanation (please do let me know if I didn't explain our use case clearly) does our strategy make sense, to use new durable subscriptions every time a client connects/restarts? If so, would it also make sense to use the the GetConsumers + DeleteConsumer trick to periodically delete any resulting stale connections?

Thanks for your help.

[scottf]

You could accomplish this with ephemerals by maintaining the consumer state yourself. Msg.MetaData is available with each message and contains a StreamSequence that you could track, for instance maybe in a kv store, a database, a file

So if a client is disconnected, you lookup last-finished-stream-sequence and create a new ephemeral consumer with a Consumer configuration using DeliverPolicy.ByStartSequence and setting StartSeq to the last-finished-stream-sequence + 1

[SkinnySackboy]

Cool I hadn't thought of that, thanks again for your help and also for your work on NATS, it's brilliant.