Clarification on per-node Pub/Sub subscription provisioning

[danielwinkler]

With regards to the Pub/Sub provisioning, I am wondering if there is a reason why every node creates its own subscription - see PubsubEndpoint::SetupAsync:

if (!IsDeadLetter)
{
    Server.Subscription.Name =
        Server.Subscription.Name.WithAssignedNodeNumber(_transport.AssignedNodeNumber);
}

The issue with this approach is that messages will be dropped whenever there is no node up to handle incoming messages (e.g., before the first subscription is created).

Although topics can be configured with message retention (see Pub/Sub topic retention), retained messages are not automatically replayed into newly created subscriptions. This means that any messages published before the first subscription exists, or after a node-specific subscription is deleted, will be lost.

If instead there was a single shared subscription that all nodes subscribed to, the system would accumulate unprocessed messages on that subscription. When a node connects, it could immediately start consuming from the backlog without losing messages.

While the bootstrapping scenario above is the simplest example, this design limitation applies more broadly: since the AssignedNodeNumber is randomly generated, subscriptions are not reused across node lifecycles. As a result, messages pending in “dead” subscriptions will never be picked up by new nodes.

@jay-zahiri @jeremydmiller

[jeremydmiller]

@danielwinkler That particular subscription should be for request/reply mechanics directly to the current node. I haven't been as deep into the GCP internals as I should have, so might be off.

And please don't use GitHub issues for questions in the future. We prefer Discord to handle pure questions, or a discussion w/ a Discord message to go look at it:)