BYO source/mirror `$JSC.R.*` replies cannot be isolated between leafnodes

[shaunco]

Observed behavior

NATS 2.14's durable/BYO sourcing is a really nice improvement for leafnode deployments where the source consumer needs to be pre-created with tightly controlled permissions rather than allowing the remote side to create arbitrary consumers.

However, I think there is an isolation problem with the CONSUMER.RESET reply path.

I have a cloud NATS account with multiple leafnodes, where each leafnode represents a separate edge gateway/device. The gateways may be physically outside the control of the cloud operator, so the cloud-side leaf credentials are intentionally locked down to only that gateway's subjects/resources. For example, if NATS was used on something like a Philips Hue gateway that is installed at a customer's home, but configured as a leaf-node to the Hue servers, it is unacceptable for there to be any East<->West traffic between Hue gateways (to be clear, this has no relation to Hue, but it is an easy example).

For example, GW123 can be restricted to things like:

GW123._sync.*
$JS.GW123.API.*
$JS.cloud.API.CONSUMER.RESET.GW123_C2D_STREAM.GW123_C2D_SOURCE_TO_EDGE

while GW456 gets the equivalent GW456 subjects.

This works quite well for basically everything else. Even the CONSUMER.RESET request itself can be restricted to the exact stream and durable consumer:

$JS.cloud.API.CONSUMER.RESET.GW123_C2D_STREAM.GW123_C2D_SOURCE_TO_EDGE

The problem is that the response comes back on:

$JSC.R.<opaque uid>

There is no domain, account, stream, consumer, gateway, etc. encoded in that subject, so the permission can only realistically be:

$JSC.R.*

(or $JSC.R.>).

In a bidirectional leaf/source topology, the leaf credentials end up needing both publish and subscribe access to this common namespace.

I tested this with a real three-server topology:

flowchart TB
    cloud["cloud"]

    GW123["GW123<br/>leaf"]
    GW456["GW456<br/>leaf"]

    cloud --- GW123
    cloud --- GW456

Loading

GW123 and GW456 are in the same cloud account, but all of their application, JetStream API, sync delivery, ACK, flow-control, and RESET permissions are gateway-specific.

On GW456 I can subscribe to:

$JSC.R.*

then restart GW123 so that its durable source/mirror relationship performs a CONSUMER.RESET.

GW456 receives the $JSC.R.<uid> response associated with GW123's source/mirror recovery.

Likewise, if the leaf credential has the publish permission required for the opposite direction of sourcing, it can publish arbitrary messages into:

$JSC.R.<anything>

I haven't tried to claim that an arbitrary injected response can necessarily spoof a valid reset response yet, but the credential can both observe the shared reply namespace and inject messages into the same namespace.

This seems especially strange because durable/BYO sourcing is useful specifically when trying to reduce the permissions granted across a leaf connection. The actual RESET operation is perfectly ACL-able down to a single known stream/consumer, then the reply jumps back into one global opaque namespace.

The key issue isn't that $JSC.R exists; it is that the opaque UID is the only identifying component of the reply subject, leaving no subject component that can be used to authorize one source/mirror relationship without authorizing every other $JSC.R reply in the account.

The new js_ack_fc_v2 format solves essentially this same namespacing problem for ACK/flow-control subjects by including domain/account identity, but it does not change $JSC.R.

Putting every gateway into a separate NATS account avoids the cross-gateway visibility, but that seems like a pretty substantial workaround for something that otherwise fits naturally into subject-level authorization. For large IoT/edge deployments it also means per-device accounts/imports/exports rather than simply giving each leaf credential access to its own subjects.

(I may be missing another mechanism for scoping these internal replies, but I haven't found one in ADR-59, ADR-60, the 2.14 docs, or the server/client APIs.)

Expected behavior

A leaf credential should not need access to a reply namespace shared with other otherwise-isolated leaf credentials in the same account.

In particular, it should be possible to authorize GW123 for the internal replies required by:

$JS.cloud.API.CONSUMER.RESET.GW123_C2D_STREAM.GW123_C2D_SOURCE_TO_EDGE

without also giving GW123 visibility into the replies generated for GW456, GW789, etc.

I don't particularly care what the exact implementation looks like.

For example, the reply subject could contain enough identity to authorize statically, somewhat analogous to the new ACK/FC v2 format:

$JSC.R.<domain>.<account>.<stream>.<consumer>.<uid>

or the server could dynamically authorize only the exact reply subject associated with an allowed request.

The important property is that a leaf authorized only for GW123 resources should not need a wildcard permission that exposes JetStream internal reply traffic belonging to GW456.

The exact CONSUMER.RESET.<stream>.<consumer> request authorization already provides the information necessary to know which source relationship the reply belongs to, so it seems like that isolation should survive the request/reply boundary.

---

While a js_ack_fc_v2 change to a non-UID named JSC.R.> subject seems the safest here, as there is no concern about break backward compatibility without opt-in, I still believe this is the better longer term fix

Server and client version

nats-server: v2.14.6
nats.go:      v1.53.1
nats CLI:    v0.4.0

nats-server v2.14.6 is the latest release as of this report.

The behavior is specifically with the NATS 2.14 durable/BYO sourcing support (StreamSource.Consumer / pre-created AckFlowControl consumer).

The same $JSC.R.<uid> behavior remains regardless of the new ACK/FC v2 subject format; js_ack_fc_v2 changes $JS.ACK / $JS.FC namespacing, not this RESET/API reply namespace.

Host environment

Local reproduction:

OS:           Ubuntu 24.04, x86_64
Topology:     3 standalone nats-server processes
              1 cloud/hub + 2 JetStream leafnodes
JetStream:    FileStorage
Leaf domains: GW123 and GW456
Cloud domain: cloud
Accounts:     all in one account, name doesn't matter

There is no Docker/Kubernetes/networking dependency in the reproduction. All three servers can run locally on different ports.

The real use case is edge/IoT gateways connecting as leafnodes to a central NATS service, where the machine running each leafnode may be physically controlled by a customer/site rather than by the operator of the cloud NATS account. For example, if NATS was used on something like a Philips Hue gateway that is installed at a customer's home, but configured as a leaf-node to the Hue servers, it is unacceptable for there to be any East<->West traffic between Hue gateways (to be clear, this has no relation to Hue, but it is an easy example).

Steps to reproduce

The important part of the topology is two separately credentialed leafnodes attached to the same cloud account.

Each leaf credential is otherwise restricted to its gateway.

Conceptually, the cloud-side permissions look like:

GW123_LEAF:

publish:
  GW123._sync.*
  gateway-specific $JS.ACK / $JS.FC subjects
  $JS.cloud.API.CONSUMER.RESET.GW123_C2D_STREAM.GW123_C2D_SOURCE_TO_EDGE
  $JSC.R.*

subscribe:
  $JS.GW123.API.*
  GW123._sync.*
  gateway-specific $JS.ACK / $JS.FC subjects
  $JSC.R.*

and:

GW456_LEAF:

publish:
  GW456._sync.*
  gateway-specific $JS.ACK / $JS.FC subjects
  $JS.cloud.API.CONSUMER.RESET.GW456_C2D_STREAM.GW456_C2D_SOURCE_TO_EDGE
  $JSC.R.*

subscribe:
  $JS.GW456.API.*
  GW456._sync.*
  gateway-specific $JS.ACK / $JS.FC subjects
  $JSC.R.*

There is deliberately no permission for GW123 to access a GW456.* application subject, $JS.GW456.API.*, GW456's RESET endpoint, etc., and vice versa.

Create a WorkQueue origin stream in the cloud for GW123:

stream:    GW123_C2D_STREAM
subjects:  GW123.c2d.>
retention: WorkQueue
storage:   FileStorage

Pre-create the durable BYO sourcing consumer on that origin:

name/durable:    GW123_C2D_SOURCE_TO_EDGE
filter:          GW123.c2d.>
deliver subject: GW123._sync.c2d

ack policy:      AckFlowControl
flow control:    true
heartbeat:       1s
max ack pending: > 0
max deliver:     -1
ack wait:        unset
backoff:         unset

On GW123 create the edge mirror/source and bind that consumer:

Mirror: &jetstream.StreamSource{
    Name:   "GW123_C2D_STREAM",
    Domain: "cloud",
    Consumer: &jetstream.StreamConsumerSource{
        Name:           "GW123_C2D_SOURCE_TO_EDGE",
        DeliverSubject: "GW123._sync.c2d",
    },
}

Create the equivalent independent topology for GW456.

At this point both leaf relationships work and the only permission namespace shared between the two leaf credentials is $JSC.R.*.

Now, from the GW456 leaf, create a local subscription to:

$JSC.R.*

Restart GW123 (or otherwise cause the GW123 source/mirror to reconnect and reset its pre-created sourcing consumer).

GW123 issues the expected, tightly scoped request:

$JS.cloud.API.CONSUMER.RESET.GW123_C2D_STREAM.GW123_C2D_SOURCE_TO_EDGE

but its reply is emitted as:

$JSC.R.<uid>

and the subscription on GW456 receives that reply.

GW456 does not have access to the GW123 stream, consumer, JetStream domain, sync subjects, or RESET request subject; it receives the response solely because $JSC.R.* is the common reply namespace required by the sourcing protocol.

For a bidirectional source/mirror setup where the leaf credential also needs $JSC.R.* publish permission, a client on GW456 can additionally publish:

$JSC.R.test

and that publication is allowed into the same namespace used for real durable-source RESET replies.

Enabling the NATS 2.14 ACK/FC v2 subject format does not change this behavior because those subjects are separate from $JSC.R.

The key issue isn't that $JSC.R exists; it is that the opaque UID is the only identifying component of the reply subject, leaving no subject component that can be used to authorize one source/mirror relationship without authorizing every other $JSC.R reply in the account.

Submission acknowledgment

• I am a human being writing in my own words and not an AI agent. I will not use an AI agent to communicate on my behalf in this issue, either directly or via copy-paste.

[shaunco]

Turns out this isn't just for BYO... even before BYO this was a problem for leafnodes that needed to call CONSUMER.CREATE because the create response also comes back on JSC.R.<random uid>.

A potentially easier fix for this is to copy the existing allow_responses dynamic response publish authorization but on the requester reply-subscription side with something like allow_requestor_reply_subscriptions. Source/Mirror needs the symmetric dynamic requester reply-subscription authorization, or a scoped reply subject containing domain/account/source identity.

[ayushk-1801]

Hi, I’d like to work on this issue. I’ve started looking into the $JSC.R.* reply isolation problem and can reproduce the behavior locally. If this is available, could you assign it to me?