JetStream sourced streams occasionally duplicate messages with identical Nats-Stream-Source

[superspartak]

Observed behavior

We are seeing duplicate messages in a JetStream stream that is sourced by multiple remote streams, even though the source metadata indicates the message originated from the same source stream sequence.

                    ┌─────────────────────────┐
                    │   Central Hub Cluster   │
                    │                         │
                    │                         │
                    │   Stream: ORDERS        │
                    │   (sources all remotes) │
                    └───────────▲─────────────┘
                                │
               Leafnode links   │  JetStream sourcing
                                │
    ┌───────────────────────────┼───────────────────────────┐
    │                           │                           │
┌───┴─────────────┐     ┌───────┴─────────────┐     ┌───────┴─────────────┐
│ Remote Cluster A│     │ Remote Cluster B    │     │ Remote Cluster C    │
│ JetStream       │     │ JetStream           │     │ JetStream           │
│ domain=H111     │     │ domain=H222         │     │ domain=...          │
│                 │     │                     │     │                     │
│ Stream: H111  │     │ Stream: H222      │     │ Stream: ...      │
│ subject: ...    │     │ subject: ...        │     │ subject: ...        │
└─────────────────┘     └─────────────────────┘     └─────────────────────┘

Specifically, the Nats-Stream-Source header is identical across the duplicates, which suggests the message is not being republished on the source, but is instead being sourced into the hub stream multiple times.

Output of nats stream view:

[#20401] Received JetStream message: stream: ORDERS        seq 1116258984
subject: H111.SALE
time: 2026-02-04 14:07:04
Nats-Stream-Source: **H111:EQ61h9KL 26344781 > > EXPORT.SALE**
...
[#20409] Received JetStream message: stream: ORDERS        seq 1116258997
subject: H111.SALE
time: 2026-02-04 14:07:06
Nats-Stream-Source: **H111:EQ61h9KL 26344781 > > EXPORT.SALE**

I also checked the stream in the source cluster. The message was indeed published once.

Going through the logs, I observed the same message being duplicated / sourced into the stream as many as 100 times.

Expected behavior

A single JetStream message should be sourced exactly once into the destination stream.

Server and client version

Central hub server: nats-server: v2.12.1
Clusters (sources): varies among v2.12.0 and 2.12.1

Host environment

Central hub server: 32 Cores, 128 GB RAM, Alpine Linux 3.17.9
Clusters (sources): 4 Cores, 16GB RAM, Alpine Linux 3.12.12

Steps to reproduce

Unfortunately, we are yet to discover any specific trigger at the moment.