Duplicate messages on a leafnode cluster connected to a supercluster.

[smolp]

I'm experiencing a weird behavior on my leafnode cluster with messages duplicated many times. I'm tracking the sequence numbers between messages and when I'm experiencing this issue it looks like this:
9867,9866,9867,9867,9867,9867,9868,9869,9870,9871,9872,9873,9874,9875,9876,9877,9878,9879,9868,9869,9870,9871,9880,9872,9873,9881,9882,9874,... while it should be more like this: 13702,13703,13704,13705,13706,13707,13708,13709,13710,13711,13712,13713,13714,13715,13716,13717,13718,..., i.e. the stream of messages should be sequential with no duplicates.

My current setup is:

Cluster 1 & 2 in EU, connected via gateways, it's a super cluster, 3 nodes each.
Cluster 3 in Asia, it's a leafnode cluster connected to 1 & 2 with 3 nodes.
We use 2.9.24 version of the nats server.

We connect cluster 3 to 1 & 2 like this:

leafnodes:
  enabled: true
  remotes:
    - urls:
        - "nats-leaf://nats-cluster-1.nats.svc.clusterset.local:7422"
        - "nats-leaf://nats-cluster-2.nats.svc.clusterset.local:7422"
      denyImports: ["topic.b"]
      denyExports: ["topic.a"]

The weird things about this issue:

1. If I can connect to only 1 of the nodes for testing, it only happens on 2 out of 3 nodes on Cluster 3. For some reason, I can connect to one of the nodes and it will work correctly, while it won't work on the other 2. When I scale nats back to 1 replica, it doesn't happen.
2. The issue only persists in internal cluster communication, so if I have an application app1 working on Cluster 3 and if I ask for data from an application working on Cluster 3, I'm getting duplicates. If an application app2 working on Cluster 1 or 2 asks for the same data, it isn't receiving any duplicates. There's no such problem on the supercluster, internal communication there is working fine.
3. I can mitigate this issue by expanding denyImports list on the topic I don't want duplicates on. The funny thing about this one is that there is no application on Cluster 1 and Cluster 2 that is publishing messages on those topics. Looks like the leafnode replicas are somehow duplicating the data and making it circle around(?).

This may be a continuation of my last question. I'm not using accounts and authorization blocks are very similar to that question.

Am I experiencing a bug of sorts or do I have something that can cause this in my setup?

----More about the setup---
So let me be clear, the setup on Cluster-1 & Cluster-2 is working fine for months now. The only problems we have are with this addition of the leafnode cluster.
The typical scenario is that user_c subscribes to a public topic like some2.topic2, and public user publishes messages to that topic.
We assume that some1.topic1.> is an internal topic within both clusters and isn't imported/exported anywhere, while some2.topic1.> can be used by both but the only application that publishes messages is on Cluster-3 with a single replica.
Then we get duplicates on some2.topic1.> if we don't specify it in a deny_import list.
It also happens when one application running on user_c publishes messages for another application running on user_c on some topic user_c.some.topic.updates. There's only one replica for each application publishing on user_c.

The authorization block for Cluster-3 looks like this:

authorization {
  default_permissions = {
    publish = ["*.topic1.>"]
    subscribe = ["*.topic1.>"]
  }

  ADMIN = {
    publish = ">"
    subscribe = ">"
  }

  USER_C = {
    subscribe = ["user_c.>", "some1.topic1.>", "some2.topic1.>"]
    publish = ["user_c.>", "some1.topic1.>", "some2.topic1.>"]
  }

  users = [
    {user: admin, password: $ADMIN_PASS, permissions: $ADMIN}
    {user: user_c, password: $USERC_PASS, permissions: $USER_C}
    {user: public, password: public}
  ]
}

As for the leafnode block:

leafnodes {
  remotes: [
    {
      urls: [
              "nats-leaf://nats-cluster-1.nats.svc.clusterset.local:7422",
              "nats-leaf://nats-cluster-2.nats.svc.clusterset.local:7422"
      ],
      deny_import: ["some1.topic1.>"]
      deny_export: ["some1.topic1.>"]
    }
  ]
}

To fix the issue we can make a new deny_import list:

deny_import: ["user_c.some.topic.updates", "some1.topic1.>", "some2.topic1.>"]

As for the Cluster-1 & Cluster-2, the authorization block is identical for both of them and looks like this:

authorization {
  default_permissions = {
    publish = ["*.topic1.>"]
    subscribe = ["*.topic1.>"]
  }

  ADMIN = {
    publish = ">"
    subscribe = ">"
  }

  USER_A = {
    subscribe = ["user_a.>", "some1.topic1.>", "some2.topic1.>", "some3.topic1.>"]
    publish = ["user_a.>", "some1.topic1.>", "some2.topic1.>", "some3.topic1.>"]
  }

  USER_B = {
    subscribe = ["user_b.>",  "some1.topic1.>", "some4.topic1.>"]
    publish = ["user_b.>",  "some1.topic1.>", "some4.topic1.>"]
  }

  ANALYTICS = {
    subscribe = ["*.some.other.topic.>", "analytics.>"]
    publish = ["*.some.other.topic.*", "analytics.>"]
  }

  users = [
    {user: admin, password: $ADMIN_PASS, permissions: $ADMIN}
    {user: user_a, password: $USER_A_PASS, permissions: $USER_A}
    {user: user_b, password: $USER_B_PASS, permissions: $USER_B}
    {user: analytics, password: $ANALYTICS_PASS, permissions: $ANALYTICS}
    {user: public, password: public}
  ]
}

The only user that can somehow access the data on user_c.> from cluster 1 & 2 is analytics, but it can access the data only on very specific topics (not the one that is having problems) and is very restricted.
Of course there are no applications running on admin users and there are no users publishing the messages we are interested in on some2.topic1.> on cluster 1 & 2 plus there is only one application publishing on the topic we are interested in, and it's located on Cluster 3.
user_a and user_b use data from all clusters and work fine.

[derekcollison]

We would need to see your complete setup. Its possible to create a loop using account imports and exports. Normal mis-configuration loops are usually caught by the system when the LN's are established.

[smolp]

I added a More about the setup block in the question. Let me know if you need more information.

[derekcollison]

A Leafnode bridges NATS systems. A Supercluster is a single system, so the LN connections from cluster 3 in Asia should only connect once to the supercluster, not to each cluster 1 & 2.

[smolp]

I see. So in the leafnode configuration should I specify only a single address like this:

  remotes: [
    {
      urls: [
              "nats-leaf://nats-cluster-1.nats.svc.clusterset.local:7422",
      ],

? Does it matter which cluster is it from? Does a supercluster have a single address that I can use?

[derekcollison]

Yes correct. For superclusters we recommend cname DNS for clusters and a geo-aware global DNS for the whole thing which is what we do with Synadia Cloud.