Could not sync KV data across domain

[kannanvr]

Hi All,
We want to have a setup like HUB with 2 leaf nodes.
Each leaf node will have the KV bucket. Now I want to sync between 2 KV bucket. Idea is when 1 leaf node goes down. another leaf node KV bucket will server the data. Thats the reason we want to sync between 2 different domains (Leaf node) bucket.
Following is my HUB and Leaf configuration

hub.conf
port: 4222

leafnodes {
    port: 7422
}

jetstream {
    domain: hub
}


leaf1.conf
port: 4111

leafnodes {
    remotes = [{ url: "nats://localhost" }]
}

jetstream {
    domain: leaf1
}
leaf2.conf
port: 4000

leafnodes {
    remotes = [{ url: "nats://localhost" }]
}

jetstream {
    domain: leaf2
}

I have created the buckets on leaf node as below.

nats kv add test -s nats://127.0.0.1:4111
nats kv add test -s nats://127.0.0.1:4000

created the stream with the below file
leaf1-stream.json

{
    "name": "KV_bucket-name",
    "subjects": [
      "$KV.bucket-name.local-subject.>"
    ],
    "retention": "limits",
    "max_consumers": -1,
    "max_msgs_per_subject": 1,
    "max_msgs": -1,
    "max_bytes": -1,
    "max_age": 0,
    "max_msg_size": -1,
    "storage": "file",
    "discard": "new",
    "num_replicas": 1,
    "duplicate_window": 120000000000,
    "sealed": false,
    "deny_delete": true,
    "deny_purge": false,
    "allow_rollup_hdrs": true,
    "allow_direct": true,
    "mirror_direct": false,
    "sources": [
      {
        "name": "KV_test",
        "external": {
          "api": "$JS.leaf2.API",
          "deliver": ""
        },
        "subject_transforms": [
          {
            "src": "$KV.bucket-name.local-subject.>",
            "dest": "$KV.bucket-name.sourced-subject.>"
          }
        ]
      }
    ]
  }

nats stream add --config leaf1-stream.json -s nats://127.0.0.1:4111

Following is the leaf2-stream.json

 {
    "name": "KV_bucket-name",
    "subjects": [
      "$KV.bucket-name.local-subject.>"
    ],
    "retention": "limits",
    "max_consumers": -1,
    "max_msgs_per_subject": 1,
    "max_msgs": -1,
    "max_bytes": -1,
    "max_age": 0,
    "max_msg_size": -1,
    "storage": "file",
    "discard": "new",
    "num_replicas": 1,
    "duplicate_window": 120000000000,
    "sealed": false,
    "deny_delete": true,
    "deny_purge": false,
    "allow_rollup_hdrs": true,
    "allow_direct": true,
    "mirror_direct": false,
    "sources": [
      {
        "name": "KV_test",
        "external": {
          "api": "$JS.leaf1.API",
          "deliver": ""
        },
        "subject_transforms": [
          {
            "src": "$KV.bucket-name.local-subject.>",
            "dest": "$KV.bucket-name.sourced-subject.>"
          }
        ]
      }
    ]
  }

nats stream add --config leaf2-stream.json -s nats://127.0.0.1:4000

But when i put the data on bucket test on leaf1 , it is not reflecting to leaf2 and viceversa

I have tried debugging a lot. Please help us to setup the KV data sync across domain

[kannanvr]

I have observed the following error whenever the new data is being shared between leaf