[bug]: `SyncUniverse` RPC's `sync_mode=1` option not working reliably

[ZZiigguurraatt]

I'm trying to run https://github.com/lightninglabs/tapdvalidation with lightninglabs/lightning-terminal#987. I previously had to add --taproot-assets.proofcourieraddr=universerpc://alice:8443 as a workaround to #1483 to get bob to sync universes with alice. However, I'm now getting the following issue when george is trying to use SyncUniverse to get proofs from bob:

controller-1  | alice funding a f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea channel with bob
controller-1  | alice connecting to bob
controller-1  | syncing the universe of bob to alice (172.99.0.4:8443)
controller-1  | done syncing universe
controller-1  | mining 10 blocks
controller-1  | bob funding a normal sats channel with charlie
controller-1  | bob connecting to charlie
controller-1  | mining 10 blocks
controller-1  | mining 10 blocks
controller-1  | charlie funding a normal sats channel with dave
controller-1  | charlie connecting to dave
controller-1  | mining 10 blocks
controller-1  | mining 10 blocks
controller-1  | dave funding a normal sats channel with edward
controller-1  | dave connecting to edward
controller-1  | mining 10 blocks
controller-1  | mining 10 blocks
controller-1  | alice sending 500000000 of f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea to bob on chain
controller-1  | syncing the universe of bob to alice (172.99.0.4:8443)
controller-1  | mining 10 blocks
controller-1  | bob funding a f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea channel with george
controller-1  | bob connecting to george
controller-1  | syncing the universe of george to bob (172.99.0.5:8443)
controller-1  | unknown
controller-1  | ERROR MESSAGE: unable to sync universe: unable to register proofs: unable to register proofs: unable to batch verify issuance proofs: unable to fetch previous asset snapshot: unable to fetch previous proof: no universe proof found, id=(universe.Identifier) issuance-308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb
controller-1  | , leaf_key=(universe.BaseLeafKey) {
controller-1  |  OutPoint: (wire.OutPoint) 0b5642f0c10208bd540e2bf251a90eee2e0be3a9f26011e58087266c2b34dc69:0,
controller-1  |  ScriptKey: (*asset.ScriptKey)(0xc0047e08f0)({
controller-1  |   PubKey: (*secp256k1.PublicKey)(0xc0040b32c0)({
controller-1  |    x: (secp256k1.FieldVal) 31254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e,
controller-1  |    y: (secp256k1.FieldVal) 73076bd12d5f1967915e7e92e883587b63d1f2036d581701e5104c8e9dd22fb8
controller-1  |   }),
controller-1  |   TweakedScriptKey: (*asset.TweakedScriptKey)(<nil>)
controller-1  |  })
controller-1  | }
controller-1  | , new_script_key=0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6
controller-1  | syncing failed, trying again
controller-1  | Traceback (most recent call last):
controller-1  |   File "/mini_META/scripts/init_network", line 149, in <module>
controller-1  |     OpenTaprootAssetChannel(funder='bob',     peer='george', AssetID=TheAssetID, AssetCapacity=200000000)
controller-1  |   File "/mini_META/scripts/mini_META_lib.py", line 549, in OpenTaprootAssetChannel
controller-1  |     litd_node_objects[peer]['tapd'].sync_universe(universe_host=universe_host, sync_mode=1)
controller-1  |   File "/usr/local/lib/python3.10/dist-packages/lndgrpc/errors.py", line 50, in wrapper
controller-1  |     raise exc
controller-1  |   File "/usr/local/lib/python3.10/dist-packages/lndgrpc/errors.py", line 26, in wrapper
controller-1  |     return fnc(*args, **kwargs)
controller-1  |   File "/usr/local/lib/python3.10/dist-packages/lndgrpc/universe.py", line 20, in sync_universe
controller-1  |     response = self.get_universe_stub().SyncUniverse(request)
controller-1  |   File "/usr/local/lib/python3.10/dist-packages/grpc/_channel.py", line 1181, in __call__
controller-1  |     return _end_unary_response_blocking(state, call, False, None)
controller-1  |   File "/usr/local/lib/python3.10/dist-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
controller-1  |     raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
controller-1  | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
controller-1  | 	status = StatusCode.UNKNOWN
controller-1  | 	details = "unable to sync universe: unable to register proofs: unable to register proofs: unable to batch verify issuance proofs: unable to fetch previous asset snapshot: unable to fetch previous proof: no universe proof found, id=(universe.Identifier) issuance-308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb
controller-1  | , leaf_key=(universe.BaseLeafKey) {
controller-1  |  OutPoint: (wire.OutPoint) 0b5642f0c10208bd540e2bf251a90eee2e0be3a9f26011e58087266c2b34dc69:0,
controller-1  |  ScriptKey: (*asset.ScriptKey)(0xc0047e08f0)({
controller-1  |   PubKey: (*secp256k1.PublicKey)(0xc0040b32c0)({
controller-1  |    x: (secp256k1.FieldVal) 31254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e,
controller-1  |    y: (secp256k1.FieldVal) 73076bd12d5f1967915e7e92e883587b63d1f2036d581701e5104c8e9dd22fb8
controller-1  |   }),
controller-1  |   TweakedScriptKey: (*asset.TweakedScriptKey)(<nil>)
controller-1  |  })
controller-1  | }
controller-1  | , new_script_key=0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6"
controller-1  | 	debug_error_string = "UNKNOWN:Error received from peer ipv4:172.99.0.10:8443 {grpc_message:"unable to sync universe: unable to register proofs: unable to register proofs: unable to batch verify issuance proofs: unable to fetch previous asset snapshot: unable to fetch previous proof: no universe proof found, id=(universe.Identifier) issuance-308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb\n, leaf_key=(universe.BaseLeafKey) {\n OutPoint: (wire.OutPoint) 0b5642f0c10208bd540e2bf251a90eee2e0be3a9f26011e58087266c2b34dc69:0,\n ScriptKey: (*asset.ScriptKey)(0xc0047e08f0)({\n  PubKey: (*secp256k1.PublicKey)(0xc0040b32c0)({\n   x: (secp256k1.FieldVal) 31254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e,\n   y: (secp256k1.FieldVal) 73076bd12d5f1967915e7e92e883587b63d1f2036d581701e5104c8e9dd22fb8\n  }),\n  TweakedScriptKey: (*asset.TweakedScriptKey)(<nil>)\n })\n}\n, new_script_key=0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6", grpc_status:2, created_time:"2025-04-24T05:26:49.450618885-04:00"}"
controller-1  | >

This error did not come up in previous versions of litd/tapd/lnd.

I'm struggling to even find the error message in the taproot-assets source code.

If I jump into george and try to do the same thing with tapcli I get the same kind of error

1280e18e0083:/$ tapcli universe sync --universe_host=172.99.0.5:8443 --proof_type='transfer' --asset_id=f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea
[tapcli] rpc error: code = Unknown desc = unable to sync universe: unable to register proofs: unable to register proofs: unable to batch verify issuance proofs: unable to fetch previous asset snapshot: unable to fetch previous proof: no universe proof found, id=(universe.Identifier) issuance-308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb
, leaf_key=(universe.BaseLeafKey) {
 OutPoint: (wire.OutPoint) 0b5642f0c10208bd540e2bf251a90eee2e0be3a9f26011e58087266c2b34dc69:0,
 ScriptKey: (*asset.ScriptKey)(0xc0044a9340)({
  PubKey: (*secp256k1.PublicKey)(0xc00406efa0)({
   x: (secp256k1.FieldVal) 31254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e,
   y: (secp256k1.FieldVal) 73076bd12d5f1967915e7e92e883587b63d1f2036d581701e5104c8e9dd22fb8
  }),
  TweakedScriptKey: (*asset.TweakedScriptKey)(<nil>)
 })
}
, new_script_key=0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6
1280e18e0083:/$ 

If I then repeat the command, but change the proof_type

1280e18e0083:/$ tapcli universe sync --universe_host=172.99.0.5:8443 --proof_type='issuance' --asset_id=f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea
{
    "synced_universes":  [
        {
            "old_asset_root":  {
                "id":  null,
                "mssmt_root":  null,
                "asset_name":  "",
                "amounts_by_asset_id":  {}
            },
            "new_asset_root":  {
                "id":  {
                    "group_key":  "3d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b",
                    "proof_type":  "PROOF_TYPE_ISSUANCE"
                },
                "mssmt_root":  {
                    "root_hash":  "1e651800389c72ea8fb3061393feeb68e48c087d98c748474b0c1d6e91d88254",
                    "root_sum":  "210000000000000000"
                },
                "asset_name":  "",
                "amounts_by_asset_id":  {}
            },
            "new_asset_leaves":  [
                {
                    "asset":  {
                        "version":  "ASSET_VERSION_V0",
                        "asset_genesis":  {
                            "genesis_point":  "65ebb74a1b62071059f9af1752d84c631cc25866a1f7ff0eb7d787dae9458378:1",
                            "name":  "Asset1",
                            "meta_hash":  "827c23a4b1a5f4f9a9a42e61aa7bf7bf7e7925b4723728b60ad1918d632e2b0a",
                            "asset_id":  "f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea",
                            "asset_type":  "NORMAL",
                            "output_index":  0
                        },
                        "amount":  "210000000000000000",
                        "lock_time":  0,
                        "relative_lock_time":  0,
                        "script_version":  0,
                        "script_key":  "0231254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e",
                        "script_key_is_local":  false,
                        "asset_group":  {
                            "raw_group_key":  "",
                            "tweaked_group_key":  "033d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b",
                            "asset_witness":  "",
                            "tapscript_root":  ""
                        },
                        "chain_anchor":  null,
                        "prev_witnesses":  [
                            {
                                "prev_id":  {
                                    "anchor_point":  "0000000000000000000000000000000000000000000000000000000000000000:0",
                                    "asset_id":  "0000000000000000000000000000000000000000000000000000000000000000",
                                    "script_key":  "000000000000000000000000000000000000000000000000000000000000000000",
                                    "amount":  "0"
                                },
                                "tx_witness":  [
                                    "2e7c5ed443304212fe58dc812feca8a138cd9fc8024528d52f968471fd98b6bc82b194b5e95485d9696918e9d3b34b272595d3e3533c9d1da91dd0d7edeaead6"
                                ],
                                "split_commitment":  null
                            }
                        ],
                        "is_spent":  false,
                        "lease_owner":  "",
                        "lease_expiry":  "0",
                        "is_burn":  false,
                        "script_key_declared_known":  false,
                        "script_key_has_script_path":  false,
                        "decimal_display":  {
                            "decimal_display":  3
                        }
                    },
                    "proof":  "544150500004000000000224788345e9da87d7b70efff7a16658c21c634cd85217aff9591007621b4ab7eb6500000001045000000030b4ac8c47daf10886d315dc10e9d59c08247806e5e36b039c9c4e9f525a862d5ca21bd02706feffa5071cb9649dedcabd2a4e267e1c6dd834bff580deedb9a1dde0030a68ffff7f200000000006f602000000000101788345e9da87d7b70efff7a16658c21c634cd85217aff9591007621b4ab7eb6501000000000000000002e803000000000000225120dfdcbb8913308efcc8b7d5e11651617806283946d4707a10432f603ccda2f98b39d0f505000000002251202319e97afd161b0bf065001f08b17e7e77eb21f55552deb5d98f6d88de69b9a202473044022025cc08de64ec5befa1050525519e440a471f158cb7d39f1d9711e245c35a2e0a022023733a5f4225d9a901ef5eefca1d7a871a31d04e0763da7cf3fc438d5fb70a210121036163b766cafc52242768b14d3a9faefc232826e2f12f90876469d03015f069050000000008220171eaeb07a84472336fb15157514800bf7a2f4eb801befdcc6035ceb7efa444f3000afd015c0001000250788345e9da87d7b70efff7a16658c21c634cd85217aff9591007621b4ab7eb650000000106417373657431827c23a4b1a5f4f9a9a42e61aa7bf7bf7e7925b4723728b60ad1918d632e2b0a00000000000401000609ff02ea11e32ad500000bad01ab01650000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000034201402e7c5ed443304212fe58dc812feca8a138cd9fc8024528d52f968471fd98b6bc82b194b5e95485d9696918e9d3b34b272595d3e3533c9d1da91dd0d7edeaead60e02000010210231254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e1121033d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b0c9f0004000000000221033fc90fbe54d4b529d90cfa5206282912d63ca45a5f9bcb973a43308e0464ce01037401490001000220308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb04220000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff022700010202220000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0d30012e0004000000010221024277c805bb4df0bb05abc9e33d80592395c55066768260aea201ee33efa6582c0503040101112000010102157b22646563696d616c5f646973706c6179223a337d0504000000031604000001241750788345e9da87d7b70efff7a16658c21c634cd85217aff9591007621b4ab7eb650000000106417373657431827c23a4b1a5f4f9a9a42e61aa7bf7bf7e7925b4723728b60ad1918d632e2b0a00000000001948000102022103a6553ff1aa8bb1dc91b35e1f8428f99e6dfc3a66e39945ad9c0c22fffec677ff0420776c54fefcec5c564d69e6d844ecb5bfe2b4ff729d4fee26171093a34f2e205c"
                }
            ]
        }
    ]
}
1280e18e0083:/$ 

it works.

Then, if I change the proof type back to transfer, it also works

``` 1280e18e0083:/$ tapcli universe sync --universe_host=172.99.0.5:8443 --proof_type='transfer' --asset_id=f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea { "synced_universes": [ { "old_asset_root": { "id": null, "mssmt_root": null, "asset_name": "", "amounts_by_asset_id": {} }, "new_asset_root": { "id": { "group_key": "3d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b", "proof_type": "PROOF_TYPE_TRANSFER" }, "mssmt_root": { "root_hash": "9a2e814691125f57cb9a30717130f6148a5940d6b7b72fc2b5567ab3e455d1c2", "root_sum": "1" }, "asset_name": "", "amounts_by_asset_id": {} }, "new_asset_leaves": [ { "asset": { "version": "ASSET_VERSION_V1", "asset_genesis": { "genesis_point": "65ebb74a1b62071059f9af1752d84c631cc25866a1f7ff0eb7d787dae9458378:1", "name": "Asset1", "meta_hash": "827c23a4b1a5f4f9a9a42e61aa7bf7bf7e7925b4723728b60ad1918d632e2b0a", "asset_id": "f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea", "asset_type": "NORMAL", "output_index": 0 }, "amount": "200000000", "lock_time": 0, "relative_lock_time": 0, "script_version": 0, "script_key": "0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6", "script_key_is_local": false, "asset_group": { "raw_group_key": "", "tweaked_group_key": "033d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b", "asset_witness": "", "tapscript_root": "" }, "chain_anchor": null, "prev_witnesses": [ { "prev_id": { "anchor_point": "0000000000000000000000000000000000000000000000000000000000000000:0", "asset_id": "0000000000000000000000000000000000000000000000000000000000000000", "script_key": "000000000000000000000000000000000000000000000000000000000000000000", "amount": "0" }, "tx_witness": [], "split_commitment": { "root_asset": { "version": "ASSET_VERSION_V0", "asset_genesis": { "genesis_point": "65ebb74a1b62071059f9af1752d84c631cc25866a1f7ff0eb7d787dae9458378:1", "name": "Asset1", "meta_hash": "827c23a4b1a5f4f9a9a42e61aa7bf7bf7e7925b4723728b60ad1918d632e2b0a", "asset_id": "f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea", "asset_type": "NORMAL", "output_index": 0 }, "amount": "209999999800000000", "lock_time": 0, "relative_lock_time": 0, "script_version": 0, "script_key": "02a48d9550a03394dc405fda546943ddcc88fb76f4402e885da852a4fbd3c7ccb3", "script_key_is_local": false, "asset_group": { "raw_group_key": "", "tweaked_group_key": "033d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b", "asset_witness": "", "tapscript_root": "" }, "chain_anchor": null, "prev_witnesses": [ { "prev_id": { "anchor_point": "0b5642f0c10208bd540e2bf251a90eee2e0be3a9f26011e58087266c2b34dc69:0", "asset_id": "f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea", "script_key": "0231254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e", "amount": "0" }, "tx_witness": [ "97b6c8effa238d66b82dc4a8b5a7866248badfceb6d79a67c34a456926939b369079325540a3ebbb8556b273f9715f6844d371500e9f218d74d822e9006745b0" ], "split_commitment": null } ], "is_spent": false, "lease_owner": "", "lease_expiry": "0", "is_burn": false, "script_key_declared_known": false, "script_key_has_script_path": false, "decimal_display": { "decimal_display": 3 } } } } ], "is_spent": false, "lease_owner": "", "lease_expiry": "0", "is_burn": false, "script_key_declared_known": false, "script_key_has_script_path": false, "decimal_display": { "decimal_display": 3 } }, "proof": "54415050000400000000022469dc342b6c268780e51160f2a9e30b2eee0ea951f22b0e54bd0802c1f042560b000000000450000000306d55460380ecc308b71ea2312686a0a5498b2999121df38849d5a88edaf0665b057a12595034eda7c71934a0b0b19b12b3f1bf9073c85fd5e95a6b168fab8e62e2030a68ffff7f200300000006fd01630200000000010269dc342b6c268780e51160f2a9e30b2eee0ea951f22b0e54bd0802c1f042560b00000000000000000069dc342b6c268780e51160f2a9e30b2eee0ea951f22b0e54bd0802c1f042560b01000000000000000003a086010000000000225120eadcd3ee51d941d2de15b3d75b7e07c25291a7968150ed6dbb4f1b92567ae2bee803000000000000225120a02c5ea371aa5ff8c8c9a5e5f3bee8128d990511ef8b80ef1af7a7f546e2ba4ea83ff40500000000225120370fb93baeac7f00e6407783d6c80ce7105dc1f9cb4a78ceacf6b98e0a2cce740140b5cd5b6a4ac7b24d6306b5a4f236f7f0e4f3f051bd6154a443c784f40525d1572936d58407d7181b53b7a49d0f8b2c36317779a51cbe905f20e661fa1432970e014058580ec842921703b54531df646a38f94a70ced77bcffbd18c4272416cf8458b6e9a650a33f349a04b34405a1d41ea0338225fc3d87277568401c896c1473c6f0000000008220113b19c129cdcd2d2c879522e0aeec4e3a94a5821b3a7334247192f4f6a02ef98000afd02f00001010250788345e9da87d7b70efff7a16658c21c634cd85217aff9591007621b4ab7eb650000000106417373657431827c23a4b1a5f4f9a9a42e61aa7bf7bf7e7925b4723728b60ad1918d632e2b0a00000000000401000605fe0bebc2000bfd024301fd023f0165000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000005fd01d44a000159258c0e4b5473185bbd0bc880b81d068e8df2daf2ec6aca35d451bc7f29853502ea11e31ee93e00ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff7ffd01860001000250788345e9da87d7b70efff7a16658c21c634cd85217aff9591007621b4ab7eb650000000106417373657431827c23a4b1a5f4f9a9a42e61aa7bf7bf7e7925b4723728b60ad1918d632e2b0a00000000000401000609ff02ea11e31ee93e000bad01ab016569dc342b6c268780e51160f2a9e30b2eee0ea951f22b0e54bd0802c1f042560b00000000f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea0231254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e0342014097b6c8effa238d66b82dc4a8b5a7866248badfceb6d79a67c34a456926939b369079325540a3ebbb8556b273f9715f6844d371500e9f218d74d822e9006745b00d2812e715439967f17d8354a1ae6ae1ca0ec2ad43a51f8597258d54012acecdb00f02ea11e32ad500000e020000102102a48d9550a03394dc405fda546943ddcc88fb76f4402e885da852a4fbd3c7ccb31121033d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b0e02000010210250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e61121033d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b0c9f000400000000022102fe1f54b5dd49c28ea64ccf34f22fc8696e6ed21ab10442bbf9064a963aadcf6a037401490001010220308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb04220000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff022700010202220000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0df802c7000400000001022103b32611d2fb23f735618811e0909a6995fae2e470ac23a400e6fded3256bcc105039c01710001000220308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb044a0001d7e65f4fa206b96348aaec35086dc9cbdd57135cddbdc4e64621057fde1e854502ea11e31ee93e00ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffef022700010202220000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff2e00040000000202210230713f278def95d2cec0a835eeab98d6a880b2e8dcc459f493be7bb9602ae03705030401010f9f000400000001022103b32611d2fb23f735618811e0909a6995fae2e470ac23a400e6fded3256bcc105037401490001000220308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb04220000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff022700010202220000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff16040000012e" } ] } ] } 1280e18e0083:/$

</details>






I've also tried to get george to sync the universe with alice, but that results in the same error:


<details>

controller-1 | alice funding a 9e23bd4b7b3704e0cecfdb292d2d7f021336907d080d6fcf31bddb20642b8add channel with bob
controller-1 | alice connecting to bob
controller-1 | syncing the universe of bob to alice (172.99.0.4:8443)
controller-1 | done syncing universe
controller-1 | mining 10 blocks
controller-1 | bob funding a normal sats channel with charlie
controller-1 | bob connecting to charlie
controller-1 | mining 10 blocks
controller-1 | mining 10 blocks
controller-1 | charlie funding a normal sats channel with dave
controller-1 | charlie connecting to dave
controller-1 | mining 10 blocks
controller-1 | mining 10 blocks
controller-1 | dave funding a normal sats channel with edward
controller-1 | dave connecting to edward
controller-1 | mining 10 blocks
controller-1 | mining 10 blocks
controller-1 | alice sending 500000000 of 9e23bd4b7b3704e0cecfdb292d2d7f021336907d080d6fcf31bddb20642b8add to bob on chain
controller-1 | syncing the universe of bob to alice (172.99.0.4:8443)
controller-1 | mining 10 blocks
controller-1 | bob funding a 9e23bd4b7b3704e0cecfdb292d2d7f021336907d080d6fcf31bddb20642b8add channel with george
controller-1 | bob connecting to george
controller-1 | syncing the universe of george to alice (172.99.0.4:8443)
controller-1 | unknown
controller-1 | ERROR MESSAGE: unable to sync universe: unable to register proofs: unable to register proofs: unable to batch verify issuance proofs: unable to fetch previous asset snapshot: unable to fetch previous proof: no universe proof found, id=(universe.Identifier) issuance-708e8571b2ca36efc260100e9525a5c1fd165cc64a147d7ec604c1ac63e490c3
controller-1 | , leaf_key=(universe.BaseLeafKey) {
controller-1 | OutPoint: (wire.OutPoint) ec2daff7e26eb56b2940ab484431d2fca30442131d3117dfb938de91a63605ab:0,
controller-1 | ScriptKey: (*asset.ScriptKey)(0xc004c54300)({
controller-1 | PubKey: (*secp256k1.PublicKey)(0xc003eba0a0)({
controller-1 | x: (secp256k1.FieldVal) e9fa1cb15a00e37ae2c8f37b94fdf71ccad02b1ac5e7aa945ae676e3cc449d5d,
controller-1 | y: (secp256k1.FieldVal) 76ee13ef5ee1fae1e5cce7f0fedc228462bb50f020fec9608972fac92cf81cb2
controller-1 | }),
controller-1 | TweakedScriptKey: (*asset.TweakedScriptKey)()
controller-1 | })
controller-1 | }
controller-1 | , new_script_key=027a1023b5d70bb7c36cc7a16e6011c9be4667671555e88ab949cf8460acbea522
controller-1 | syncing failed, trying again
controller-1 | Traceback (most recent call last):
controller-1 | File "/mini_META/scripts/init_network", line 149, in
controller-1 | OpenTaprootAssetChannel(funder='bob', peer='george', AssetID=TheAssetID, AssetCapacity=200000000)
controller-1 | File "/mini_META/scripts/mini_META_lib.py", line 551, in OpenTaprootAssetChannel
controller-1 | litd_node_objects[peer]['tapd'].sync_universe(universe_host=universe_host, sync_mode=1)
controller-1 | File "/usr/local/lib/python3.10/dist-packages/lndgrpc/errors.py", line 50, in wrapper
controller-1 | raise exc
controller-1 | File "/usr/local/lib/python3.10/dist-packages/lndgrpc/errors.py", line 26, in wrapper
controller-1 | return fnc(*args, **kwargs)
controller-1 | File "/usr/local/lib/python3.10/dist-packages/lndgrpc/universe.py", line 20, in sync_universe
controller-1 | response = self.get_universe_stub().SyncUniverse(request)
controller-1 | File "/usr/local/lib/python3.10/dist-packages/grpc/_channel.py", line 1181, in call
controller-1 | return _end_unary_response_blocking(state, call, False, None)
controller-1 | File "/usr/local/lib/python3.10/dist-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
controller-1 | raise _InactiveRpcError(state) # pytype: disable=not-instantiable
controller-1 | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
controller-1 | status = StatusCode.UNKNOWN
controller-1 | details = "unable to sync universe: unable to register proofs: unable to register proofs: unable to batch verify issuance proofs: unable to fetch previous asset snapshot: unable to fetch previous proof: no universe proof found, id=(universe.Identifier) issuance-708e8571b2ca36efc260100e9525a5c1fd165cc64a147d7ec604c1ac63e490c3
controller-1 | , leaf_key=(universe.BaseLeafKey) {
controller-1 | OutPoint: (wire.OutPoint) ec2daff7e26eb56b2940ab484431d2fca30442131d3117dfb938de91a63605ab:0,
controller-1 | ScriptKey: (*asset.ScriptKey)(0xc004c54300)({
controller-1 | PubKey: (*secp256k1.PublicKey)(0xc003eba0a0)({
controller-1 | x: (secp256k1.FieldVal) e9fa1cb15a00e37ae2c8f37b94fdf71ccad02b1ac5e7aa945ae676e3cc449d5d,
controller-1 | y: (secp256k1.FieldVal) 76ee13ef5ee1fae1e5cce7f0fedc228462bb50f020fec9608972fac92cf81cb2
controller-1 | }),
controller-1 | TweakedScriptKey: (*asset.TweakedScriptKey)()
controller-1 | })
controller-1 | }
controller-1 | , new_script_key=027a1023b5d70bb7c36cc7a16e6011c9be4667671555e88ab949cf8460acbea522"
controller-1 | debug_error_string = "UNKNOWN:Error received from peer ipv4:172.99.0.10:8443 {created_time:"2025-04-24T05:36:15.574152077-04:00", grpc_status:2, grpc_message:"unable to sync universe: unable to register proofs: unable to register proofs: unable to batch verify issuance proofs: unable to fetch previous asset snapshot: unable to fetch previous proof: no universe proof found, id=(universe.Identifier) issuance-708e8571b2ca36efc260100e9525a5c1fd165cc64a147d7ec604c1ac63e490c3\n, leaf_key=(universe.BaseLeafKey) {\n OutPoint: (wire.OutPoint) ec2daff7e26eb56b2940ab484431d2fca30442131d3117dfb938de91a63605ab:0,\n ScriptKey: (*asset.ScriptKey)(0xc004c54300)({\n PubKey: (*secp256k1.PublicKey)(0xc003eba0a0)({\n x: (secp256k1.FieldVal) e9fa1cb15a00e37ae2c8f37b94fdf71ccad02b1ac5e7aa945ae676e3cc449d5d,\n y: (secp256k1.FieldVal) 76ee13ef5ee1fae1e5cce7f0fedc228462bb50f020fec9608972fac92cf81cb2\n }),\n TweakedScriptKey: (*asset.TweakedScriptKey)()\n })\n}\n, new_script_key=027a1023b5d70bb7c36cc7a16e6011c9be4667671555e88ab949cf8460acbea522"}"
controller-1 | >

</details>

[ZZiigguurraatt]

Here is my network topology

#         TA         SAT            SAT         SAT           TA
# alice <----> bob <----> charlie <----> dave <----> edward <----> frank
#               ^            ^             ^
#               |            |             |
#              T|A          T|A           T|A
#               |            |             |
#               v            v             v
#             goerge       henry         irina

[guggero]

This is expected. You can't sync transfer proofs without first knowing about the issuance of an asset.

[ZZiigguurraatt]

I take a look at the itests and we have Zane as the proof courier

https://github.com/lightninglabs/lightning-terminal/blob/72477225a8892b419e8903bf746a443920598cf7/itest/litd_custom_channels_test.go#L299-L311

and then we seem to sync dave, erin, fabia, yara to charlie

https://github.com/lightninglabs/lightning-terminal/blob/72477225a8892b419e8903bf746a443920598cf7/itest/litd_custom_channels_test.go#L388-L390

https://github.com/lightninglabs/lightning-terminal/blob/72477225a8892b419e8903bf746a443920598cf7/itest/assets_test.go#L371-L388

but I'm not sure why they sync universes with charlie instead of zane who is supposed to be the proof courier?

	// Charlie  --[assets]-->  Dave  --[sats]-->  Erin  --[assets]-->  Fabia
	//                          |
	//                          |
	//                       [assets]
	//                          |
	//                          v
	//                        Yara
	//

[ZZiigguurraatt]

This is expected. You can't sync transfer proofs without first knowing about the issuance of an asset.

Why can bob sync the universe of alice without requesting a transfer of the issuance proof first?

Why was it not required in older versions of tapd to transfer the issuance proof first?

[ZZiigguurraatt]

https://lightning.engineering/api-docs/api/taproot-assets/universe/sync-universe/#universerpcuniversesyncmode says

SYNC_FULL
A syncing mode that indicates that all asset proofs should be synced. This includes normal transfers as well.

why would I expect to need to do something else first to get all asset proofs?

[ZZiigguurraatt]

In my code I'm using sync_mode=1

[guggero]

By default, syncing proofs is "pull", not "pull and push".
Which means: If you sync a universe with another one, the syncer will fetch all proofs from the syncee, but not the other way around.

The exception is if you actually add a universe as a federation server before minting an asset, then the issuance proof will be pushed to the federation universe servers on minting.

In our itests we use Zane as the proof courier only. Meaning, it will not act as a full universe, only for facilitating the transfers.
And when a proof is pushed to a courier, it happens recursively through the whole proof chain, starting at the issuance. Meaning: The proof courier universe node doesn't have to be synced to anyone, it can just be a standalone tapd.

The reason we sync all the nodes to Charlie instead of Zane is because Charlie mints the asset. So it already has the issuance proofs the others need to know about to know how to receive the assets.

So tl;dr, what you'll want is:

• Configure one of the nodes as the proof courier, doesn't matter which one, but use the same courier address on all nodes
• Mint the asset on one node
• Sync issuance proofs to the other nodes
• Ignore transfer proofs, you don't need to handle those, that's part of the proof courier job

[ZZiigguurraatt]

By default, syncing proofs is "pull", not "pull and push". Which means: If you sync a universe with another one, the syncer will fetch all proofs from the syncee, but not the other way around.

Above when I did syncing the universe of george to bob (172.99.0.5:8443), george is the syncer, so I was expecting george to fetch all issuance and transfer proofs from bob since my sync_mode is SYNC_FULL. Could it be that george is receiving the transfer proofs first and as they are validated, they are rejected? Like I'm wondering if we should ensure when using SYNC_FULL that issuance proofs must all come in first?

The exception is if you actually add a universe as a federation server before minting an asset, then the issuance proof will be pushed to the federation universe servers on minting.

In our itests we use Zane as the proof courier only. Meaning, it will not act as a full universe, only for facilitating the transfers. And when a proof is pushed to a courier, it happens recursively through the whole proof chain, starting at the issuance. Meaning: The proof courier universe node doesn't have to be synced to anyone, it can just be a standalone tapd.

Is there a reason you don't make charlie the proof courier?

[guggero]

Like I'm wondering if we should ensure when using SYNC_FULL that issuance proofs must all come in first?

If you're doing a full sync, the issuance proofs are synced first.

But what a full sync does also depends on your sync configuration. See options like --universe.sync-all-assets or CLI commands like tapcli universe federation config --help (which maps to https://lightning.engineering/api-docs/api/taproot-assets/universe/set-federation-sync-config/)

Is there a reason you don't make charlie the proof courier?

Not a technical one. Just that we can be sure to exclude any side effects of a node using itself as the courier. And because it's easier to find out if the courier did its job if we can detect the proof in a different node's universe.

[ZZiigguurraatt]

If you're doing a full sync, the issuance proofs are synced first.

But what a full sync does also depends on your sync configuration. See options like --universe.sync-all-assets or CLI commands like tapcli universe federation config --help (which maps to https://lightning.engineering/api-docs/api/taproot-assets/universe/set-federation-sync-config/)

I do have --taproot-assets.universe.sync-all-assets set, so why would this be failing if I'm also trying to do a full sync?

[ZZiigguurraatt]

By default, syncing proofs is "pull", not "pull and push".

What is the non-default option I can set to do a push and pull?

[ZZiigguurraatt]

I think one confusion here is universerpc.ProofType (https://lightning.engineering/api-docs/api/taproot-assets/universe/sync-universe/#universerpcprooftype) vs universerpc.UniverseSyncMode (https://lightning.engineering/api-docs/api/taproot-assets/universe/sync-universe/#universerpcuniversesyncmode) as both have options that seem to conflict with each other. Which one takes precedence?

[ZZiigguurraatt]

As a workaround, I've tried the following sequence:

1. sync with sync_mode=0
2. sleep 10 seconds
3. sync with sync_mode=1

This works about 50% of the time. When it does fail, it always fails with the step where sync_mode=1 and not when sync_mode=0.