Skip to content

[bug]: SyncUniverse RPC's sync_mode=1 option not working reliably #1484

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ZZiigguurraatt opened this issue Apr 24, 2025 · 19 comments
Open
Labels
bug Something isn't working needs triage

Comments

@ZZiigguurraatt
Copy link

ZZiigguurraatt commented Apr 24, 2025

I'm trying to run https://github.com/lightninglabs/tapdvalidation with lightninglabs/lightning-terminal#987. I previously had to add --taproot-assets.proofcourieraddr=universerpc://alice:8443 as a workaround to #1483 to get bob to sync universes with alice. However, I'm now getting the following issue when george is trying to use SyncUniverse to get proofs from bob:

controller-1  | alice funding a f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea channel with bob
controller-1  | alice connecting to bob
controller-1  | syncing the universe of bob to alice (172.99.0.4:8443)
controller-1  | done syncing universe
controller-1  | mining 10 blocks
controller-1  | bob funding a normal sats channel with charlie
controller-1  | bob connecting to charlie
controller-1  | mining 10 blocks
controller-1  | mining 10 blocks
controller-1  | charlie funding a normal sats channel with dave
controller-1  | charlie connecting to dave
controller-1  | mining 10 blocks
controller-1  | mining 10 blocks
controller-1  | dave funding a normal sats channel with edward
controller-1  | dave connecting to edward
controller-1  | mining 10 blocks
controller-1  | mining 10 blocks
controller-1  | alice sending 500000000 of f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea to bob on chain
controller-1  | syncing the universe of bob to alice (172.99.0.4:8443)
controller-1  | mining 10 blocks
controller-1  | bob funding a f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea channel with george
controller-1  | bob connecting to george
controller-1  | syncing the universe of george to bob (172.99.0.5:8443)
controller-1  | unknown
controller-1  | ERROR MESSAGE: unable to sync universe: unable to register proofs: unable to register proofs: unable to batch verify issuance proofs: unable to fetch previous asset snapshot: unable to fetch previous proof: no universe proof found, id=(universe.Identifier) issuance-308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb
controller-1  | , leaf_key=(universe.BaseLeafKey) {
controller-1  |  OutPoint: (wire.OutPoint) 0b5642f0c10208bd540e2bf251a90eee2e0be3a9f26011e58087266c2b34dc69:0,
controller-1  |  ScriptKey: (*asset.ScriptKey)(0xc0047e08f0)({
controller-1  |   PubKey: (*secp256k1.PublicKey)(0xc0040b32c0)({
controller-1  |    x: (secp256k1.FieldVal) 31254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e,
controller-1  |    y: (secp256k1.FieldVal) 73076bd12d5f1967915e7e92e883587b63d1f2036d581701e5104c8e9dd22fb8
controller-1  |   }),
controller-1  |   TweakedScriptKey: (*asset.TweakedScriptKey)(<nil>)
controller-1  |  })
controller-1  | }
controller-1  | , new_script_key=0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6
controller-1  | syncing failed, trying again
controller-1  | Traceback (most recent call last):
controller-1  |   File "/mini_META/scripts/init_network", line 149, in <module>
controller-1  |     OpenTaprootAssetChannel(funder='bob',     peer='george', AssetID=TheAssetID, AssetCapacity=200000000)
controller-1  |   File "/mini_META/scripts/mini_META_lib.py", line 549, in OpenTaprootAssetChannel
controller-1  |     litd_node_objects[peer]['tapd'].sync_universe(universe_host=universe_host, sync_mode=1)
controller-1  |   File "/usr/local/lib/python3.10/dist-packages/lndgrpc/errors.py", line 50, in wrapper
controller-1  |     raise exc
controller-1  |   File "/usr/local/lib/python3.10/dist-packages/lndgrpc/errors.py", line 26, in wrapper
controller-1  |     return fnc(*args, **kwargs)
controller-1  |   File "/usr/local/lib/python3.10/dist-packages/lndgrpc/universe.py", line 20, in sync_universe
controller-1  |     response = self.get_universe_stub().SyncUniverse(request)
controller-1  |   File "/usr/local/lib/python3.10/dist-packages/grpc/_channel.py", line 1181, in __call__
controller-1  |     return _end_unary_response_blocking(state, call, False, None)
controller-1  |   File "/usr/local/lib/python3.10/dist-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
controller-1  |     raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
controller-1  | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
controller-1  | 	status = StatusCode.UNKNOWN
controller-1  | 	details = "unable to sync universe: unable to register proofs: unable to register proofs: unable to batch verify issuance proofs: unable to fetch previous asset snapshot: unable to fetch previous proof: no universe proof found, id=(universe.Identifier) issuance-308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb
controller-1  | , leaf_key=(universe.BaseLeafKey) {
controller-1  |  OutPoint: (wire.OutPoint) 0b5642f0c10208bd540e2bf251a90eee2e0be3a9f26011e58087266c2b34dc69:0,
controller-1  |  ScriptKey: (*asset.ScriptKey)(0xc0047e08f0)({
controller-1  |   PubKey: (*secp256k1.PublicKey)(0xc0040b32c0)({
controller-1  |    x: (secp256k1.FieldVal) 31254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e,
controller-1  |    y: (secp256k1.FieldVal) 73076bd12d5f1967915e7e92e883587b63d1f2036d581701e5104c8e9dd22fb8
controller-1  |   }),
controller-1  |   TweakedScriptKey: (*asset.TweakedScriptKey)(<nil>)
controller-1  |  })
controller-1  | }
controller-1  | , new_script_key=0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6"
controller-1  | 	debug_error_string = "UNKNOWN:Error received from peer ipv4:172.99.0.10:8443 {grpc_message:"unable to sync universe: unable to register proofs: unable to register proofs: unable to batch verify issuance proofs: unable to fetch previous asset snapshot: unable to fetch previous proof: no universe proof found, id=(universe.Identifier) issuance-308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb\n, leaf_key=(universe.BaseLeafKey) {\n OutPoint: (wire.OutPoint) 0b5642f0c10208bd540e2bf251a90eee2e0be3a9f26011e58087266c2b34dc69:0,\n ScriptKey: (*asset.ScriptKey)(0xc0047e08f0)({\n  PubKey: (*secp256k1.PublicKey)(0xc0040b32c0)({\n   x: (secp256k1.FieldVal) 31254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e,\n   y: (secp256k1.FieldVal) 73076bd12d5f1967915e7e92e883587b63d1f2036d581701e5104c8e9dd22fb8\n  }),\n  TweakedScriptKey: (*asset.TweakedScriptKey)(<nil>)\n })\n}\n, new_script_key=0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6", grpc_status:2, created_time:"2025-04-24T05:26:49.450618885-04:00"}"
controller-1  | >

This error did not come up in previous versions of litd/tapd/lnd.

I'm struggling to even find the error message in the taproot-assets source code.

If I jump into george and try to do the same thing with tapcli I get the same kind of error

1280e18e0083:/$ tapcli universe sync --universe_host=172.99.0.5:8443 --proof_type='transfer' --asset_id=f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea
[tapcli] rpc error: code = Unknown desc = unable to sync universe: unable to register proofs: unable to register proofs: unable to batch verify issuance proofs: unable to fetch previous asset snapshot: unable to fetch previous proof: no universe proof found, id=(universe.Identifier) issuance-308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb
, leaf_key=(universe.BaseLeafKey) {
 OutPoint: (wire.OutPoint) 0b5642f0c10208bd540e2bf251a90eee2e0be3a9f26011e58087266c2b34dc69:0,
 ScriptKey: (*asset.ScriptKey)(0xc0044a9340)({
  PubKey: (*secp256k1.PublicKey)(0xc00406efa0)({
   x: (secp256k1.FieldVal) 31254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e,
   y: (secp256k1.FieldVal) 73076bd12d5f1967915e7e92e883587b63d1f2036d581701e5104c8e9dd22fb8
  }),
  TweakedScriptKey: (*asset.TweakedScriptKey)(<nil>)
 })
}
, new_script_key=0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6
1280e18e0083:/$ 

If I then repeat the command, but change the proof_type

1280e18e0083:/$ tapcli universe sync --universe_host=172.99.0.5:8443 --proof_type='issuance' --asset_id=f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea
{
    "synced_universes":  [
        {
            "old_asset_root":  {
                "id":  null,
                "mssmt_root":  null,
                "asset_name":  "",
                "amounts_by_asset_id":  {}
            },
            "new_asset_root":  {
                "id":  {
                    "group_key":  "3d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b",
                    "proof_type":  "PROOF_TYPE_ISSUANCE"
                },
                "mssmt_root":  {
                    "root_hash":  "1e651800389c72ea8fb3061393feeb68e48c087d98c748474b0c1d6e91d88254",
                    "root_sum":  "210000000000000000"
                },
                "asset_name":  "",
                "amounts_by_asset_id":  {}
            },
            "new_asset_leaves":  [
                {
                    "asset":  {
                        "version":  "ASSET_VERSION_V0",
                        "asset_genesis":  {
                            "genesis_point":  "65ebb74a1b62071059f9af1752d84c631cc25866a1f7ff0eb7d787dae9458378:1",
                            "name":  "Asset1",
                            "meta_hash":  "827c23a4b1a5f4f9a9a42e61aa7bf7bf7e7925b4723728b60ad1918d632e2b0a",
                            "asset_id":  "f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea",
                            "asset_type":  "NORMAL",
                            "output_index":  0
                        },
                        "amount":  "210000000000000000",
                        "lock_time":  0,
                        "relative_lock_time":  0,
                        "script_version":  0,
                        "script_key":  "0231254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e",
                        "script_key_is_local":  false,
                        "asset_group":  {
                            "raw_group_key":  "",
                            "tweaked_group_key":  "033d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b",
                            "asset_witness":  "",
                            "tapscript_root":  ""
                        },
                        "chain_anchor":  null,
                        "prev_witnesses":  [
                            {
                                "prev_id":  {
                                    "anchor_point":  "0000000000000000000000000000000000000000000000000000000000000000:0",
                                    "asset_id":  "0000000000000000000000000000000000000000000000000000000000000000",
                                    "script_key":  "000000000000000000000000000000000000000000000000000000000000000000",
                                    "amount":  "0"
                                },
                                "tx_witness":  [
                                    "2e7c5ed443304212fe58dc812feca8a138cd9fc8024528d52f968471fd98b6bc82b194b5e95485d9696918e9d3b34b272595d3e3533c9d1da91dd0d7edeaead6"
                                ],
                                "split_commitment":  null
                            }
                        ],
                        "is_spent":  false,
                        "lease_owner":  "",
                        "lease_expiry":  "0",
                        "is_burn":  false,
                        "script_key_declared_known":  false,
                        "script_key_has_script_path":  false,
                        "decimal_display":  {
                            "decimal_display":  3
                        }
                    },
                    "proof":  "544150500004000000000224788345e9da87d7b70efff7a16658c21c634cd85217aff9591007621b4ab7eb6500000001045000000030b4ac8c47daf10886d315dc10e9d59c08247806e5e36b039c9c4e9f525a862d5ca21bd02706feffa5071cb9649dedcabd2a4e267e1c6dd834bff580deedb9a1dde0030a68ffff7f200000000006f602000000000101788345e9da87d7b70efff7a16658c21c634cd85217aff9591007621b4ab7eb6501000000000000000002e803000000000000225120dfdcbb8913308efcc8b7d5e11651617806283946d4707a10432f603ccda2f98b39d0f505000000002251202319e97afd161b0bf065001f08b17e7e77eb21f55552deb5d98f6d88de69b9a202473044022025cc08de64ec5befa1050525519e440a471f158cb7d39f1d9711e245c35a2e0a022023733a5f4225d9a901ef5eefca1d7a871a31d04e0763da7cf3fc438d5fb70a210121036163b766cafc52242768b14d3a9faefc232826e2f12f90876469d03015f069050000000008220171eaeb07a84472336fb15157514800bf7a2f4eb801befdcc6035ceb7efa444f3000afd015c0001000250788345e9da87d7b70efff7a16658c21c634cd85217aff9591007621b4ab7eb650000000106417373657431827c23a4b1a5f4f9a9a42e61aa7bf7bf7e7925b4723728b60ad1918d632e2b0a00000000000401000609ff02ea11e32ad500000bad01ab01650000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000034201402e7c5ed443304212fe58dc812feca8a138cd9fc8024528d52f968471fd98b6bc82b194b5e95485d9696918e9d3b34b272595d3e3533c9d1da91dd0d7edeaead60e02000010210231254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e1121033d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b0c9f0004000000000221033fc90fbe54d4b529d90cfa5206282912d63ca45a5f9bcb973a43308e0464ce01037401490001000220308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb04220000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff022700010202220000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0d30012e0004000000010221024277c805bb4df0bb05abc9e33d80592395c55066768260aea201ee33efa6582c0503040101112000010102157b22646563696d616c5f646973706c6179223a337d0504000000031604000001241750788345e9da87d7b70efff7a16658c21c634cd85217aff9591007621b4ab7eb650000000106417373657431827c23a4b1a5f4f9a9a42e61aa7bf7bf7e7925b4723728b60ad1918d632e2b0a00000000001948000102022103a6553ff1aa8bb1dc91b35e1f8428f99e6dfc3a66e39945ad9c0c22fffec677ff0420776c54fefcec5c564d69e6d844ecb5bfe2b4ff729d4fee26171093a34f2e205c"
                }
            ]
        }
    ]
}
1280e18e0083:/$ 

it works.

Then, if I change the proof type back to transfer, it also works

``` 1280e18e0083:/$ tapcli universe sync --universe_host=172.99.0.5:8443 --proof_type='transfer' --asset_id=f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea { "synced_universes": [ { "old_asset_root": { "id": null, "mssmt_root": null, "asset_name": "", "amounts_by_asset_id": {} }, "new_asset_root": { "id": { "group_key": "3d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b", "proof_type": "PROOF_TYPE_TRANSFER" }, "mssmt_root": { "root_hash": "9a2e814691125f57cb9a30717130f6148a5940d6b7b72fc2b5567ab3e455d1c2", "root_sum": "1" }, "asset_name": "", "amounts_by_asset_id": {} }, "new_asset_leaves": [ { "asset": { "version": "ASSET_VERSION_V1", "asset_genesis": { "genesis_point": "65ebb74a1b62071059f9af1752d84c631cc25866a1f7ff0eb7d787dae9458378:1", "name": "Asset1", "meta_hash": "827c23a4b1a5f4f9a9a42e61aa7bf7bf7e7925b4723728b60ad1918d632e2b0a", "asset_id": "f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea", "asset_type": "NORMAL", "output_index": 0 }, "amount": "200000000", "lock_time": 0, "relative_lock_time": 0, "script_version": 0, "script_key": "0250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e6", "script_key_is_local": false, "asset_group": { "raw_group_key": "", "tweaked_group_key": "033d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b", "asset_witness": "", "tapscript_root": "" }, "chain_anchor": null, "prev_witnesses": [ { "prev_id": { "anchor_point": "0000000000000000000000000000000000000000000000000000000000000000:0", "asset_id": "0000000000000000000000000000000000000000000000000000000000000000", "script_key": "000000000000000000000000000000000000000000000000000000000000000000", "amount": "0" }, "tx_witness": [], "split_commitment": { "root_asset": { "version": "ASSET_VERSION_V0", "asset_genesis": { "genesis_point": "65ebb74a1b62071059f9af1752d84c631cc25866a1f7ff0eb7d787dae9458378:1", "name": "Asset1", "meta_hash": "827c23a4b1a5f4f9a9a42e61aa7bf7bf7e7925b4723728b60ad1918d632e2b0a", "asset_id": "f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea", "asset_type": "NORMAL", "output_index": 0 }, "amount": "209999999800000000", "lock_time": 0, "relative_lock_time": 0, "script_version": 0, "script_key": "02a48d9550a03394dc405fda546943ddcc88fb76f4402e885da852a4fbd3c7ccb3", "script_key_is_local": false, "asset_group": { "raw_group_key": "", "tweaked_group_key": "033d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b", "asset_witness": "", "tapscript_root": "" }, "chain_anchor": null, "prev_witnesses": [ { "prev_id": { "anchor_point": "0b5642f0c10208bd540e2bf251a90eee2e0be3a9f26011e58087266c2b34dc69:0", "asset_id": "f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea", "script_key": "0231254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e", "amount": "0" }, "tx_witness": [ "97b6c8effa238d66b82dc4a8b5a7866248badfceb6d79a67c34a456926939b369079325540a3ebbb8556b273f9715f6844d371500e9f218d74d822e9006745b0" ], "split_commitment": null } ], "is_spent": false, "lease_owner": "", "lease_expiry": "0", "is_burn": false, "script_key_declared_known": false, "script_key_has_script_path": false, "decimal_display": { "decimal_display": 3 } } } } ], "is_spent": false, "lease_owner": "", "lease_expiry": "0", "is_burn": false, "script_key_declared_known": false, "script_key_has_script_path": false, "decimal_display": { "decimal_display": 3 } }, "proof": "54415050000400000000022469dc342b6c268780e51160f2a9e30b2eee0ea951f22b0e54bd0802c1f042560b000000000450000000306d55460380ecc308b71ea2312686a0a5498b2999121df38849d5a88edaf0665b057a12595034eda7c71934a0b0b19b12b3f1bf9073c85fd5e95a6b168fab8e62e2030a68ffff7f200300000006fd01630200000000010269dc342b6c268780e51160f2a9e30b2eee0ea951f22b0e54bd0802c1f042560b00000000000000000069dc342b6c268780e51160f2a9e30b2eee0ea951f22b0e54bd0802c1f042560b01000000000000000003a086010000000000225120eadcd3ee51d941d2de15b3d75b7e07c25291a7968150ed6dbb4f1b92567ae2bee803000000000000225120a02c5ea371aa5ff8c8c9a5e5f3bee8128d990511ef8b80ef1af7a7f546e2ba4ea83ff40500000000225120370fb93baeac7f00e6407783d6c80ce7105dc1f9cb4a78ceacf6b98e0a2cce740140b5cd5b6a4ac7b24d6306b5a4f236f7f0e4f3f051bd6154a443c784f40525d1572936d58407d7181b53b7a49d0f8b2c36317779a51cbe905f20e661fa1432970e014058580ec842921703b54531df646a38f94a70ced77bcffbd18c4272416cf8458b6e9a650a33f349a04b34405a1d41ea0338225fc3d87277568401c896c1473c6f0000000008220113b19c129cdcd2d2c879522e0aeec4e3a94a5821b3a7334247192f4f6a02ef98000afd02f00001010250788345e9da87d7b70efff7a16658c21c634cd85217aff9591007621b4ab7eb650000000106417373657431827c23a4b1a5f4f9a9a42e61aa7bf7bf7e7925b4723728b60ad1918d632e2b0a00000000000401000605fe0bebc2000bfd024301fd023f0165000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000005fd01d44a000159258c0e4b5473185bbd0bc880b81d068e8df2daf2ec6aca35d451bc7f29853502ea11e31ee93e00ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff7ffd01860001000250788345e9da87d7b70efff7a16658c21c634cd85217aff9591007621b4ab7eb650000000106417373657431827c23a4b1a5f4f9a9a42e61aa7bf7bf7e7925b4723728b60ad1918d632e2b0a00000000000401000609ff02ea11e31ee93e000bad01ab016569dc342b6c268780e51160f2a9e30b2eee0ea951f22b0e54bd0802c1f042560b00000000f3487f9592307b8abcbaa7fb94fb2f7618141afc6269c6a8bce6b3bd03774eea0231254f00de3a7e4f437491020318d5d168c3b44f586679255aa7582efa9da29e0342014097b6c8effa238d66b82dc4a8b5a7866248badfceb6d79a67c34a456926939b369079325540a3ebbb8556b273f9715f6844d371500e9f218d74d822e9006745b00d2812e715439967f17d8354a1ae6ae1ca0ec2ad43a51f8597258d54012acecdb00f02ea11e32ad500000e020000102102a48d9550a03394dc405fda546943ddcc88fb76f4402e885da852a4fbd3c7ccb31121033d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b0e02000010210250aaeb166f4234650d84a2d8a130987aeaf6950206e0905401ee74ff3f8d18e61121033d55ec3fc9561c6d96d7145a7c6a49865c601e462e3ca6896801d13bed79557b0c9f000400000000022102fe1f54b5dd49c28ea64ccf34f22fc8696e6ed21ab10442bbf9064a963aadcf6a037401490001010220308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb04220000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff022700010202220000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0df802c7000400000001022103b32611d2fb23f735618811e0909a6995fae2e470ac23a400e6fded3256bcc105039c01710001000220308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb044a0001d7e65f4fa206b96348aaec35086dc9cbdd57135cddbdc4e64621057fde1e854502ea11e31ee93e00ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffef022700010202220000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff2e00040000000202210230713f278def95d2cec0a835eeab98d6a880b2e8dcc459f493be7bb9602ae03705030401010f9f000400000001022103b32611d2fb23f735618811e0909a6995fae2e470ac23a400e6fded3256bcc105037401490001000220308c4a82da14ef62bf0c42dc901c17d94c65042c0f261e86212f9c1a1c9f17eb04220000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff022700010202220000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff16040000012e" } ] } ] } 1280e18e0083:/$
</details>






I've also tried to get george to sync the universe with alice, but that results in the same error:


<details>

controller-1 | alice funding a 9e23bd4b7b3704e0cecfdb292d2d7f021336907d080d6fcf31bddb20642b8add channel with bob
controller-1 | alice connecting to bob
controller-1 | syncing the universe of bob to alice (172.99.0.4:8443)
controller-1 | done syncing universe
controller-1 | mining 10 blocks
controller-1 | bob funding a normal sats channel with charlie
controller-1 | bob connecting to charlie
controller-1 | mining 10 blocks
controller-1 | mining 10 blocks
controller-1 | charlie funding a normal sats channel with dave
controller-1 | charlie connecting to dave
controller-1 | mining 10 blocks
controller-1 | mining 10 blocks
controller-1 | dave funding a normal sats channel with edward
controller-1 | dave connecting to edward
controller-1 | mining 10 blocks
controller-1 | mining 10 blocks
controller-1 | alice sending 500000000 of 9e23bd4b7b3704e0cecfdb292d2d7f021336907d080d6fcf31bddb20642b8add to bob on chain
controller-1 | syncing the universe of bob to alice (172.99.0.4:8443)
controller-1 | mining 10 blocks
controller-1 | bob funding a 9e23bd4b7b3704e0cecfdb292d2d7f021336907d080d6fcf31bddb20642b8add channel with george
controller-1 | bob connecting to george
controller-1 | syncing the universe of george to alice (172.99.0.4:8443)
controller-1 | unknown
controller-1 | ERROR MESSAGE: unable to sync universe: unable to register proofs: unable to register proofs: unable to batch verify issuance proofs: unable to fetch previous asset snapshot: unable to fetch previous proof: no universe proof found, id=(universe.Identifier) issuance-708e8571b2ca36efc260100e9525a5c1fd165cc64a147d7ec604c1ac63e490c3
controller-1 | , leaf_key=(universe.BaseLeafKey) {
controller-1 | OutPoint: (wire.OutPoint) ec2daff7e26eb56b2940ab484431d2fca30442131d3117dfb938de91a63605ab:0,
controller-1 | ScriptKey: (*asset.ScriptKey)(0xc004c54300)({
controller-1 | PubKey: (*secp256k1.PublicKey)(0xc003eba0a0)({
controller-1 | x: (secp256k1.FieldVal) e9fa1cb15a00e37ae2c8f37b94fdf71ccad02b1ac5e7aa945ae676e3cc449d5d,
controller-1 | y: (secp256k1.FieldVal) 76ee13ef5ee1fae1e5cce7f0fedc228462bb50f020fec9608972fac92cf81cb2
controller-1 | }),
controller-1 | TweakedScriptKey: (*asset.TweakedScriptKey)()
controller-1 | })
controller-1 | }
controller-1 | , new_script_key=027a1023b5d70bb7c36cc7a16e6011c9be4667671555e88ab949cf8460acbea522
controller-1 | syncing failed, trying again
controller-1 | Traceback (most recent call last):
controller-1 | File "/mini_META/scripts/init_network", line 149, in
controller-1 | OpenTaprootAssetChannel(funder='bob', peer='george', AssetID=TheAssetID, AssetCapacity=200000000)
controller-1 | File "/mini_META/scripts/mini_META_lib.py", line 551, in OpenTaprootAssetChannel
controller-1 | litd_node_objects[peer]['tapd'].sync_universe(universe_host=universe_host, sync_mode=1)
controller-1 | File "/usr/local/lib/python3.10/dist-packages/lndgrpc/errors.py", line 50, in wrapper
controller-1 | raise exc
controller-1 | File "/usr/local/lib/python3.10/dist-packages/lndgrpc/errors.py", line 26, in wrapper
controller-1 | return fnc(*args, **kwargs)
controller-1 | File "/usr/local/lib/python3.10/dist-packages/lndgrpc/universe.py", line 20, in sync_universe
controller-1 | response = self.get_universe_stub().SyncUniverse(request)
controller-1 | File "/usr/local/lib/python3.10/dist-packages/grpc/_channel.py", line 1181, in call
controller-1 | return _end_unary_response_blocking(state, call, False, None)
controller-1 | File "/usr/local/lib/python3.10/dist-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
controller-1 | raise _InactiveRpcError(state) # pytype: disable=not-instantiable
controller-1 | grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
controller-1 | status = StatusCode.UNKNOWN
controller-1 | details = "unable to sync universe: unable to register proofs: unable to register proofs: unable to batch verify issuance proofs: unable to fetch previous asset snapshot: unable to fetch previous proof: no universe proof found, id=(universe.Identifier) issuance-708e8571b2ca36efc260100e9525a5c1fd165cc64a147d7ec604c1ac63e490c3
controller-1 | , leaf_key=(universe.BaseLeafKey) {
controller-1 | OutPoint: (wire.OutPoint) ec2daff7e26eb56b2940ab484431d2fca30442131d3117dfb938de91a63605ab:0,
controller-1 | ScriptKey: (*asset.ScriptKey)(0xc004c54300)({
controller-1 | PubKey: (*secp256k1.PublicKey)(0xc003eba0a0)({
controller-1 | x: (secp256k1.FieldVal) e9fa1cb15a00e37ae2c8f37b94fdf71ccad02b1ac5e7aa945ae676e3cc449d5d,
controller-1 | y: (secp256k1.FieldVal) 76ee13ef5ee1fae1e5cce7f0fedc228462bb50f020fec9608972fac92cf81cb2
controller-1 | }),
controller-1 | TweakedScriptKey: (*asset.TweakedScriptKey)()
controller-1 | })
controller-1 | }
controller-1 | , new_script_key=027a1023b5d70bb7c36cc7a16e6011c9be4667671555e88ab949cf8460acbea522"
controller-1 | debug_error_string = "UNKNOWN:Error received from peer ipv4:172.99.0.10:8443 {created_time:"2025-04-24T05:36:15.574152077-04:00", grpc_status:2, grpc_message:"unable to sync universe: unable to register proofs: unable to register proofs: unable to batch verify issuance proofs: unable to fetch previous asset snapshot: unable to fetch previous proof: no universe proof found, id=(universe.Identifier) issuance-708e8571b2ca36efc260100e9525a5c1fd165cc64a147d7ec604c1ac63e490c3\n, leaf_key=(universe.BaseLeafKey) {\n OutPoint: (wire.OutPoint) ec2daff7e26eb56b2940ab484431d2fca30442131d3117dfb938de91a63605ab:0,\n ScriptKey: (*asset.ScriptKey)(0xc004c54300)({\n PubKey: (*secp256k1.PublicKey)(0xc003eba0a0)({\n x: (secp256k1.FieldVal) e9fa1cb15a00e37ae2c8f37b94fdf71ccad02b1ac5e7aa945ae676e3cc449d5d,\n y: (secp256k1.FieldVal) 76ee13ef5ee1fae1e5cce7f0fedc228462bb50f020fec9608972fac92cf81cb2\n }),\n TweakedScriptKey: (*asset.TweakedScriptKey)()\n })\n}\n, new_script_key=027a1023b5d70bb7c36cc7a16e6011c9be4667671555e88ab949cf8460acbea522"}"
controller-1 | >


</details>
@ZZiigguurraatt ZZiigguurraatt added bug Something isn't working needs triage labels Apr 24, 2025
@ZZiigguurraatt
Copy link
Author

Here is my network topology

#         TA         SAT            SAT         SAT           TA
# alice <----> bob <----> charlie <----> dave <----> edward <----> frank
#               ^            ^             ^
#               |            |             |
#              T|A          T|A           T|A
#               |            |             |
#               v            v             v
#             goerge       henry         irina

@guggero
Copy link
Member

guggero commented Apr 24, 2025

This is expected. You can't sync transfer proofs without first knowing about the issuance of an asset.

@ZZiigguurraatt
Copy link
Author

I take a look at the itests and we have Zane as the proof courier

https://github.com/lightninglabs/lightning-terminal/blob/72477225a8892b419e8903bf746a443920598cf7/itest/litd_custom_channels_test.go#L299-L311

and then we seem to sync dave, erin, fabia, yara to charlie

https://github.com/lightninglabs/lightning-terminal/blob/72477225a8892b419e8903bf746a443920598cf7/itest/litd_custom_channels_test.go#L388-L390

https://github.com/lightninglabs/lightning-terminal/blob/72477225a8892b419e8903bf746a443920598cf7/itest/assets_test.go#L371-L388

but I'm not sure why they sync universes with charlie instead of zane who is supposed to be the proof courier?

	// Charlie  --[assets]-->  Dave  --[sats]-->  Erin  --[assets]-->  Fabia
	//                          |
	//                          |
	//                       [assets]
	//                          |
	//                          v
	//                        Yara
	//

@ZZiigguurraatt
Copy link
Author

This is expected. You can't sync transfer proofs without first knowing about the issuance of an asset.

Why can bob sync the universe of alice without requesting a transfer of the issuance proof first?

Why was it not required in older versions of tapd to transfer the issuance proof first?

@ZZiigguurraatt
Copy link
Author

https://lightning.engineering/api-docs/api/taproot-assets/universe/sync-universe/#universerpcuniversesyncmode says

SYNC_FULL
A syncing mode that indicates that all asset proofs should be synced. This includes normal transfers as well.

why would I expect to need to do something else first to get all asset proofs?

@ZZiigguurraatt
Copy link
Author

In my code I'm using sync_mode=1

@guggero
Copy link
Member

guggero commented Apr 24, 2025

By default, syncing proofs is "pull", not "pull and push".
Which means: If you sync a universe with another one, the syncer will fetch all proofs from the syncee, but not the other way around.

The exception is if you actually add a universe as a federation server before minting an asset, then the issuance proof will be pushed to the federation universe servers on minting.

In our itests we use Zane as the proof courier only. Meaning, it will not act as a full universe, only for facilitating the transfers.
And when a proof is pushed to a courier, it happens recursively through the whole proof chain, starting at the issuance. Meaning: The proof courier universe node doesn't have to be synced to anyone, it can just be a standalone tapd.

The reason we sync all the nodes to Charlie instead of Zane is because Charlie mints the asset. So it already has the issuance proofs the others need to know about to know how to receive the assets.

So tl;dr, what you'll want is:

  • Configure one of the nodes as the proof courier, doesn't matter which one, but use the same courier address on all nodes
  • Mint the asset on one node
  • Sync issuance proofs to the other nodes
  • Ignore transfer proofs, you don't need to handle those, that's part of the proof courier job

@ZZiigguurraatt
Copy link
Author

By default, syncing proofs is "pull", not "pull and push". Which means: If you sync a universe with another one, the syncer will fetch all proofs from the syncee, but not the other way around.

Above when I did syncing the universe of george to bob (172.99.0.5:8443), george is the syncer, so I was expecting george to fetch all issuance and transfer proofs from bob since my sync_mode is SYNC_FULL. Could it be that george is receiving the transfer proofs first and as they are validated, they are rejected? Like I'm wondering if we should ensure when using SYNC_FULL that issuance proofs must all come in first?

The exception is if you actually add a universe as a federation server before minting an asset, then the issuance proof will be pushed to the federation universe servers on minting.

In our itests we use Zane as the proof courier only. Meaning, it will not act as a full universe, only for facilitating the transfers. And when a proof is pushed to a courier, it happens recursively through the whole proof chain, starting at the issuance. Meaning: The proof courier universe node doesn't have to be synced to anyone, it can just be a standalone tapd.

Is there a reason you don't make charlie the proof courier?

@guggero
Copy link
Member

guggero commented Apr 24, 2025

Like I'm wondering if we should ensure when using SYNC_FULL that issuance proofs must all come in first?

If you're doing a full sync, the issuance proofs are synced first.

But what a full sync does also depends on your sync configuration. See options like --universe.sync-all-assets or CLI commands like tapcli universe federation config --help (which maps to https://lightning.engineering/api-docs/api/taproot-assets/universe/set-federation-sync-config/)

Is there a reason you don't make charlie the proof courier?

Not a technical one. Just that we can be sure to exclude any side effects of a node using itself as the courier. And because it's easier to find out if the courier did its job if we can detect the proof in a different node's universe.

@ZZiigguurraatt
Copy link
Author

If you're doing a full sync, the issuance proofs are synced first.

But what a full sync does also depends on your sync configuration. See options like --universe.sync-all-assets or CLI commands like tapcli universe federation config --help (which maps to https://lightning.engineering/api-docs/api/taproot-assets/universe/set-federation-sync-config/)

I do have --taproot-assets.universe.sync-all-assets set, so why would this be failing if I'm also trying to do a full sync?

@ZZiigguurraatt
Copy link
Author

By default, syncing proofs is "pull", not "pull and push".

What is the non-default option I can set to do a push and pull?

@ZZiigguurraatt
Copy link
Author

I think one confusion here is universerpc.ProofType (https://lightning.engineering/api-docs/api/taproot-assets/universe/sync-universe/#universerpcprooftype) vs universerpc.UniverseSyncMode (https://lightning.engineering/api-docs/api/taproot-assets/universe/sync-universe/#universerpcuniversesyncmode) as both have options that seem to conflict with each other. Which one takes precedence?

@ZZiigguurraatt
Copy link
Author

As a workaround, I've tried the following sequence:

  1. sync with sync_mode=0
  2. sleep 10 seconds
  3. sync with sync_mode=1

This works about 50% of the time. When it does fail, it always fails with the step where sync_mode=1 and not when sync_mode=0.

@ZZiigguurraatt
Copy link
Author

As a workaround, I've tried the following sequence:

1. sync with `sync_mode=0`

2. sleep 10 seconds

3. sync with `sync_mode=1`

This works about 50% of the time. When it does fail, it always fails with the step where sync_mode=1 and not when sync_mode=0.

I take some of this back. I looked closer at my error log and realized I had a second place in my code that also tried to sync universes, so both places weren't running the first sync with sync_mode=0.

I've replaced both with the following sequence:

  1. sync with sync_mode=0
  2. sync with sync_mode=1

I've had 10 consecutive successful runs this way.

It seems that sync_mode=1 ("SYNC_FULL, A syncing mode that indicates that all asset proofs should be synced. This includes normal transfers as well.") is randomly not working right but if I force syncing the issuance proofs first, the transfer proofs always work.

@ZZiigguurraatt ZZiigguurraatt changed the title [bug]: unable to sync universe manually [bug]: SyncUniverse RPC's sync_mode=1 option not working reliably Apr 24, 2025
@Roasbeef
Copy link
Member

Roasbeef commented Apr 28, 2025

I do have --taproot-assets.universe.sync-all-assets set, so why would this be failing if I'm also trying to do a full sync?

Have you added any of the relevant nodes to your universe federation? That flag controls if the universe federation should check the normal config, or just ignore that and try to sync all the assets periodically from the set of federation members.

As far as the SyncUniverse RPC, if it gets all the assets or not depends on your universe federation config. We fetch, then pass this along while processing the SyncUniverse RPC.

The sync all flag applies only to the passive federation nodes you can add.

In your testbed, what does tapcli universe f c return?

as both have options that seem to conflict with each other. Which one takes precedence?

Only sync_type is present:

message SyncRequest {
// TODO(roasbeef): accept connection type? so can pass along self-signed
// cert, also brontide based RPC handshake
string universe_host = 1;
// The sync mode. This determines what type of proofs are synced.
UniverseSyncMode sync_mode = 2;
// The set of assets to sync. If none are specified, then all assets are
// synced.
repeated SyncTarget sync_targets = 3;
}

@Roasbeef
Copy link
Member

Also I edited the issue a bit to use <details></details> to collapse some of the long segments to make it easier to read.

@ZZiigguurraatt
Copy link
Author

Also I edited the issue a bit to use <details></details> to collapse some of the long segments to make it easier to read.

cool, didn't know you could do that!

@ZZiigguurraatt
Copy link
Author

In your testbed, what does tapcli universe f c return?

4d44bef66ecb:/$ tapcli universe f c
NAME:
   tapcli universe federation config - 
  Manage the sync behavior of the local Universe. These settings are
  defined by the proof type (issuance or transfer), the sync behavior
  (insert from remote Universe or export to remote Universe), and the
  scope (all assets or specific assets).
        

USAGE:
   tapcli universe federation config command [command options] [arguments...]

COMMANDS:
   global, g  Change the global sync behavior of the local Universe
   local, l   Change the sync behavior of the local Universe for a specific asset
   info, i    Get the sync behavior of the local Universe

OPTIONS:
   --help, -h  show help
   
4d44bef66ecb:/$ 

@ZZiigguurraatt
Copy link
Author

Have you added any of the relevant nodes to your universe federation?

I don't use a federation.

4d44bef66ecb:/$ tapcli universe federation list
{
    "servers": []
}
4d44bef66ecb:/$ 

I feel like if I want to manually SyncUniverse with a specific peer, I should not need a federation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs triage
Projects
None yet
Development

No branches or pull requests

3 participants