universe+tapdb: add new `block_height` field to `universe_leaves` use that to implement block height based supply tree queries #1596

Roasbeef · 2025-06-13T00:26:37Z

In this PR, we add a new method FetchSupplyLeavesByHeight, that can be used to query for the leaves of a supply tree by height. This will be useful for writing the new syncing state machine and the sub-system that serves the supply tree syncer. Such a syncer would start from the initially created pre-commitment, then fetch all the leaves for the future updates, assemble those into the supply tree, then verify that everything matches up.

Prompts used during development (aider+Gemini 2.5-pro-06-5)

Initial draft creation

Based on by working memory of this new feature, I scanned the relevant files, then made mental nodes of what needed to be updated. I then drafted this fairly detailed prompt that guided aider w.r.t which files to update, and the set of high level changes to make.

#### Today we assemble the individual updates (burn, ignore, mint) for a given asset  
#### specifier (group key is what we care about) into distinct trees for each type.  
#### We then further aggregrate each of those roots into a unified supply tree for  
#### the entire asset specifier. This tree commits to a root sum of the supply (you  
#### can subtract out any of the roots from the root sum to derive a snapshot of the  
#### supply such as the amt burnt, etc).  
####   
#### We then also have a system to assemble these updates, then commit then in a  
#### batch, this exists in: tapdb/supply_commit.go and tapdb/supply_tree.go.  
####   
#### For each of the supply sub-trees, we actually re-use the existing  
#### universe_leaves table to point to each of the respective leaves.  
####   
#### In this task, we want to add a block height to this leaf, as this'll allow us  
#### to scan the leaves in order, based on the type, so we can send out the set of  
#### leaves added in a state transition to a party that wants to sync the set of  
#### leaves.  
####   
####   
#### For this, we may need to update the set of queries in:  
#### tapdb/sqlc/queries/universe.sql. Of interest is also the  
#### applySupplyUpdatesInternal helper function, as we'll want to make sure that the  
#### helper functions that it calls properly inserts the height into the ultimately  
#### universe leaf. For the mint and burn types, we can obtain this height from the  
#### issuance proof.  
####   
#### For the ignore type, we'll need to add a block height field to  
#### universe.IgnoreTuple (universe/ignore_records.go).  
####   
#### For the leaves, we'll likely need to add a new migration (38) which adds the  
#### new height field (can be NULL) to the universe_leaves table.  
####   
#### We'll likely also want to add a new BlockHeight() method to the  
#### SupplyUpdateEvent interface.  
####   
#### Next, we'll want a new method on the SupplyTreeStore that given a start+end  
#### height and a specifier, the set of updates whose universe_leaves have a block  
#### height between that param. This may need to update  
#### tapdb/sqlc/queries/supply_tree.sql.  
####   
#### Also see tapdb/querier.go.  
####   
#### Ruminate on my sketch above, then let's create an execution plan remembering  
#### the ultimate goal. From there we'll implement each portion.

Subsequent iterations

From here I made an edit or two to get things compiling. Then I executed a series of prompts to refine the code, refactor slightly, then add unit tests.

inside multiverse.UpsertProof Leaf, let's use SparseDecode instead along with BlockHeightRecord as we don't need to decode the entire proof just to get the block height

let's use the same sparse decode in BaseUniverseStore.UpsertProofLeaf

let's do the same for SupplyTreeStore.RegisterMintSupply (sparse decode)

can we morph that into a function closure invocation that itself returns lfn.Option[uint32]? so we don't leak that intermediate variable to the higher scope

great! but now we have a bit of duplication, can we make a helper function can call that intead of having the same duplication everywhere? we can put the helper function in tapdb/sqlutils.go

nice! now let's extend the unit tests in tapdb/supply_tree_test.go to cover the new FetchSupplyLeavesByHeight method.

I messed when committing and dropped a commit entirely (lol), so I had aider re-add it:

help me ree-add QuerySupplyLeavesByHeight back to tapdb/sqlc/queries/supply_tree.sql, I deleted it accidentally

ffranr · 2025-06-13T15:03:32Z

tapdb/supply_tree.go

+// SupplyUpdate is a struct that holds a supply update event and its block
+// height.
+type SupplyUpdate struct {
+	supplycommit.SupplyUpdateEvent
+	BlockHeight uint32
+}
+


supplycommit.SupplyUpdateEvent already includes

// BlockHeight returns the block height of the update. BlockHeight() uint32

so I don't think we need the separate BlockHeight uint32 field in SupplyUpdate.

In this commit, we add a migration to add a block_height to the universe leaf table. We'll use this in the future to be able to allow clients to sync block by block for the new supply universe trees.

Roasbeef · 2025-07-03T01:46:35Z

Rebased.

In this commit, we add a new method to the SupplyTreeStore that is able to read out the leaves of a supply tree based on a start and end height. This will be useful for writing the new syncing state machine and the sub-system that serves the supply tree syncer.

coveralls · 2025-07-03T02:42:09Z

Pull Request Test Coverage Report for Build 16040068155

Details

155 of 216 (71.76%) changed or added relevant lines in 10 files are covered.
43 unchanged lines in 7 files lost coverage.
Overall coverage increased (+0.04%) to 39.302%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
tapdb/sqlc/universe.sql.go	1	2	50.0%
tapdb/universe.go	20	22	90.91%
tapdb/multiverse.go	12	16	75.0%
tapdb/sqlutils.go	10	14	71.43%
universe/ignore_records.go	9	13	69.23%
tapdb/sqlc/supply_tree.sql.go	19	27	70.37%
universe/supplycommit/states.go	1	9	11.11%
tapdb/supply_tree.go	64	94	68.09%

Files with Coverage Reduction	New Missed Lines	%
universe/supplycommit/states.go	1	91.28%
tapdb/universe.go	4	79.68%
commitment/tap.go	5	71.59%
address/address.go	6	69.55%
tapchannel/aux_invoice_manager.go	6	80.51%
internal/test/copy.go	7	81.85%
address/mock.go	14	88.24%

Totals
Change from base Build 15982579791:	0.04%
Covered Lines:	30750
Relevant Lines:	78240

💛 - Coveralls

lightninglabs-deploy · 2025-07-10T03:06:49Z

@ffranr: review reminder

guggero

Looks great! Got a couple of suggestions and the linter isn't fully happy yet. Other than that LGTM 🎉

guggero · 2025-07-11T11:30:22Z

tapdb/ignore_tree.go

+			blockHeight = lfn.Some(height)
+		}
+
+		sqlBlockHeight := lfn.MapOptionZ(


Looks like we already have a helper for this: sqlOptInt32.

guggero · 2025-07-11T11:31:21Z

tapdb/sqlc/queries/universe.sql

 ) ON CONFLICT (minting_point, script_key_bytes, leaf_node_namespace)
    -- This is a NOP, minting_point and script_key_bytes are the unique fields
    -- that caused the conflict.
    DO UPDATE SET minting_point = EXCLUDED.minting_point,
                  script_key_bytes = EXCLUDED.script_key_bytes,
-                  leaf_node_namespace = EXCLUDED.leaf_node_namespace;
+                  leaf_node_namespace = EXCLUDED.leaf_node_namespace,
+                  block_height = EXCLUDED.block_height;


This invalidates the This is a NOP, comment above! This would allow updating the block height specifically. Is that the intention? If yes, then we should update the comment.

guggero · 2025-07-11T11:32:11Z

tapdb/universe.go

+	// If the block height isn't specified, then we'll attempt to extract it
+	// from the proof itself.
+	if blockHeight.IsNone() {
+		if leafProof.BlockHeight > 0 {


nit: could combine in a single && condition?

guggero · 2025-07-11T11:32:25Z

tapdb/universe.go

+		}
+	}
+
+	sqlBlockHeight := lfn.MapOptionZ(


nit: use sqlOptInt32.

guggero · 2025-07-11T11:33:32Z

tapdb/supply_tree.go

+
+	readTx := NewBaseUniverseReadTx()
+	dbErr := s.db.ExecTx(ctx, &readTx, func(db BaseUniverseStore) error {
+		for _, treeType := range []supplycommit.SupplySubTree{


nit: feels like this would make a good package-level variable with the list of tree types we often iterate over? Would also help a bit with the formatting/readability.

ffranr

Pretty much there. One question on SQL query re-use.

ffranr · 2025-07-17T13:36:14Z

tapdb/sqlc/queries/supply_tree.sql

+-- name: QuerySupplyLeavesByHeight :many
+SELECT
+    leaves.script_key_bytes,
+    gen.gen_asset_id,
+    nodes.value AS supply_leaf_bytes,
+    nodes.sum AS sum_amt,
+    gen.asset_id,
+    leaves.block_height
+FROM universe_leaves AS leaves
+JOIN mssmt_nodes AS nodes
+    ON leaves.leaf_node_key = nodes.key
+    AND leaves.leaf_node_namespace = nodes.namespace


We already have a where clause in QueryUniverseSupplyLeaves. Instead of adding a new query here, can't we just extend QueryUniverseSupplyLeaves with an optional block height?

Roasbeef added enhancement New feature or request commitments supply commit Work on the supply commitment feature, enabling issuers to attest to total asset supply on-chain. labels Jun 13, 2025

ffranr reviewed Jun 13, 2025

View reviewed changes

levmi added this to Taproot-Assets Project Board Jun 16, 2025

github-project-automation bot moved this to 🆕 New in Taproot-Assets Project Board Jun 16, 2025

levmi assigned Roasbeef Jun 16, 2025

Roasbeef force-pushed the supply-leaf-height-query branch from 3e7ed65 to c57d19d Compare June 18, 2025 01:51

Roasbeef force-pushed the supply-commit-machine branch from c2c394f to b1173b9 Compare June 18, 2025 01:51

Roasbeef force-pushed the supply-leaf-height-query branch from c57d19d to bd5d0b4 Compare June 18, 2025 02:06

Roasbeef force-pushed the supply-commit-machine branch from b1173b9 to c6f9962 Compare June 18, 2025 02:06

Roasbeef force-pushed the supply-leaf-height-query branch from bd5d0b4 to 37f23fa Compare June 18, 2025 23:02

Roasbeef force-pushed the supply-commit-machine branch from c6f9962 to da0df38 Compare June 18, 2025 23:02

Roasbeef force-pushed the supply-leaf-height-query branch from 37f23fa to be0a52c Compare June 18, 2025 23:25

Roasbeef force-pushed the supply-commit-machine branch 2 times, most recently from ee045a1 to 598067d Compare June 18, 2025 23:51

Roasbeef force-pushed the supply-leaf-height-query branch from be0a52c to f2a69aa Compare June 18, 2025 23:57

levmi moved this from 🆕 New to 🏗 In progress in Taproot-Assets Project Board Jun 19, 2025

levmi moved this from 🏗 In progress to 👀 In review in Taproot-Assets Project Board Jun 19, 2025

Roasbeef force-pushed the supply-leaf-height-query branch from f2a69aa to 93ed469 Compare June 20, 2025 23:49

Roasbeef force-pushed the supply-commit-machine branch from 598067d to 9f03d81 Compare June 20, 2025 23:49

Roasbeef force-pushed the supply-leaf-height-query branch from 93ed469 to 633373a Compare June 21, 2025 00:08

Roasbeef force-pushed the supply-commit-machine branch from 9f03d81 to e77d0cf Compare June 21, 2025 00:08

Roasbeef force-pushed the supply-leaf-height-query branch from 633373a to 3ca1517 Compare June 21, 2025 00:12

Roasbeef force-pushed the supply-commit-machine branch from e77d0cf to 3044a87 Compare June 21, 2025 00:12

Roasbeef force-pushed the supply-leaf-height-query branch from 3ca1517 to 988837b Compare June 21, 2025 00:20

Roasbeef force-pushed the supply-commit-machine branch from 3044a87 to 9854971 Compare June 21, 2025 00:20

Roasbeef force-pushed the supply-leaf-height-query branch from 988837b to a9ef139 Compare June 21, 2025 01:22

Roasbeef force-pushed the supply-commit-machine branch from 9854971 to 37bdcfa Compare June 21, 2025 01:22

Roasbeef force-pushed the supply-leaf-height-query branch from a9ef139 to ca7db9a Compare June 25, 2025 00:27

Roasbeef force-pushed the supply-commit-machine branch from 37bdcfa to b5a6c5e Compare June 25, 2025 00:27

Roasbeef force-pushed the supply-leaf-height-query branch from ca7db9a to ec56941 Compare June 25, 2025 01:59

Roasbeef force-pushed the supply-commit-machine branch from b5a6c5e to c59b408 Compare June 25, 2025 02:00

Roasbeef force-pushed the supply-leaf-height-query branch from ec56941 to 16f9f9a Compare June 26, 2025 22:30

Roasbeef force-pushed the supply-commit-machine branch from c59b408 to 99529ae Compare June 26, 2025 22:30

Roasbeef force-pushed the supply-leaf-height-query branch from 16f9f9a to 059ed3f Compare June 27, 2025 00:08

Roasbeef force-pushed the supply-commit-machine branch from 99529ae to 87ba694 Compare June 27, 2025 00:08

Roasbeef added 5 commits July 2, 2025 18:38

universe: add BlockHeight method to SupplyUpdateEvent

29fa8bb

tapdb/sqlc: add migration to add block_height to universe leaf

87da0cd

In this commit, we add a migration to add a block_height to the universe leaf table. We'll use this in the future to be able to allow clients to sync block by block for the new supply universe trees.

tapdb/sqlc/queries: add QuerySupplyLeavesByHeight query

05641ed

tapdb: add SparseBlockHeight helper function

1fb2eb6

tapdb: update all trees to specify a block height for a universe leaf

a9bc79d

Roasbeef force-pushed the supply-leaf-height-query branch from 059ed3f to 5025c8c Compare July 3, 2025 01:45

Roasbeef changed the base branch from supply-commit-machine to main July 3, 2025 01:45

Roasbeef force-pushed the supply-leaf-height-query branch from 5025c8c to acafde1 Compare July 3, 2025 02:20

Roasbeef requested a review from ffranr July 3, 2025 20:07

ffranr requested a review from guggero July 10, 2025 14:21

guggero reviewed Jul 11, 2025

View reviewed changes

ffranr reviewed Jul 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

universe+tapdb: add new `block_height` field to `universe_leaves` use that to implement block height based supply tree queries #1596

universe+tapdb: add new `block_height` field to `universe_leaves` use that to implement block height based supply tree queries #1596

Roasbeef commented Jun 13, 2025

Uh oh!

ffranr Jun 13, 2025

Uh oh!

Roasbeef commented Jul 3, 2025

Uh oh!

coveralls commented Jul 3, 2025

Uh oh!

lightninglabs-deploy commented Jul 10, 2025

Uh oh!

guggero left a comment

Uh oh!

guggero Jul 11, 2025

Uh oh!

guggero Jul 11, 2025

Uh oh!

guggero Jul 11, 2025

Uh oh!

guggero Jul 11, 2025

Uh oh!

guggero Jul 11, 2025

Uh oh!

ffranr left a comment

Uh oh!

ffranr Jul 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

universe+tapdb: add new block_height field to universe_leaves use that to implement block height based supply tree queries #1596

Are you sure you want to change the base?

universe+tapdb: add new block_height field to universe_leaves use that to implement block height based supply tree queries #1596

Conversation

Roasbeef commented Jun 13, 2025

Initial draft creation

Subsequent iterations

inside multiverse.UpsertProof Leaf, let's use SparseDecode instead along with BlockHeightRecord as we don't need to decode the entire proof just to get the block height

let's use the same sparse decode in BaseUniverseStore.UpsertProofLeaf

let's do the same for SupplyTreeStore.RegisterMintSupply (sparse decode)

can we morph that into a function closure invocation that itself returns lfn.Option[uint32]? so we don't leak that intermediate variable to the higher scope

great! but now we have a bit of duplication, can we make a helper function can call that intead of having the same duplication everywhere? we can put the helper function in tapdb/sqlutils.go

nice! now let's extend the unit tests in tapdb/supply_tree_test.go to cover the new FetchSupplyLeavesByHeight method.

help me ree-add QuerySupplyLeavesByHeight back to tapdb/sqlc/queries/supply_tree.sql, I deleted it accidentally

Uh oh!

ffranr Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

Roasbeef commented Jul 3, 2025

Uh oh!

coveralls commented Jul 3, 2025

Pull Request Test Coverage Report for Build 16040068155

Details

💛 - Coveralls

Uh oh!

lightninglabs-deploy commented Jul 10, 2025

Uh oh!

guggero left a comment

Choose a reason for hiding this comment

Uh oh!

guggero Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

guggero Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

guggero Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

guggero Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

guggero Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

ffranr left a comment

Choose a reason for hiding this comment

Uh oh!

ffranr Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

universe+tapdb: add new `block_height` field to `universe_leaves` use that to implement block height based supply tree queries #1596

universe+tapdb: add new `block_height` field to `universe_leaves` use that to implement block height based supply tree queries #1596

inside multiverse.UpsertProof Leaf, let's use `SparseDecode` instead along with `BlockHeightRecord` as we don't need to decode the entire proof just to get the block height

let's use the same sparse decode in `BaseUniverseStore`.UpsertProofLeaf

let's do the same for `SupplyTreeStore`.RegisterMintSupply (sparse decode)

nice! now let's extend the unit tests in tapdb/supply_tree_test.go to cover the new `FetchSupplyLeavesByHeight` method.

help me ree-add `QuerySupplyLeavesByHeight` back to tapdb/sqlc/queries/supply_tree.sql, I deleted it accidentally

ffranr Jul 17, 2025 •

edited

Loading