Skip to content

feat!: migrate additional indexes from synchronous to async operation #7101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
UdjinM6 wants to merge 22 commits into
base: develop
Choose a base branch
from
Open

feat!: migrate additional indexes from synchronous to async operation #7101

Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
f3fcb97
feat: migrate TimestampIndex from synchronous to async operation
UdjinM6 Jan 19, 2026
33d2fc5
test: update timestampindex functional test for async behavior
UdjinM6 Jan 19, 2026
464cdf8
feat: migrate SpentIndex from synchronous to async operation
UdjinM6 Jan 19, 2026
3ea9394
test: update feature_spentindex for async behavior
UdjinM6 Jan 19, 2026
18276a6
feat: migrate AddressIndex from synchronous to async operation
UdjinM6 Jan 19, 2026
8ad06be
test: update feature_addressindex for async behavior
UdjinM6 Jan 19, 2026
fa7be34
feat: add CompactRange template method to CDBWrapper
UdjinM6 Jan 19, 2026
fa60ccb
feat: migrate old synchronous index data to new async index databases
UdjinM6 Jan 19, 2026
6e5f7ab
fix: add missing include in `src/zmq/zmqpublishnotifier.cpp`
UdjinM6 Jan 19, 2026
5fd7033
docs: add release notes
UdjinM6 Jan 19, 2026
fd1de98
refactor: address code review feedback
UdjinM6 Jan 19, 2026
5be629b
refactor: apply clang-format
UdjinM6 Jan 19, 2026
e84b1fa
feat: add async indexes to getindexinfo RPC
UdjinM6 Jan 19, 2026
98c3c1b
fix: check index sync status before querying and show sync progress
UdjinM6 Jan 19, 2026
331335e
fix: add explicit vtxundo size validation and in-class initialization…
UdjinM6 Mar 12, 2026
97dcc6d
fix: address review suggestions
UdjinM6 Mar 27, 2026
8f3f62b
fix: add vtxundo size validation in SpentIndex::WriteBlock
UdjinM6 Mar 31, 2026
5796588
fix: reverse transaction processing order in AddressIndex::Rewind
UdjinM6 Mar 31, 2026
386acdf
fix: add undo data size validation in AddressIndex::Rewind
UdjinM6 Mar 31, 2026
1ca7374
fix: harden BlockDisconnected sibling check and getaddressbalance hei…
UdjinM6 Mar 31, 2026
c41584e
refactor: move BlockUntilSyncedToCurrentChain from index queries to R…
UdjinM6 Apr 1, 2026
0695e97
refactor: move BlockDisconnected handling to BaseIndex
UdjinM6 Apr 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 35 additions & 18 deletions test/functional/feature_timestampindex.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,51 +8,68 @@
#

from test_framework.test_framework import BitcoinTestFramework
from test_framework.test_node import ErrorMatch
from test_framework.util import assert_equal


class TimestampIndexTest(BitcoinTestFramework):

def set_test_params(self):
self.setup_clean_chain = True
self.num_nodes = 4
self.num_nodes = 2

def setup_network(self):
self.add_nodes(self.num_nodes)
# Nodes 0/1 are "wallet" nodes
self.start_node(0)
self.start_node(1, ["-timestampindex"])
# Nodes 2/3 are used for testing
self.start_node(2)
self.start_node(3, ["-timestampindex"])
self.connect_nodes(0, 1)
self.connect_nodes(0, 2)
self.connect_nodes(0, 3)

self.sync_all()

def run_test(self):
self.log.info("Test that settings can be disabled without -reindex...")
self.restart_node(1, ["-timestampindex=0"])
self.generate(self.nodes[0], 1)
self.stop_node(1)
self.generate(self.nodes[0], 1, sync_fun=self.no_op)
self.start_node(1, ["-timestampindex=0"])
self.connect_nodes(0, 1)
self.sync_all()
self.log.info("Test that settings can't be enabled without -reindex...")

self.log.info("Test that settings can be enabled without -reindex...")
self.stop_node(1)
self.nodes[1].assert_start_raises_init_error(["-timestampindex"], "You need to rebuild the database using -reindex to enable -timestampindex", match=ErrorMatch.PARTIAL_REGEX)
self.start_node(1, ["-timestampindex", "-reindex"])
self.generate(self.nodes[0], 1, sync_fun=self.no_op)
self.start_node(1, ["-timestampindex"])
Comment thread
knst marked this conversation as resolved.
self.connect_nodes(0, 1)
self.sync_all()

self.log.info("Mining 5 blocks...")
blockhashes = self.generate(self.nodes[0], 5)
self.log.info("Check timestamp index via getblockhashes rpc")
blockhashes = []
for _ in range(5):
self.bump_mocktime(1)
blockhashes += self.generate(self.nodes[0], 1)
low = self.nodes[0].getblock(blockhashes[0])["time"]
high = self.nodes[0].getblock(blockhashes[4])["time"]
self.log.info("Checking timestamp index...")
self.bump_mocktime(1)
hashes = self.nodes[1].getblockhashes(high, low)
assert_equal(len(hashes), 5)
assert_equal(sorted(blockhashes), sorted(hashes))
self.log.info("Passed")

Comment on lines +42 to +53
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Wait for timestampindex sync before getblockhashes.

getblockhashes now errors while the index is syncing; a short wait prevents flakes.

🧪 Suggested wait 
         for _ in range(5):
             self.bump_mocktime(1)
             blockhashes += self.generate(self.nodes[0], 1)
+
+        self.wait_until(lambda: self.nodes[1].getindexinfo("timestampindex")["timestampindex"]["synced"])
🤖 Prompt for AI Agents 
In `@test/functional/feature_timestampindex.py` around lines 42 - 53, The test
calls nodes[1].getblockhashes while the timestamp index may still be syncing;
add a short wait/poll after the last bump_mocktime(1) and before calling
nodes[1].getblockhashes(high, low) to ensure the timestampindex is synced.
Implement this by polling the node (e.g., using
nodes[1].getindexinfo("timestampindex") or nodes[1].getindexinfo() and checking
the timestampindex's sync flag) with a small sleep and a reasonable timeout, and
only proceed to call getblockhashes once the index reports synced to avoid
flakes.

self.log.info("Testing reorg handling...")
# Invalidate the last 2 blocks on both nodes
self.nodes[0].invalidateblock(blockhashes[3])
self.nodes[1].invalidateblock(blockhashes[3])

# Verify that invalidated blocks are no longer in the index
hashes_after_invalidate = self.nodes[1].getblockhashes(high, low)
assert_equal(len(hashes_after_invalidate), 3)
assert_equal(sorted(hashes_after_invalidate), sorted(blockhashes[:3]))

# Reconsider the blocks on both nodes
self.nodes[0].reconsiderblock(blockhashes[3])
self.nodes[1].reconsiderblock(blockhashes[3])
self.sync_all()

# Verify all blocks are back in the index
hashes_after_reconsider = self.nodes[1].getblockhashes(high, low)
assert_equal(len(hashes_after_reconsider), 5)
assert_equal(sorted(hashes_after_reconsider), sorted(blockhashes))
Comment on lines +54 to +72
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Reorg assertions should wait for async index updates.

The index can lag after invalidate/reconsider; polling avoids racey lengths.

🧪 Suggested waits 
         self.nodes[0].invalidateblock(blockhashes[3])
         self.nodes[1].invalidateblock(blockhashes[3])
+
+        self.wait_until(lambda: len(self.nodes[1].getblockhashes(high, low)) == 3)
@@
         self.nodes[0].reconsiderblock(blockhashes[3])
         self.nodes[1].reconsiderblock(blockhashes[3])
         self.sync_all()
+
+        self.wait_until(lambda: len(self.nodes[1].getblockhashes(high, low)) == 5)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
self.log.info("Testing reorg handling...")
# Invalidate the last 2 blocks on both nodes
self.nodes[0].invalidateblock(blockhashes[3])
self.nodes[1].invalidateblock(blockhashes[3])
# Verify that invalidated blocks are no longer in the index
hashes_after_invalidate = self.nodes[1].getblockhashes(high, low)
assert_equal(len(hashes_after_invalidate), 3)
assert_equal(sorted(hashes_after_invalidate), sorted(blockhashes[:3]))
# Reconsider the blocks on both nodes
self.nodes[0].reconsiderblock(blockhashes[3])
self.nodes[1].reconsiderblock(blockhashes[3])
self.sync_all()
# Verify all blocks are back in the index
hashes_after_reconsider = self.nodes[1].getblockhashes(high, low)
assert_equal(len(hashes_after_reconsider), 5)
assert_equal(sorted(hashes_after_reconsider), sorted(blockhashes))
self.log.info("Testing reorg handling...")
# Invalidate the last 2 blocks on both nodes
self.nodes[0].invalidateblock(blockhashes[3])
self.nodes[1].invalidateblock(blockhashes[3])
self.wait_until(lambda: len(self.nodes[1].getblockhashes(high, low)) == 3)
# Verify that invalidated blocks are no longer in the index
hashes_after_invalidate = self.nodes[1].getblockhashes(high, low)
assert_equal(len(hashes_after_invalidate), 3)
assert_equal(sorted(hashes_after_invalidate), sorted(blockhashes[:3]))
# Reconsider the blocks on both nodes
self.nodes[0].reconsiderblock(blockhashes[3])
self.nodes[1].reconsiderblock(blockhashes[3])
self.sync_all()
self.wait_until(lambda: len(self.nodes[1].getblockhashes(high, low)) == 5)
# Verify all blocks are back in the index
hashes_after_reconsider = self.nodes[1].getblockhashes(high, low)
assert_equal(len(hashes_after_reconsider), 5)
assert_equal(sorted(hashes_after_reconsider), sorted(blockhashes))
🤖 Prompt for AI Agents 
In `@test/functional/feature_timestampindex.py` around lines 54 - 72, The
assertions after calling invalidateblock and reconsiderblock are racing with
async index updates; replace the direct asserts on getblockhashes results with a
short polling loop that retries getblockhashes(high, low) until the expected
length and contents are observed (or a timeout elapses). Apply this for both the
"hashes_after_invalidate" check (expecting 3 and matching blockhashes[:3]) and
the "hashes_after_reconsider" check (expecting 5 and matching blockhashes),
using the same nodes[1].getblockhashes call and keeping sync_all around the
reconsiderblock calls; ensure the loop fails the test if the timeout is reached.



if __name__ == '__main__':
Expand Down