chainimport[2/3]: import block and filter headers on startup before falling back to P2P synchronization

[mohamedawnallah]

Description

This PR is part two of a three-part series. It only import block and filter headers on startup when target header stores are completely empty except for the existence of gensis block/filter header otherwise it is entirely skipped.

Towards #70.
Prev #320.
Next #324.
Replaces #285.

Demo (Side-Header Loading A/B Testing Experiments/neutrino_playground)

Mainnet Network (905,086 Headers)

before_mainnet_905086.mov

after_mainnet_905086.mov

Testnet V3 Network (4,551,195 Headers)

after-testnetv3-4551195.mp4

Note:

• Syncing the Testnet V3 network without importing headers took approximately 3 times longer than syncing with headers imported given the same operational environment
• The recording for the before version regards Testnet V3 network was too large to include here

Tasks

• Enable importing block and filter headers to completely empty target header stores
• Implement File/HTTP(s) Header Import Source
• Process New Headers Region
• Unit Tests
• Integration Tests
• Benchmarking

[guggero]

Did a first pass and left some comments and questions.
I think the reason this initially got stalled was unclear goals. I'm aware the design space is quite large here, so it's easy to lose track of what a first PR should be able to achieve.

So my recommendation is: Try to determine (and then document!) the minimum feature set that we'd want to have for a first PR, then only implement that.

Looking at the ideas in this PR, my personal opinion of what that could be is:

• Be able to pre-load headers, validate and load them into an empty DB
• Be able to specify different types of sources (file based or HTTP based, ideally in the format specified by https://github.com/guggero/block-dn/)

[mohamedawnallah]

Thanks @guggero for the feedback. I've addressed the feedback and explained the rationale behind multiple modes

[mohamedawnallah]

Discussed offline regards the way forward in this PR. This first PR regards neutrino side-header loading will focus solely on new Neutrino users who don't have headers data or existing users who are willing to wipe their headers store in order to use the import-based header loading. Overlapping, divergence, and incremental update cases will not be addressed in this first PR.

[guggero]

Thanks for the update. Looks a bit smaller, but still quite a bit of code to go through.
Would be great if we could add unit tests for the individual parts.

But I think we're on the right track, at least conceptually I feel like we can go with this approach for a first PR.

[guggero]

Great work on the unit and performance tests! Looks really good. Going to run some manual tests now, so perhaps @ziggie1984 can also start to take a look at the code.

[mohamedawnallah]

Have we experimented with a larger batch size to see how quickly we can reach the tip of testnet3?

@Roasbeef - I have experimented with batch sizes as seen in the benchmarking report in #317 (comment) and its impact on memory as seen in the benchmarking report in #317 (comment).

I've also attached a new demo in the PR description showing how it performs using batch size 16384 (2^14) headers on testnet3

[guggero]

Very cool! Was able to do a full mainnet sync in ~18 seconds using the following test code:

func TestNeutrinoSyncMainnet(t *testing.T) {
	tempDir := t.TempDir()

	// Create filters db for indexing the headers.
	db, err := walletdb.Create(
		"bdb", tempDir+"/filters.db", true, dbOpenTimeout,
	)
	require.NoError(t, err)
	defer db.Close()

	// Create service with headers import but with no peers.
	// This will test that the headers are imported correctly without
	// network sync. No peers specified – we want to test the import
	// functionality by itself.
	instance := "https://block-dn.org/"
	blockHeadersSource := instance + "headers/import/900000"
	filterHeadersSource := instance + "filter-headers/import/900000"
	importConfig := neutrino.Config{
		DataDir:     tempDir,
		Database:    db,
		ChainParams: chaincfg.MainNetParams,
		HeadersImport: &neutrino.HeadersImportConfig{
			BlockHeadersSource:      blockHeadersSource,
			FilterHeadersSource:     filterHeadersSource,
			WriteBatchSizePerRegion: 32000,
		},
	}

	importSvc, err := neutrino.NewChainService(importConfig)
	require.NoError(t, err)

	// Start the import service.
	err = importSvc.Start(context.Background())
	require.NoError(t, err)

	require.Eventually(t, func() bool {
		t.Logf("Is current: %v", importSvc.IsCurrent())
		return importSvc.IsCurrent()
	}, time.Minute, time.Second)

	err = importSvc.Stop()
	require.NoError(t, err)
}

Thanks to this commit: guggero/block-dn@e8b4de1
See documentation here: https://block-dn.org/

Very cool to see the block-DN project to become useful 🎉

[mohamedawnallah]

The push after asking for review is just rebasing on the dependency PR #320 and updating few outdated comments