You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+2-131Lines changed: 2 additions & 131 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -30,25 +30,19 @@ The Hybrid indexer is written in Rust. It can be configured to connect to any Su
30
30
31
31
It reads events in all blocks using [subxt](https://github.com/paritytech/subxt) and index these events in a Key-value database using the [sled](http://sled.rs/) library. This is considerably more efficient than storing the index in an SQL database.
32
32
33
-
subxt currently has an [issue](https://github.com/paritytech/subxt/issues/793#issuecomment-1386902010) where it is not possible to query blocks prior to V14 metadata (block #7,229,126 on Polkadot). Resolving this issue is not within the scope of the grant. Once this grant is completed a further grant application will be made that includes resolving this issue.
34
-
35
-
When decoding events, subxt needs to have the correct metadata. The metadata changes whenever a chain performs a runtime upgrade. Hybrid Indexer handles this in a very elegant way. When indexing begins it downloads the metadata for the starting block. When it encounters a decoding error it downloads the metadata for the current block and retries decoding. This means that the indexer does not have to be built with the metadata and block number of every runtime upgrade.
36
-
37
-
To index an event, it needs to be converted into a Rust type that matches the metadata. Sometimes the metadata for an event will change during a runtime upgrade. To handle this the indexer will have Rust types for current and historic versions of all events. When an event fails to be converted into a Rust type the previous type will be tried.
38
-
39
33
All events in all pallets that have identifying parameters will be indexed. For example the Transfer event in the Balances pallet is identifiable by the `AccountId` of both `from` and `to`.
40
34
41
35
Other examples of identifying event parameters are `assetId` in the Assets pallet, `code_hash` in the contracts pallet, `CollectionId` and `ItemId` in the NFTs pallet, and `MultiLocation` in the XCM pallet.
42
36
43
37
Additionally, all events are indexed by event variant.
44
38
45
-
To download a block, a query first has to be made to determine the hash for a given block number. In order to ensure throughput is as high as possible, multiple queries to the full node will be active at the same time to avoid round-trip delay. Block processing will be in a separate thread.
39
+
To download a block, a query first has to be made to determine the hash for a given block number. In order to ensure throughput is as high as possible, multiple queries to the full node will be active at the same time to avoid round-trip delay.
46
40
47
41
In the same manner that each Substrate chain is a separate Rust build that uses Substrate crates, each chain will need a separate Hybrid Indexer build that is configured to index the correct pallets.
48
42
49
43
When a chain is going to potentially perform a runtime upgrade, the Hybrid Indexer for the chain will need a new release with any updated events. If an instance of the indexer is not updated before the runtime upgrade occurs, it can be restarted with the new version at the correct block number.
50
44
51
-
WSS queries will be handled via the highly scalable [tokio_tungstenite](https://github.com/snapview/tokio-tungstenite) Rust library.
45
+
WSS queries are handled via the highly scalable [tokio_tungstenite](https://github.com/snapview/tokio-tungstenite) Rust library.
52
46
53
47
In addition to the identifier being searched for, queries will be able to include start block, offset, and limit to control which events are returned.
54
48
@@ -59,126 +53,3 @@ The database keys will be constructed in such a way so that events can be found
59
53
`AccountId/BlockNumber/EventIndex`
60
54
61
55
Database entries will be key-only. No value will be stored. The blocknumber and event index are all that need to be returned for each event found. This reduces the size of the index database and increases decentralization. The frontend can query the chain in a decentralized manner to retrieve the event.
62
-
63
-
### Tutorial
64
-
65
-
Hybrid is still early in the development process. Currently it can index event parameters from many events in Polkadot: AccountId, AccountIndex, AuctionIndex, BountyIndex, CandidateHash, EraIndex, MessageId, ParaId, PoolId, PreimageHash, ProposalHash, ProposalIndex, RefIndex, RegistrarIndex, SessionIndex, TipHash. Additionally, all events are indexed by event variant.
The earlist block that can be indexed on Polkadot is 7,229,126.
86
-
87
-
The indexer simultaneously indexes historical blocks (batch) and finalized blocks (head).
88
-
89
-
When the indexer starts it will start batch indexing from where indexing finished last time it was run. This can be overidden with the --block-height parameter.
90
-
91
-
Head blocks are always indexed as they are finalized. Once batch indexing has caught up with head it will stop and only blocks being finalized will be indexed.
92
-
93
-
#### Run the dapp
94
-
95
-
Go to [hybrid-dapp](https://github.com/hybrid-explorer/hybrid-dapp/tree/milestone-2) and follow the tutorial to run the frontend to query the indexer.
96
-
97
-
### Docker
98
-
99
-
After setting up Docker, run the following command to build the image:
100
-
101
-
```
102
-
docker build .
103
-
```
104
-
105
-
Then run the image with the correct port mapping:
106
-
107
-
```
108
-
docker run --rm -p 8172:8172 [image_hash]
109
-
```
110
-
111
-
Now run the dockerfile for [hybrid-dapp](https://github.com/hybrid-explorer/hybrid-dapp/tree/milestone-2#docker).
112
-
113
-
### Testing Guide
114
-
115
-
Ensure that you have both the the indexer and dapp running, either by following the above tutorial, or by following the Docker instructions.
116
-
117
-
#### Deliverable 2.1 & 2.2
118
-
119
-
Event subscription API / Live dapp
120
-
121
-
1. Select Pallet / Variant search key.
122
-
2. Select Balances pallet.
123
-
3. Select Transfer variant.
124
-
4. Click Search.
125
-
5. Observe that as new balance transfers occur on Polkadot, they automatically appear in the results.
126
-
127
-
#### Deliverable 2.3
128
-
129
-
preimage_hash, era_index and session_index are now indexed in addition to the keys in milestone 1.
130
-
131
-
Using the dapp, test the following search queries and verify the results:
0 commit comments