Skip to content

Commit 2d82f8e

Browse files
committed
Update README.md
1 parent c0a2028 commit 2d82f8e

File tree

1 file changed

+2
-131
lines changed

1 file changed

+2
-131
lines changed

β€ŽREADME.md

Lines changed: 2 additions & 131 deletions
Original file line numberDiff line numberDiff line change
@@ -30,25 +30,19 @@ The Hybrid indexer is written in Rust. It can be configured to connect to any Su
3030

3131
It reads events in all blocks using [subxt](https://github.com/paritytech/subxt) and index these events in a Key-value database using the [sled](http://sled.rs/) library. This is considerably more efficient than storing the index in an SQL database.
3232

33-
subxt currently has an [issue](https://github.com/paritytech/subxt/issues/793#issuecomment-1386902010) where it is not possible to query blocks prior to V14 metadata (block #7,229,126 on Polkadot). Resolving this issue is not within the scope of the grant. Once this grant is completed a further grant application will be made that includes resolving this issue.
34-
35-
When decoding events, subxt needs to have the correct metadata. The metadata changes whenever a chain performs a runtime upgrade. Hybrid Indexer handles this in a very elegant way. When indexing begins it downloads the metadata for the starting block. When it encounters a decoding error it downloads the metadata for the current block and retries decoding. This means that the indexer does not have to be built with the metadata and block number of every runtime upgrade.
36-
37-
To index an event, it needs to be converted into a Rust type that matches the metadata. Sometimes the metadata for an event will change during a runtime upgrade. To handle this the indexer will have Rust types for current and historic versions of all events. When an event fails to be converted into a Rust type the previous type will be tried.
38-
3933
All events in all pallets that have identifying parameters will be indexed. For example the Transfer event in the Balances pallet is identifiable by the `AccountId` of both `from` and `to`.
4034

4135
Other examples of identifying event parameters are `assetId` in the Assets pallet, `code_hash` in the contracts pallet, `CollectionId` and `ItemId` in the NFTs pallet, and `MultiLocation` in the XCM pallet.
4236

4337
Additionally, all events are indexed by event variant.
4438

45-
To download a block, a query first has to be made to determine the hash for a given block number. In order to ensure throughput is as high as possible, multiple queries to the full node will be active at the same time to avoid round-trip delay. Block processing will be in a separate thread.
39+
To download a block, a query first has to be made to determine the hash for a given block number. In order to ensure throughput is as high as possible, multiple queries to the full node will be active at the same time to avoid round-trip delay.
4640

4741
In the same manner that each Substrate chain is a separate Rust build that uses Substrate crates, each chain will need a separate Hybrid Indexer build that is configured to index the correct pallets.
4842

4943
When a chain is going to potentially perform a runtime upgrade, the Hybrid Indexer for the chain will need a new release with any updated events. If an instance of the indexer is not updated before the runtime upgrade occurs, it can be restarted with the new version at the correct block number.
5044

51-
WSS queries will be handled via the highly scalable [tokio_tungstenite](https://github.com/snapview/tokio-tungstenite) Rust library.
45+
WSS queries are handled via the highly scalable [tokio_tungstenite](https://github.com/snapview/tokio-tungstenite) Rust library.
5246

5347
In addition to the identifier being searched for, queries will be able to include start block, offset, and limit to control which events are returned.
5448

@@ -59,126 +53,3 @@ The database keys will be constructed in such a way so that events can be found
5953
`AccountId/BlockNumber/EventIndex`
6054

6155
Database entries will be key-only. No value will be stored. The blocknumber and event index are all that need to be returned for each event found. This reduces the size of the index database and increases decentralization. The frontend can query the chain in a decentralized manner to retrieve the event.
62-
63-
### Tutorial
64-
65-
Hybrid is still early in the development process. Currently it can index event parameters from many events in Polkadot: AccountId, AccountIndex, AuctionIndex, BountyIndex, CandidateHash, EraIndex, MessageId, ParaId, PoolId, PreimageHash, ProposalHash, ProposalIndex, RefIndex, RegistrarIndex, SessionIndex, TipHash. Additionally, all events are indexed by event variant.
66-
67-
#### Run tests
68-
69-
```
70-
cargo test
71-
```
72-
73-
#### Build
74-
75-
```
76-
cargo build --release
77-
```
78-
79-
#### Run indexer
80-
81-
```
82-
./target/release/hybrid-indexer --block-height 15870000
83-
```
84-
85-
The earlist block that can be indexed on Polkadot is 7,229,126.
86-
87-
The indexer simultaneously indexes historical blocks (batch) and finalized blocks (head).
88-
89-
When the indexer starts it will start batch indexing from where indexing finished last time it was run. This can be overidden with the --block-height parameter.
90-
91-
Head blocks are always indexed as they are finalized. Once batch indexing has caught up with head it will stop and only blocks being finalized will be indexed.
92-
93-
#### Run the dapp
94-
95-
Go to [hybrid-dapp](https://github.com/hybrid-explorer/hybrid-dapp/tree/milestone-2) and follow the tutorial to run the frontend to query the indexer.
96-
97-
### Docker
98-
99-
After setting up Docker, run the following command to build the image:
100-
101-
```
102-
docker build .
103-
```
104-
105-
Then run the image with the correct port mapping:
106-
107-
```
108-
docker run --rm -p 8172:8172 [image_hash]
109-
```
110-
111-
Now run the dockerfile for [hybrid-dapp](https://github.com/hybrid-explorer/hybrid-dapp/tree/milestone-2#docker).
112-
113-
### Testing Guide
114-
115-
Ensure that you have both the the indexer and dapp running, either by following the above tutorial, or by following the Docker instructions.
116-
117-
#### Deliverable 2.1 & 2.2
118-
119-
Event subscription API / Live dapp
120-
121-
1. Select Pallet / Variant search key.
122-
2. Select Balances pallet.
123-
3. Select Transfer variant.
124-
4. Click Search.
125-
5. Observe that as new balance transfers occur on Polkadot, they automatically appear in the results.
126-
127-
#### Deliverable 2.3
128-
129-
preimage_hash, era_index and session_index are now indexed in addition to the keys in milestone 1.
130-
131-
Using the dapp, test the following search queries and verify the results:
132-
133-
```
134-
PreimageHash: 0xdb2b6cb38c2f6704ed067da2e9001bc57314be4f0117f664a93c0d18610110c5
135-
15764612 preimage Noted A preimage has been noted.
136-
hash_: 0xdb2b6cb38c2f6704ed067da2e9001bc57314be4f0117f664a93c0d18610110c5
137-
```
138-
139-
```
140-
EraIndex: 1076
141-
15825858 fastUnstake BatchChecked A batch was partially checked for the given eras, but the process did not finish.
142-
eras: [ "1,076" ]
143-
```
144-
145-
```
146-
SessionIndex: 6552
147-
15649648 session NewSession New session has happened. Note that the argument is the session index, not the block number as the type might suggest.
148-
sessionIndex: 6,552
149-
```
150-
151-
#### Deliverable 2.4
152-
153-
All events are now indexed by their variant.
154-
155-
1. Select Pallet / Variant search key.
156-
2. Select System pallet.
157-
3. Select NewAccount variant.
158-
4. Click Search.
159-
5. Observe that events with the selected variant are found.
160-
161-
#### Deliverable 2.5
162-
163-
Event contents are now loaded directly from the chain, rather than from the indexer.
164-
165-
Observe that when searching for events, the details column is populated with information about the event.
166-
167-
#### Deliverable 2.6
168-
169-
Asynchrous block downloading - blocks will be downloaded as fast as possible for improved indexing speed.
170-
171-
In the previous milestone the indexer downloaded blocks one at a time. Blocks indexed per second was in the single digits.
172-
173-
Now the indexer maintains up to 128 current requests. This results in approx 100x increase in block indexing rate.
174-
175-
Observe that when batch indexing, the indexer reports blocks per second in the triple digits.
176-
```
177-
πŸ“š #15870600, 155 blocks/sec
178-
πŸ“š #15870700, 179 blocks/sec
179-
πŸ“š #15870800, 136 blocks/sec
180-
πŸ“š #15870900, 158 blocks/sec
181-
πŸ“š #15871000, 284 blocks/sec
182-
πŸ“š #15871100, 161 blocks/sec
183-
πŸ“š #15871200, 168 blocks/sec
184-
```

0 commit comments

Comments
Β (0)