Metadata crawler improvments #665
Conversation
Andrei-Dolgolev
changed the title
| @@ -38,8 +44,14 @@ def crawl_uri(metadata_uri: str) -> Any: | |||
| result = None | |||
| while retry < 3: | |||
There was a problem hiding this comment.
I sugest to re-write this func to avoid increment of retry var only in exceptions. But would do this by default at the end of loop and if response.status == 200: then write result and break the while loop.
But if this version works well, then probably there are no need any changes.
| already_parsed = get_current_metadata_for_address( | ||
| db_session=db_session, blockchain_type=blockchain_type, address=address | ||
| logger.info( | ||
| f"Start crawling {len(not_updated_tokens)} tokens of address {address}" | ||
| ) | ||
|
|
||
| for requests_chunk in [ |
There was a problem hiding this comment.
Hard to read, probably better to add another variable on top of this for loop.
|
|
||
| if token_uri_data.token_id not in already_parsed: | ||
| metadata = crawl_uri(token_uri_data.token_uri) | ||
| with ThreadPoolExecutor( |
There was a problem hiding this comment.
With thread pool executor better to store results and wait them like in this code https://github.com/bugout-dev/moonstream/blob/e5240d2fd85685383bb40ea4cad741ec90b4fd6d/crawlers/mooncrawl/mooncrawl/blockchain.py#L367
There was a problem hiding this comment.
So this threads which inside another loop from list comprehension range will be under better control))
Changes
How to test these changes?
Related issues