[WIP] feat: separate performance collection and distribution #802
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
vgorkavenko
changed the title
…collector # Conflicts: # poetry.lock # pyproject.toml # src/main.py # src/modules/csm/csm.py # src/web3py/types.py # tests/fork/conftest.py
| "total_validators": len(validators) | ||
| }) | ||
|
|
||
| for l_epoch, r_epoch in self.state.frames: |
There was a problem hiding this comment.
How to understand that self.state.frames is not empty?
I mean, looks like fulfill_state need some pre-requirments before run. If yes - better to check this requirments and drop exceptions if the requirements are not met.
| }) | ||
|
|
||
| for epoch in sequence(l_epoch, r_epoch): | ||
| if epoch not in self.state.unprocessed_epochs: |
src/modules/csm/csm.py
Outdated
| if epoch_data is None: | ||
| raise ValueError(f"Epoch {epoch} is missing in Performance Collector") | ||
|
|
||
| misses, props_vids, props_flags, syncs_vids, syncs_misses = ( |
| if is_pivot_missing: | ||
| br[pivot_index] = None | ||
|
|
||
| logger.debug({ |
There was a problem hiding this comment.
| logger.debug({ | |
| logger.info({ |
maybe
| sync_bits = hex_bitvector_to_list(sync_aggregate.sync_committee_bits) | ||
| for index_in_committee in get_set_indices(sync_bits): | ||
| committee[index_in_committee].included = True | ||
| # Go through only UNSET indexes to get misses |
There was a problem hiding this comment.
Comments should answer "why", not "what". Code represents "what" by itself.
src/modules/performance/common/db.py
Outdated
| present_duties = session.exec( | ||
| select(Duty.epoch).where(Duty.epoch >= l_epoch, Duty.epoch <= r_epoch).order_by(Duty.epoch) | ||
| ).all() | ||
| present = [int(epoch) for epoch in present_duties] |
There was a problem hiding this comment.
present could be set. if epoch not in present is not optimal so much if present is list.
| db: DutiesDB = Depends(get_db), | ||
| ): | ||
| await validate_epoch_bounds(from_epoch, to_epoch) | ||
| return bool(db.is_range_available(from_epoch, to_epoch)) |
There was a problem hiding this comment.
If you want to use async view, your queries to DB must be async as well, or you will block event loop.
Remove async from views, or add async/await to DutiesDB methods.
| _db_instance: Optional[DutiesDB] = None | ||
|
|
||
|
|
||
| async def get_db() -> DutiesDB: |
There was a problem hiding this comment.
There is probability of race condition here. As i understand, the main approach in fastapi for this is using lifespan function.
|
|
||
| @app.get("/epochs", response_model=list[Duty]) | ||
| async def epochs_data( | ||
| from_epoch: EpochNumber = Query(..., alias="from"), |
There was a problem hiding this comment.
Need limits on from/to + timeouts
|
|
||
| uvicorn.run( | ||
| app, | ||
| host="0.0.0.0", |
Description
Separate Performance Collection module for CSMv3 and CMv2 Oracle daemons
https://hackmd.io/OkdagPjpTYKm7yN9sqHGhA?both
Related Issue/Task
How Has This Been Tested?
Describe how you tested the changes:
pytest)Checklist
CSM_STATE_VERSIONis bumped (if the new version affects data in the cache)