perf: rm pending queue from MultiproofManager #19178
Conversation
mattsse
added
the
C-perf
yongkangc
force-pushed
the
yk/remove_pending
branch
from
4db576c to
22715cc
Compare
yongkangc
added
the
S-needs-benchmark
yongkangc
changed the title
yongkangc
requested review from
Rjected,
fgimenez,
gakonst,
klkvr,
mattsse,
mediocregopher and
rkrasiuk
as code owners
yongkangc
changed the title
| chunks += 1; | ||
| }; | ||
|
|
||
| if should_chunk && let Some(chunk_size) = self.chunk_size { |
There was a problem hiding this comment.
leaving a comment to not forget to need to fix this, because now we always chunk even if all workers are busy
we should look into this for this PR leaving a note (cc @shekhirin @mediocregopher @mattsse ) ref: |
|
@mattsse @shekhirin @mediocregopher just added in a way for us to track avaliable worker and chunk if we have avaliable worker. In the future we should prob go for dynamic logic but this is a simple one for now |
This reverts commit 9419424.
- Introduced a counter for available workers to manage task processing. - Updated multiproof spawning logic to utilize worker availability for chunking. - Enhanced worker decrement/increment logic during task execution.
…ultiproof manager - Eliminated the worker count variable and associated logic from the multiproof manager. - Updated multiproof spawning to check for available account workers instead of a general worker count. - Cleaned up related comments and code for clarity and maintainability.
- We no longer seed the counters with the worker count. In ProofWorkerHandle::new we switched to zero-initializing both atomics
yongkangc
force-pushed
the
yk/remove_pending
branch
from
9b484a2 to
74d8d3c
Compare
- Added histograms for pending storage and account tasks in the MultiproofManager. - Implemented methods in ProofWorkerHandle to retrieve the count of pending storage and account tasks. - Updated multiproof processing logic to record these new metrics.
- Added recording of pending storage and account tasks in the MultiproofManager upon calculation completion. - Enhanced metrics tracking to provide better insights into task management.
BenchmarksMainnet Performance Changes: Base: Statistics: |
closes: #19177
This is a follow up from #19171 where we removed the cli args. here, we remove the inflight limits completely.
The core changes are instead of manually allocating task from a VecDeque, we make use of the unbounded crossbeam channel as queue to allocate tasks and perform backpressure.