Repository navigation
Parallelise operations - #51
benjeffery-ofh wants to merge 1 commit into
Conversation
|
Initial findings are that this is 200x faster for the copy up operation |
tomwhite
left a comment
tomwhite
left a comment
There was a problem hiding this comment.
Great work on speeding things up! I had a few preliminary comments.
In addition to the asyncio changes, this adds CLI tests, and checks variant chunk alignment - both of which are orthogonal to the main change. Would it be possible to break up the changes?
| assert seen["io_concurrency"] == 3 | ||
|
|
||
|
|
||
| def test_old_copy_concurrency_flag_is_removed(): |
| return array | ||
|
|
||
|
|
||
| def _assert_variant_chunk_alignment(arrays, *, variant_chunk_size, operation): |
There was a problem hiding this comment.
It should be possible to make this function work for all operations - given a set of target arrays check they all have the given chunking.
| import inspect | ||
| import os | ||
|
|
||
| DEFAULT_IO_CONCURRENCY = 32 |
There was a problem hiding this comment.
What's the rationale for this number? Should it be a small factor times the number of cores on the machine (like in bio2zarr)?
|
Hi @benjeffery-ofh - do you have any updates to this PR? It would be good to get it merged (or at least parts of it), as it's a big change and I've got a series of changes I'd like to make that will conflict. |
|
I'll break this out into the separate PRs |
|
Thanks! |
|
Closed in favour of smaller PRs |
WIP