Shim new streaming backend#212
Conversation
|
Added almost all functionality. I'm trying very hard to maintain existing acquire-zarr behavior. One thing that will have to change is some of the downsampling behavior. acquire-zarr treats x and y in a special way, where chucky does not. This impacts some rules determining when to stop downsampling certain dimensions as we get to higher (coarser) levels of detail (lod). If the user doesn't want to allow the aspect of ratios of dimensions to change, then we must stop downsampling as soon as any dimension becomes too small; "Too small" means it's one chuck or fewer in that dimension. If aspect ratios can change, then we just stop downsampling dimensions that are too small and continue with the others until there are no dimensions left - everything fits in one chunk. This is almost what acquire-zarr does. The aspect ratio of x and y are held fixed, but the others can change. Chunk sizes also vary. In chucky, variable chunk size causes all kinds of headaches - so I'll constrain to a constant chunk size. I'll add an option letting a user select whether they want a variable aspect ratio or not. I've already got a max lod parameter. The lod's will be computed until any/all dimensions are 1 or fewer chunks wide or the max lod is hit. |
aliddell
left a comment
aliddell
left a comment
There was a problem hiding this comment.
Couple of things need looking at.
| assert np.array_equal(array, data) | ||
|
|
||
|
|
||
| @pytest.mark.timeout(300) |
There was a problem hiding this comment.
not when things are working :D I vote for keeping it.
| .idea | ||
| .vscode | ||
| .vs | ||
| .claude |
There was a problem hiding this comment.
This is going to get out of date very quickly (if it isn't already) and it probably won't be relevant after merge. Consider deleting this file and moving the contents to the PR description instead.
| const char* metadata_key, | ||
| const char* metadata) | ||
| { | ||
| if (!stream || !metadata_key || !metadata) { |
There was a problem hiding this comment.
Rejecting NULL-valued metadata_key is a behavior change. It should be accepted if and only if we have just the one array.
| const ZarrArraySettings* as = &settings->arrays[index]; | ||
| if (!as->output_key) { | ||
| *key = NULL; | ||
| return ZarrStatusCode_Success; |
There was a problem hiding this comment.
Seems wrong to return Success with a NULL-valued *key.
On closer inspection, the original code dedupes and regularizes the keys, so the behavior is changed here.
Co-authored-by: Alan Liddell <[email protected]>
Co-authored-by: Alan Liddell <[email protected]>
Co-authored-by: Alan Liddell <[email protected]>
2 participants
chucky is a new streaming backend for acquire.
It adds support for gpu acceleration and there are some improvements to cpu-based streaming as well.
Getting it in is going to take a few steps.
I'm starting with this PR by creating a "shim". The shim is a thin layer implementing the public c api of acquire using chucky. I will use that to gradually walk through acquire-zarr's tests, changing chucky as necessary.
This should allow us to make some standalone libraries/wheels we can test for a bit and identify when to switch the back-ends more permanently.