Skip to content

Allow multiple streams to share memory/resource pools #191

@aliddell

Description

@aliddell

Motivation

When writing multiple independent arrays (e.g., different positions in a multi-position acquisition), users may want streams to share

  • Memory buffers for chunk management, including the frame buffer
  • Thread pools for compression and I/O
  • S3 connection pools

Currently, each stream manages its own resources, which can be inefficient for many-stream scenarios and could force resource-conscious users into unorthodox solutions which require workarounds.

Use case

# Multiple independent arrays in one acquisition
with aqz.ResourcePool() as pool:
    streams = [
        aqz.ZarrStream(settings_pos0, resource_pool=pool),
        aqz.ZarrStream(settings_pos1, resource_pool=pool),
        # ... many positions
    ]
    
    for frame_data in acquisition:
        for i, stream in enumerate(streams):
            stream.append(frame_data[i])

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions