Support data layout of DynaCell acquisition with HCS writer

[mattersoflight]

This issue is meant to be a discussion of one use case of acquire-zarr's features for writing data in HCS OME Zarr format. What features are needed in acquire-zarr to enable acquisition of data with the following layout? Which are already implemented and which ones will need to be developed?

We can also change the data layout to reduce the complexity of implementation and improve generalization to multiple use cases.

Background
DynaCell datasets are acquired with Mantis (https://pubmed.ncbi.nlm.nih.gov/39282007/), which has a label-free and a light-sheet arm, which can have different TCZYX dimensions at the time of acquisition. In this case, each field of view is imaged simultaneously with both arms. The number of FOVs in each well can vary from well to well.

Layout of acquisition data

This is the layout of two zarr stores (one per camera) after the NDTiff data is converted to ome-zarr format. We would like to be able to write directly to zarr. These channels get processed through different steps (see, biahub) before converging into a single multi-channel zarr store that is shared with DynaCell dataset. biahub code can be updated to use a better layout as an input if we identify one.
This data can be found at:
/hpc/instruments/cm.mantis/2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV/0-convert/2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV_chunked_1

├── 2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV_chunked_labelfree_1.zarr
│   └── 0
│       ├── 1
│       │   ├── 000000
│       │   ├── 000001
│       │   ├── 001000
│       │   ├── 001001
│       │   ├── 002000
│       │   └── 002001
│       ├── 2
│       │   ├── 000000
│       │   ├── 000001
│       │   ├── 001000
│       │   ├── 001001
│       │   ├── 002000
│       │   └── 002001
│       ├── 3
│       │   ├── 000000
│       │   ├── 000001
│       │   ├── 001000
│       │   └── 001001
│       ├── 4
│       │   ├── 000000
│       │   ├── 000001
│       │   ├── 001000
│       │   ├── 001001
│       │   ├── 002000
│       │   └── 002001
│       ├── 5
│       │   ├── 000000
│       │   ├── 000001
│       │   ├── 001000
│       │   ├── 001001
│       │   ├── 002000
│       │   └── 002001
│       ├── 6
│       │   ├── 000000
│       │   ├── 000001
│       │   ├── 001000
│       │   ├── 001001
│       │   ├── 002000
│       │   └── 002001
│       └── 7
│           ├── 000000
│           ├── 000001
│           ├── 001000
│           ├── 001001
│           ├── 002000
│           └── 002001
└── 2025_07_24_A549_SEC61_TOMM20_G3BP1_ZIKV_chunked_lightsheet_1.zarr
    └── 0
        ├── 1
        │   ├── 000000
        │   ├── 000001
        │   ├── 001000
        │   ├── 001001
        │   ├── 002000
        │   └── 002001
        ├── 2
        │   ├── 000000
        │   ├── 000001
        │   ├── 001000
        │   ├── 001001
        │   ├── 002000
        │   └── 002001
        ├── 3
        │   ├── 000000
        │   ├── 000001
        │   ├── 001000
        │   └── 001001
        ├── 4
        │   ├── 000000
        │   ├── 000001
        │   ├── 001000
        │   ├── 001001
        │   ├── 002000
        │   └── 002001
        ├── 5
        │   ├── 000000
        │   ├── 000001
        │   ├── 001000
        │   ├── 001001
        │   ├── 002000
        │   └── 002001
        ├── 6
        │   ├── 000000
        │   ├── 000001
        │   ├── 001000
        │   ├── 001001
        │   ├── 002000
        │   └── 002001
        └── 7
            ├── 000000
            ├── 000001
            ├── 001000
            ├── 001001
            ├── 002000
            └── 002001

[ieivanov]

Here is another example of a slightly more typical experiment, which includes a well where we only image one FOV:

Reading file:	 /hpc/instruments/cm.mantis/2024_11_07_A549_SEC61_ZIKV_DENV/0-convert/2024_11_07_A549_SEC61_ZIKV_DENV_chunked_1/2024_11_07_A549_SEC61_ZIKV_DENV_chunked_labelfree_1.zarr
Zarr hierarchy:
/
 └── 0
     ├── 1
     │   ├── 000000
     │   │   └── 0 (25, 1, 105, 1496, 1496) uint16
     │   ├── 000001
     │   │   └── 0 (25, 1, 105, 1496, 1496) uint16
     │   ├── 001000
     │   │   └── 0 (25, 1, 105, 1496, 1496) uint16
     │   └── 001001
     │       └── 0 (25, 1, 105, 1496, 1496) uint16
     ├── 2
     │   ├── 000000
     │   │   └── 0 (25, 1, 105, 1496, 1496) uint16
     │   ├── 000001
     │   │   └── 0 (25, 1, 105, 1496, 1496) uint16
     │   ├── 001000
     │   │   └── 0 (25, 1, 105, 1496, 1496) uint16
     │   └── 001001
     │       └── 0 (25, 1, 105, 1496, 1496) uint16
     ├── 3
     │   ├── 000000
     │   │   └── 0 (25, 1, 105, 1496, 1496) uint16
     │   ├── 000001
     │   │   └── 0 (25, 1, 105, 1496, 1496) uint16
     │   ├── 001000
     │   │   └── 0 (25, 1, 105, 1496, 1496) uint16
     │   └── 001001
     │       └── 0 (25, 1, 105, 1496, 1496) uint16
     ├── 4
     │   ├── 000000
     │   │   └── 0 (25, 1, 105, 1496, 1496) uint16
     │   ├── 000001
     │   │   └── 0 (25, 1, 105, 1496, 1496) uint16
     │   ├── 001000
     │   │   └── 0 (25, 1, 105, 1496, 1496) uint16
     │   └── 001001
     │       └── 0 (25, 1, 105, 1496, 1496) uint16
     └── 8
         └── 000000
             └── 0 (25, 1, 105, 1496, 1496) uint16

=== Summary ===
Format:			 omezarr v0.4
Axes:			 T (time); C (channel); Z (space); Y (space); X (space); 
Channel names:		 ['BF']
Row names:		 ['0']
Column names:		 ['1', '2', '3', '4', '8']
Wells:			 5
Positions:		 17
Chunk size:		 (1, 1, 105, 1496, 1496)
No. bytes decompressed:		 199742928000 [186.0 GiB]

This datset can be opened with iohub in Python code:

>>> from iohub import open_ome_zarr
>>> dataset = open_ome_zarr('/hpc/instruments/cm.mantis/2024_11_07_A549_SEC61_ZIKV_DENV/0-convert/2024_11_07_A549_SEC61_ZIKV_DENV_chunked_1/2024_11_07_A549_SEC61_ZIKV_DENV_chunked_labelfree_1.zarr', mode='r')

Reading file:	 /hpc/instruments/cm.mantis/2024_11_07_A549_SEC61_ZIKV_DENV/0-convert/2024_11_07_A549_SEC61_ZIKV_DENV_chunked_1/2024_11_07_A549_SEC61_ZIKV_DENV_chunked_lightsheet_1.zarr
Zarr hierarchy:
/
 └── 0
     ├── 1
     │   ├── 000000
     │   │   └── 0 (25, 2, 1168, 300, 2048) uint16
     │   ├── 000001
     │   │   └── 0 (25, 2, 1168, 300, 2048) uint16
     │   ├── 001000
     │   │   └── 0 (25, 2, 1168, 300, 2048) uint16
     │   └── 001001
     │       └── 0 (25, 2, 1168, 300, 2048) uint16
     ├── 2
     │   ├── 000000
     │   │   └── 0 (25, 2, 1168, 300, 2048) uint16
     │   ├── 000001
     │   │   └── 0 (25, 2, 1168, 300, 2048) uint16
     │   ├── 001000
     │   │   └── 0 (25, 2, 1168, 300, 2048) uint16
     │   └── 001001
     │       └── 0 (25, 2, 1168, 300, 2048) uint16
     ├── 3
     │   ├── 000000
     │   │   └── 0 (25, 2, 1168, 300, 2048) uint16
     │   ├── 000001
     │   │   └── 0 (25, 2, 1168, 300, 2048) uint16
     │   ├── 001000
     │   │   └── 0 (25, 2, 1168, 300, 2048) uint16
     │   └── 001001
     │       └── 0 (25, 2, 1168, 300, 2048) uint16
     ├── 4
     │   ├── 000000
     │   │   └── 0 (25, 2, 1168, 300, 2048) uint16
     │   ├── 000001
     │   │   └── 0 (25, 2, 1168, 300, 2048) uint16
     │   ├── 001000
     │   │   └── 0 (25, 2, 1168, 300, 2048) uint16
     │   └── 001001
     │       └── 0 (25, 2, 1168, 300, 2048) uint16
     └── 8
         └── 000000
             └── 0 (25, 2, 1168, 300, 2048) uint16

=== Summary ===
Format:			 omezarr v0.4
Axes:			 T (time); C (channel); Z (space); Y (space); X (space); 
Channel names:		 ['GFP EX488 EM525-45', 'mCherry EX561 EM600-37']
Row names:		 ['0']
Column names:		 ['1', '2', '3', '4', '8']
Wells:			 5
Positions:		 17
Chunk size:		 (1, 1, 292, 300, 2048)
No. bytes decompressed:		 1219952640000 [1.1 TiB]

This datset can be opened with iohub in Python code:

>>> from iohub import open_ome_zarr
>>> dataset = open_ome_zarr('/hpc/instruments/cm.mantis/2024_11_07_A549_SEC61_ZIKV_DENV/0-convert/2024_11_07_A549_SEC61_ZIKV_DENV_chunked_1/2024_11_07_A549_SEC61_ZIKV_DENV_chunked_lightsheet_1.zarr', mode='r')

[ieivanov]

Our zebrafish experiments follow a very similar layout:

Reading file:	 /hpc/instruments/cm.mantis/2025_08_01_zebrafish_golden_trio/0-convert/2025_08_01_zebrafish_golden_trio_chunked_1/2025_08_01_zebrafish_golden_trio_chunked_labelfree_1.zarr
Zarr hierarchy:
/
 └── 0
     ├── 1
     │   ├── 000000
     │   │   └── 0 (50, 1, 145, 750, 1024) uint16
     │   ├── 001000
     │   │   └── 0 (50, 1, 145, 750, 1024) uint16
     │   ├── 002000
     │   │   └── 0 (50, 1, 145, 750, 1024) uint16
     │   └── 003000
     │       └── 0 (50, 1, 145, 750, 1024) uint16
     ├── 2
     │   ├── 000000
     │   │   └── 0 (50, 1, 145, 750, 1024) uint16
     │   ├── 001000
     │   │   └── 0 (50, 1, 145, 750, 1024) uint16
     │   └── 002000
     │       └── 0 (50, 1, 145, 750, 1024) uint16
     ├── 3
     │   ├── 000000
     │   │   └── 0 (50, 1, 145, 750, 1024) uint16
     │   ├── 001000
     │   │   └── 0 (50, 1, 145, 750, 1024) uint16
     │   ├── 002000
     │   │   └── 0 (50, 1, 145, 750, 1024) uint16
     │   └── 003000
     │       └── 0 (50, 1, 145, 750, 1024) uint16
     └── 8
         └── 000000
             └── 0 (50, 1, 145, 750, 1024) uint16

=== Summary ===
Format:			 omezarr v0.4
Axes:			 T (time); C (channel); Z (space); Y (space); X (space); 
Channel names:		 ['BF']
Row names:		 ['0']
Column names:		 ['1', '2', '3', '8']
Wells:			 4
Positions:		 12
Chunk size:		 (1, 1, 145, 750, 1024)
No. bytes decompressed:		 133632000000 [124.5 GiB]

This datset can be opened with iohub in Python code:

>>> from iohub import open_ome_zarr
>>> dataset = open_ome_zarr('/hpc/instruments/cm.mantis/2025_08_01_zebrafish_golden_trio/0-convert/2025_08_01_zebrafish_golden_trio_chunked_1/2025_08_01_zebrafish_golden_trio_chunked_labelfree_1.zarr', mode='r')

Reading file:	 /hpc/instruments/cm.mantis/2025_08_01_zebrafish_golden_trio/0-convert/2025_08_01_zebrafish_golden_trio_chunked_1/2025_08_01_zebrafish_golden_trio_chunked_lightsheet_1.zarr
Zarr hierarchy:
/
 └── 0
     ├── 1
     │   ├── 000000
     │   │   └── 0 (50, 2, 401, 500, 1024) uint16
     │   ├── 001000
     │   │   └── 0 (50, 2, 401, 500, 1024) uint16
     │   ├── 002000
     │   │   └── 0 (50, 2, 401, 500, 1024) uint16
     │   └── 003000
     │       └── 0 (50, 2, 401, 500, 1024) uint16
     ├── 2
     │   ├── 000000
     │   │   └── 0 (50, 2, 401, 500, 1024) uint16
     │   ├── 001000
     │   │   └── 0 (50, 2, 401, 500, 1024) uint16
     │   └── 002000
     │       └── 0 (50, 2, 401, 500, 1024) uint16
     ├── 3
     │   ├── 000000
     │   │   └── 0 (50, 2, 401, 500, 1024) uint16
     │   ├── 001000
     │   │   └── 0 (50, 2, 401, 500, 1024) uint16
     │   ├── 002000
     │   │   └── 0 (50, 2, 401, 500, 1024) uint16
     │   └── 003000
     │       └── 0 (50, 2, 401, 500, 1024) uint16
     └── 8
         └── 000000
             └── 0 (50, 2, 401, 500, 1024) uint16

=== Summary ===
Format:			 omezarr v0.4
Axes:			 T (time); C (channel); Z (space); Y (space); X (space); 
Channel names:		 ['GFP EX488 EM525-45', 'mCherry EX561 EM600-37']
Row names:		 ['0']
Column names:		 ['1', '2', '3', '8']
Wells:			 4
Positions:		 12
Chunk size:		 (1, 1, 401, 500, 1024)
No. bytes decompressed:		 492748800000 [458.9 GiB]

This datset can be opened with iohub in Python code:

>>> from iohub import open_ome_zarr
>>> dataset = open_ome_zarr('/hpc/instruments/cm.mantis/2025_08_01_zebrafish_golden_trio/0-convert/2025_08_01_zebrafish_golden_trio_chunked_1/2025_08_01_zebrafish_golden_trio_chunked_lightsheet_1.zarr', mode='r')

Here each "well" is a fish and we image 8 neuromasts per fish.

[aliddell]

@ieivanov Can you explain the scheme for naming the FOVs? And is each FOV genuinely multiscale or does each array just live in a single directory called 0 nested under each FOV?

[ieivanov]

During acquisition time we (currently) do not compute/acquire multiscales - each FOV has a single array at the 0 level.

In the future it may be useful to do so - in our Optical Pooled Screening (OPS) experiments we image the same sample at 5x and then at 20x. We currently save these acquisitions separately, but it may be helpful to leverage the zarr multiscales for that. Not a high priority for us, but I wanted to point out that such a use case is possible at acquisition time as well.