ArraySettings.output_key and downsampling_method shouldn't gate the presence/absence of ngff multiscales group

[tlambert03]

This is basically the same issue as #170, but extended to any array. And it also touches on the output_key issue mentioned in #181 (but i'll leave that issue to focus on docs, while this one focuses on the expected output). Briefly: there's too much magic going on when it comes to deciding what sort of zarr hierarchy to create (output_key and downsampling_method are both doing double duty in determining what the structure of the zarr hierarchy looks like)

Here's the TLDR; feature/bug-fix request, and the background/rationale below:

1. I think ArraySettings.output_key should only determine path, not structure (its presence/absence shouldn't be used to gate whether you get an array node or an ome multiscales group) (side-note: i do like that there is a way to directly write only the array node, but i think that result is currently a little conflated and unclear)
2. I think downsampling_method should only determine the downsampling method, and possibly whether or not downsampling occurs (its presence should also not be used to gate whether you get an array node or multiscales group)
3. chunk_size_px should not be used to determine how many downscaling levels you get (sure, it can determine what the maximum valid number of scales could be, but you should be able to select number of downscales without modifying the chunk layout of the top level array).

---

Note

Background: generally speaking, I think most typical experiments fall into one of three categories
(at least as far as what ome-zarr can currently represent)

1. A single ≤5D image (TCZYX, at a single position). Whether it has multiple scales or not is optional.
2. An HCS plate, where each FOV is a single ≤5D image
3. A collection of ≤5D images, usually corresponding to different positions. There's no great spec for this at the moment (there's an old ongoing discussion Collections Specification ome/ngff#31), but the best option at the moment is piggy backing on the bioformats2raw.layout transitional spec

This issue is mostly about 3 (collections), but touches on 1 (multiscales) images.

Generating a valid OME multiscales image for a single position is easy enough. Just use the standard pattern, but make sure not to provide output_key. This works (though it's semantically a bit confusing/magic ... and that's what #181 is discussing)

settings = StreamSettings(
    arrays=[
        ArraySettings(
            output_key="",  # any other value will NOT result in an OME multiscales group
            ...
        )
    ]
)

However, generating a collection of multiscale images is currently not possible, unless you also opt-in to downsampling.

This will NOT work... You will get root.zarr/Pos0 and root.zarr/Pos1, but providing output_key means that each of these is directly an Array node, no multiscales metadata is written.

settings = StreamSettings(
    arrays=[
        ArraySettings(
            output_key="Pos0",
            ...
        ),
        ArraySettings(
            output_key="Pos1",
            ...
        )
    ]
)

This will work, but it's a hack (the presence of downsampling_method shouldn't be related to group structure), and there's no way to say "i only want a single scale"... the number of downsampled scales is always automagically calculated from the chunking scheme.

settings = StreamSettings(
      arrays=[
          ArraySettings(
              output_key="Pos0",
              downsampling_method=DownsamplingMethod.MEAN,
              ...
          ),
          ArraySettings(
              output_key="Pos1",
              downsampling_method=DownsamplingMethod.MEAN,
              ...
          )
      ]
  )

[tlambert03]

additional thought: I do like access to the lower-level acquire-zarr API that essentially only writes array nodes, and doesn't worry about ANY ome-metadata. So i'm not arguing here that acquire-zarr should be heavy-handed all the time and only ever creating ome-zarr groups. Mostly, I'm saying that the user should have better control over when an array node is made vs when a multiscales group is made, without auto-selection based on tangentially related parameters.

(in fact, I could even get behind a low-level API that never writes group node metadata at all, leaving that all to the python side, since metadata is fiddly from a code perspective, and not so challenging from a performance perspective)

[aliddell]

I don't like the idea of forcing every image to be an NGFF multiscales group, but you're right that having downsampling_method be the gate for this decision is wrongheaded. In the C API we expose a multiscale flag that can be True when downsampling_method is None. Better might be to add the following to the ArraySettings configuration:

class OMEMultiscaleSettings:
    downsampling_method: Optional[DownsamplingMethod]
    n_levels: Optional[int]

class ArraySettings:
    ...
    multiscale: Optional[OMEMultiscaleSettings]

and an ArraySettings which has a multiscale member with None entries in either downsampling_method or n_levels is simply a single-level multiscales array. What do you think of that?

[tlambert03]

I don't like the idea of forcing every image to be an NGFF multiscales group

absolutely... I think that's a key feature 👍 . in fact, I have recently been using acquire-zarr in such a way as to avoid it writing any OME metadata, handling all that in the python side (this relates to #186)... so I don't want that to go away. (despite my title seeming to promote otherwise 😂 ... i changed the title for clarity)

So, the main the main point here was just that output_key and downsampling shouldn't gate it.

and an ArraySettings which has a multiscale member with None entries in either downsampling_method or n_levels is simply a single-level multiscales array. What do you think of that?

yeah having n_levels is a good step. I think maybe I'd keep it flat and just add downsampling_levels to ArraySettings? (moving downsampling_method from ArraySettings to a deeper nested OMEMultiscaleSettings is perhaps more breaking than needed here?)

But I also think the point I made about output_key is important and needs to be discussed here too. I think the fact that output_key="" gives you ngff metadata at store/zarr.json... while output_key="0" gives you the same directory structure but no ngff metadata at store/zarr.json is confusing:

# ome meta
settings = az.StreamSettings(
    arrays=[az.ArraySettings(output_key="", ...)],
    store_path="store",
)
# no ome meta
settings = az.StreamSettings(
    arrays=[az.ArraySettings(output_key="0", ...)],
    store_path="store",
)

I can imagine the logic that led to that... but I still that using output_key to gate the presence or absence of the group is a bit too magic. I can try to think of a proposed api if you want suggestions...

[aliddell]

But I also think the point I made about output_key is important and needs to be discussed here too. I think the fact that output_key="" gives you ngff metadata at store/zarr.json... while output_key="0" gives you the same directory structure but no ngff metadata at store/zarr.json is confusing:

ome meta

settings = az.StreamSettings(
arrays=[az.ArraySettings(output_key="", ...)],
store_path="store",
)

no ome meta

settings = az.StreamSettings(
arrays=[az.ArraySettings(output_key="0", ...)],
store_path="store",
)
I can imagine the logic that led to that... but I still that using output_key to gate the presence or absence of the group is a bit too magic. I can try to think of a proposed api if you want suggestions...

Yes, that is confusing. I thought that I had disallowed numerical array keys for this reason, though that accidentally may not apply here since "" is special. Something to look into, or on the other hand, we should probably just change the behavior. That will probably break things for some people, and it will break some tests, but it's ultimately a step in the right direction.

This points to being explicit about whether or not every array is an NGFF multiscales group or not, independent of any downsampling. Right now there are several potential options:

• Not an NGFF multiscales group
• An NGFF multiscales group, but without any downsampling
• An NGFF multiscales group, with downsampling until the array size is smaller than the chunk size (current behavior)

What I'm guessing is that you want to replace the 3rd bullet point with something like "An NGFF multiscales group, with downsampling until a user-specified number of levels is hit," though this is going to be a larger piece of work than it seems at first, so let's keep it in the back of our minds but leave it alone for now and consider just the first three.

I'm inclined to add a write_multiscales_metadata: bool field to ArraySettings (defaulting to False). This would match the semantics we already have in the C API with the multiscale flag, where that flag and downsampling_method are independent. We could deprecate using downsampling_method as an implicit gate for multiscales metadata, treating it as shorthand during a transition period. Eventually we could add n_levels as well, if we go that route.

Does that work for you?

[tlambert03]

thanks!

What I'm guessing is that you want to replace the 3rd bullet point with something like ...

I guess I sort of have two thought processes :) In one case, I'm thinking from the perspective of an acquire-zarr supporter/contributor who wants the API to be expressive, flexible, and clear. In the other, I'm thinking about how personally want to use it.

I do think that something like "An NGFF multiscales group, with downsampling until a user-specified number of levels is hit is a good target... but that's also not something I'm in any sort of a hurry for, so putting that on a back-burner is absolutely fine. I think the three key options that a user should be able to pick between are:

1. An NGFF multiscales group, without downsampling
2. An NGFF multiscales group, with downsampling (leaving out n_levels totally) fine
3. Not an NGFF group or even a zarr group at all, just an array node at every ArraySettings.output_key

For what it's worth (and perhaps surprisingly), it's actually that third case that I'm using the most at the moment, and that gets back to #186... (see related #186 (comment)) where I'd like to see acquire-zarr have an option to only produce array nodes.

[aliddell]

3. Not an NGFF group or even a zarr group at all, just an array node at every ArraySettings.output_key

Yeah, I think I get what you mean here. I'll follow up in #186 about that specific use case. For right now, let's focus on (1) and (2).

1. NGFF multiscales group, no downsampling - not currently possible unless chunk size equals array size
2. NGFF multiscales group, with downsampling - currently works via downsampling_method (unless your output key is empty)

We can make two PRs:

• Add write_multiscales_metadata to ArraySettings. This gives explicit control over NGFF multiscales metadata independent of downsampling (addresses case 1, also deprecates the implicit downsampling_method → multiscales behavior).
• Fix the output_key="" confusion so it writes a simple array at the root, not NGFF metadata. This is separate from the multiscales question.

Now that I think about it, I was going to ask you in #186 why you don't just open a separate streams at each of these paths, but as things stand right now, if you do that, you get intermediate metadata that you don't want (because it has to be an NGFF multiscales group right now). If you have this second bullet point, would that be enough to solve your problem, or is there some other reason you want to aggregate three separate arrays into a single base path?

[tlambert03]

direct answers first, ramblings second :)

Both seem great, if I understand:

• write_multiscales_metadata gates whether you get NGFF multiscales group
• standardizing output_key= to always point to an array node relative to StreamSettings.store_path

Just to confirm:

settings = az.StreamSettings(
    arrays=[az.ArraySettings(output_key="", ...)],
    store_path="store",
)
# store is an array node

settings = az.StreamSettings(
    arrays=[az.ArraySettings(output_key="array", ...)],
    store_path="store",
)
# store is a group node (pending #186)
# store/array is an array node

# store is an array node
settings = az.StreamSettings(
    arrays=[az.ArraySettings(
        output_key="array",
        write_multiscales_metadata=True, ...)
    ],
    store_path="store",
)
# store is a group node (pending #186)
# store/array is a group node with multiscales metadata
# store/array/0 is an array node

---

NGFF multiscales group, no downsampling - not currently possible unless chunk size equals array size

wait... isn an NGFF multiscales group with a single level (no downsampling) exactly what you normally get if you 1) Don't pass ArraySettings.output_key and 2) don't pass downsampling_method? even if chunk_size is much less than array size?

I was going to ask you in #186 why you don't just open a separate streams

I also did evaluate this option, and after looking through the codebase, I think I came to the conclusion that there is no global threadpool, and that could make for extremely high resource usage if you weren't careful? is that correct? And also yes... if two streams had shared intermediate paths there would be conflicting file access between streams.

is there some other reason you want to aggregate three separate arrays into a single base path?

It's not so much about "a single base path"... it's more just that I want an option for acquire-zarr to write just write array nodes (creating only bare intermediate directories as needed to satisfy ArraySettings.output_key)

The ultimate reason is that I want to allow acquire-zarr to be a backend for yaozarrs.write (http://tlambert03.github.io/yaozarrs/API_Reference/yaozarrs.write.v05/) which is what ome-writers now uses. It's proven to be a very nice pattern: python worries about all the finicky metadata stuff. You get all the lovely parsing/validation/casting of pydantic models, and you have simple API required of the array-writing functionality (the only part that really needs to be high performance). It works great for zarr-python, zarrs-python, and tensorstore. I'd like to use it with acquire-zarr (primarily because of the built-in awareness of higher-dimensional chunking and excellent sharding performance), but the main blocker is getting acquire-zarr to "do less" in that situation: i.e. have an array-only API like the others that leaves my groups alone.

I don't expect you to want to look through a lot of code... but if you were curious, the key files that show how I'm using acquire zarr are:

• https://github.com/pymmcore-plus/ome-writers/blob/main/src/ome_writers/_backends/_acquire_zarr.py
  (with _post_prepare, write, and finalize as the key methods)
• https://github.com/pymmcore-plus/ome-writers/blob/main/src/ome_writers/_backends/_yaozarrs.py
  (where YaozarrsBackend.prepare has all the hierarchy setup logic)
• https://github.com/tlambert03/yaozarrs/blob/main/src/yaozarrs/write/v05/_write.py
  (which actually implements the complex ngff validation and tree setup, and an "array writer" is required to do just the array writing)

if you were so inclined, I suspect an LLM could answer most questions with those three files.

---

sorry to go so deep on all of this! thanks for your time

[aliddell]

wait... isn an NGFF multiscales group with a single level (no downsampling) exactly what you normally get if you 1) Don't pass ArraySettings.output_key and 2) don't pass downsampling_method? even if chunk_size is much less than array size?

Yeah, that and HCS images are like the only exceptions haha.

I also did evaluate this option, and after looking through the codebase, I think I came to the conclusion that there is no global threadpool, and that could make for extremely high resource usage if you weren't careful? is that correct? And also yes... if two streams had shared intermediate paths there would be conflicting file access between streams.

Yeah, that's correct. The goal of opening a stream in acquire-zarr is a coherent dataset with proper hierarchy, not arbitrary array batching. But I can see a way to share resource pools across multiple streams relatively easily. Would an API call to spawn multiple streams with shared resource pools be interesting?

[tlambert03]

Would an API call to spawn multiple streams with shared resource pools be interesting?

I guess I'd leave that up to you. I'm perfectly happy with the single stream managing all the arrays, it literally just comes down to that single line change on streaming/zarr.stream.cpp#L1473-L1475 for writing empty group json being the only problem... so that seemed like the easiest change.

fwiw, my use case isn't even arbitrary array batching. It's multiple positions in a dataset, honestly, even plate acquisition, etc... (just letting yaozarrs do the hierarchy creation). They are a coherent dataset in a standard hierarchy... I just happen to want to create that hierarchy in this case, and that one line change lets me do it without acquire-zarr writing empty things (but... we can continue that particular discussion over in #186)

[aliddell]

@tlambert03 does #194 close this one as well?

[tlambert03]

sort of ?

Isn't downsampling_method still gating the presence/absence of a multiscales group? As I return to the code now, looking for clarity on "when do I get a zarr group with multiscales metadata" it still seems to be based on downsampling, though the C API has ZarrArraySettings.multiscale that would appear to gate it? I'm honestly not sure ... still a little confused.

Is there any way to create an OME-NGFF gropu without also opting into the full pyramidal calculation and downsampling? Or, if someone wants the metadata, are they forced to also accept all the pyramid levels?

[tlambert03]

I was just working on this a bit... but I don't really know how best to reconcile this. Here's a branch that i feel is not confusing (it passes through that bool multiscale from the C API) but it's a breaking change at the moment, since it requires that you use write_multiscales_metadata=True even if downsampling_method is also set.

https://github.com/acquire-project/acquire-zarr/compare/main..tlambert03:expose-multiscale-bool

That's what this issue was about. It's not particularly pressing for me. so if you'd rather just close this and not worry about it, that's fine with me. but i do think there original point here hasn't exactly been solved