Repository navigation
bug with setitem with oindex and sharding #2834
Description
Zarr version
v3.0.2
Numcodecs version
?
Python Version
3.12
Operating System
any
Installation
any
Description
Discovered in #2825
Steps to reproduce
import zarr
import numpy as np
store = zarr.storage.MemoryStore()
group = zarr.group(store)
array = group.create_array(
name="zoo",
shape=(1,2,1),
chunks=(1,2,1),
shards=(1,2,1),
dtype=np.int32,
)
zindexer = (np.array([0]), np.array([0, 0]), np.array([0]))
new_data = np.full(array.oindex[zindexer].shape, fill_value=1)
array.oindex[zindexer] = new_dataraises the error:
ValueError: shape mismatch: value array of shape (1,2,1) could not be broadcast to indexing result of shape (2,)
It succeeds with shards=None.
Additional output
No response
Activity
brokkoli71 commentedon Feb 18, 2025
I think the problem is that the information that the selection is an
OrthogonalSelectiongets lost on the way.In
ShardingCodec._encode_partial_singlea new indexer is created to index the inner chunks:There,
get_indexerguesses what indexing should be done based on the selection tuple, and in this case guessesCoordinateIndexerinstead ofOrthogonalIndexermoradology commentedon Mar 4, 2025
Looks like there are some problematic ambiguities that will be a plague
get_indexerwithout a bit more information - don't think there's enough information, in principle, to decide what to do here.Pure Fancy Indexing:Selection using only scalars and integer/boolean array–likes (no slices).
Tests in
is_pure_fancy_indexing:Orthogonal Indexing:A subset of pure fancy indexing where each axis is indexed independently (Cartesian product), typically expecting 1D arrays.
Tests in
is_pure_orthogonal_indexing:Example 1:
Selection: (np.array([0, 1]), np.array([2, 3])) for a 2D array.
Ambiguity:
Coordinate interpretation: Zips the arrays to select coordinates (0,2) and (1,3).
Orthogonal interpretation: Forms a Cartesian product yielding (0,2), (0,3), (1,2), (1,3).
Example 2:
Selection: (np.array([0]), 1) for a 2D array.
Ambiguity:
One axis uses an iterable while the other is a scalar.
Could be interpreted as a coordinate selection by “zipping” (i.e., pairing the single value with 1) or forced into orthogonal mode, treating the iterable independently.
Example 3:
Selection: (np.array([0]), np.array([[1]])) where the second index isn’t 1D.
Ambiguity:
The extra dimension prevents clear classification.
Once flattened (i.e., via .ravel()), it becomes a 1D array and may be treated as orthogonal. Without normalization, it’s unclear whether coordinate or orthogonal behavior is intended.
brokkoli71 commentedon Apr 15, 2025
One thing I would suggest as a consequence is to rename
get_indexerto something likeguess_indexer.Furthermore, we would need to pass the indexer or at least information about the type of indexer to the codecs and thus to the sharding codec so it would not need to guess the indexer.
moradology commentedon Apr 15, 2025
I've got a WIP that attempts this; will put it up as a draft today or tomorrow
brokkoli71 commentedon Apr 29, 2025
Hey @moradology,
I just stumbled across this again — did you have any luck making progress on it? Happy to help if there’s anything I can do to move it forward.
moradology commentedon Apr 30, 2025
Hey @brokkoli71 - Thank you. I made some progress but got pulled off to attend to more pressing problems. Locally, things aren't in a good place to push up but this is on the top of my list as soon as I secure some more time to work on OSS.
wilson0028 commentedon Jul 22, 2025
We are running into this same issue with sharding. Are there any updates with this bug? Thank you in advance!
tighten hypothesis index strategies to sorted, unique lists
Implement version-specific default chunk and shard sizes in iohub uti…
d-v-b commentedon Sep 24, 2026
this was closed via #4316