Merge branch 'main' into fix/codec-chain-validation

Author: d-v-b

changes/4351.bugfix.md

@@ -0,0 +1 @@
+Missing Zarr format 3 imagecodecs now name `imagecodecs-zarr` as a known provider, alongside `virtual-tiff` where both packages register the codec.

packages/zarr-indexing/CONTRIBUTING.md

@@ -8,16 +8,16 @@ just test        # run the test suite (extra args go to pytest)
 just lint        # ruff, same invocation as CI
 just typecheck   # pyright, same invocation as CI
 just docs-check  # strict build of the docs site
-just check       # all of the above
+just check       # the checks above plus TensorStore parity
 just docs-serve  # serve the docs site locally
 ```
 
-Run them from this directory, or from anywhere in the repository as
+Run them from this directory, or from the repository root as
 `just packages/zarr-indexing/<recipe>`.
 
-The test recipe runs against the workspace-root environment, because the
-chunk-resolution tests exercise this package against `zarr`'s chunk grids and
-`zarr` is deliberately not a dependency of this package.
+The test recipe layers this package into the repository-root environment.
+Chunk-resolution tests use this package’s own grids; the Dask example also
+uses `zarr`, which is not a dependency of the base indexing package.
 
 ## License
 

packages/zarr-indexing/README.md

@@ -5,19 +5,19 @@ Composable, lazy coordinate transforms for Zarr array indexing.
 Documentation: <https://zarr-indexing.readthedocs.io/>
 
 This package implements TensorStore-inspired index transforms. The core idea:
-every indexing operation (slicing, fancy indexing, etc.) produces a coordinate
-mapping from user space to storage space. These mappings compose lazily — no
-I/O until you explicitly read or write.
+each supported indexing operation (slicing, fancy indexing, etc.) produces a coordinate
+mapping from user space to storage space. These mappings compose without reading selected source values. `LazyArray`
+materializes them on request; the transform algebra itself performs no source I/O.
 
 Key types:
 
 - `LazyArray` — wraps a system-memory/basic-indexing source and adds a `.lazy`
   accessor: `LazyArray.from_numpy(numpy_array).lazy[10:50, ::2].lazy.oindex[[3, 1, 1], :]`
   composes a transform and returns a new view without reading data, and
   `result()` materializes it into owned system memory. `LazyArray(source)` uses
-  the conservative basic reader; `from_numpy` explicitly selects NumPy's
-  optimized reader. Device arrays require an explicit custom reader responsible
-  for transferring values into the supplied system-memory output buffer.
+  the basic reader; `from_numpy` selects `numpy_reader`, which currently uses
+  the same slab-and-gather implementation. Device sources that refuse NumPy
+  conversion need a custom reader to transfer values into the output buffer.
 - `Reader` — the explicit backend execution boundary: transforms say which
   values belong in the result, while readers say how a backend obtains them
 - `IndexDomain` — a rectangular region of integer coordinates
@@ -32,9 +32,11 @@ Key types:
   dimension can depend on the input
 - `compose` — chain two transforms into one
 
-The package depends only on NumPy and the standard library; it does not import
-`zarr`. It is developed in the [zarr-python](https://github.com/zarr-developers/zarr-python)
-repository and consumed by `zarr` to resolve array indexing operations.
+The base package depends on NumPy and the standard library; its optional testing
+module also requires Hypothesis. The package does not import `zarr`. It is developed
+in the [zarr-python](https://github.com/zarr-developers/zarr-python) repository,
+and its examples include reading Zarr arrays through Dask. Installing it does not
+replace Zarr's indexing implementation.
 
 ## Installation
 

packages/zarr-indexing/benchmarks/README.md

@@ -22,6 +22,19 @@ whether input arrays and transform construction are included. Repeated local
 column access measures cache reuse, which trades allocation against retained
 memory. Bounded coordinate batches avoid constructing the full coordinate array.
 
+The current script reports `peak_mib` as the incremental peak tracked by
+`tracemalloc` during a separate invocation after the timed calls. It is not
+process RSS or total retained memory. Inputs and most transforms are constructed
+before measurement. The case walks construct plans and consume projections
+without retaining them; `all_coordinates` materializes the complete coordinate
+array. There is no bounded-batch coordinate workload in this script.
+
+`local_rows` reuses one prepared table: its first timed invocation populates the
+local-coordinate cache, while later timed invocations and the allocation probe
+reuse that cache. Its median mixes a cold first call with warm calls, and its
+reported peak excludes the already-retained cache. A separate fresh-table
+measurement would be needed to quantify cold cache construction.
+
 These scripts measure planning rather than codec or storage throughput. Repeat
 measurements with alternating operation order before interpreting small timing
 differences. Preserve raw benchmark output as an experiment artifact rather than

packages/zarr-indexing/benchmarks/chunk_planning.py

@@ -25,6 +25,11 @@
 
 
 def measure(operation: Callable[[], Any], repeats: int) -> dict[str, float]:
+    """Time repeated calls, then trace one additional call's incremental peak.
+
+    State retained by earlier calls can affect the extra call; this is not a
+    measurement of process RSS or all memory owned by the operation's inputs.
+    """
     samples = []
     for _ in range(repeats):
         start = time.perf_counter()
@@ -84,6 +89,8 @@ def read_local_rows() -> None:
         for row in range(len(table)):
             table.local[table.run(row)]
 
+    # The first timed call fills table.local; later calls, including the traced
+    # allocation probe, reuse it. The table and retained cache are not re-created.
     results["local_rows"] = measure(read_local_rows, args.repeats)
     part = plan_chunks(
         IndexTransform.from_shape((100, 100, 100)),

packages/zarr-indexing/changes/4345.bugfix.md

@@ -0,0 +1 @@
+Reject invalid wire index-array values and unrepresentable normalized bounds/ranks, and raise an explicit error for unsupported intersections sharing an affine and lookup input axis instead of returning incorrect coordinates. Group negative chunk-coordinate tuples without merging distinct chunks.

packages/zarr-indexing/changes/4345.doc.md

@@ -0,0 +1,3 @@
+Correct indexing, reader, serialization, cache, and integration descriptions to match supported behavior; qualify NumPy/TensorStore compatibility and performance claims.
+
+Describe current contracts in source and test docstrings instead of narrating prior implementations. Clarify that immutable index coordinates do not snapshot source values.

packages/zarr-indexing/changes/4346.misc.md

@@ -0,0 +1 @@
+Expand planner property tests across signed origins and chunk IDs, custom grids, mixed affine and lookup dependencies, duplicate coordinates, and empty domains. Verify exact request coverage and storage mapping with an independent pointwise oracle.

packages/zarr-indexing/docs/api/grid.md

@@ -11,8 +11,9 @@ objects whose `shape` is the valid data size and whose `codec_shape` preserves
 the full codec-buffer size at a regular-grid boundary.
 
 `dimension_grids_from_chunks` returns these compact dimensions: integer chunk
-shapes become `FixedDimension` instances and explicit per-axis edge sequences
-become `VaryingDimension` instances. `DimensionGridLike` remains the narrow
+shapes become `FixedDimension` instances and positive per-axis edge sequences
+become `VaryingDimension` instances. An empty-axis sequence of zeros (including
+an empty sequence) becomes `FixedDimension(size=0, extent=0)`. `DimensionGridLike` remains the narrow
 protocol used by the chunk planner, while `EdgeDimensionGrid` is kept for
 explicit edge-based and coordinate-origin examples.
 

packages/zarr-indexing/docs/api/index.md

@@ -25,12 +25,11 @@ and the wire format built on top of it.
 - [`zarr_indexing.domain`](domain.md) — `IndexDomain`, a rectangular region of
   integer coordinates with an explicit (possibly non-zero) origin
 - [`zarr_indexing.output_map`](output_map.md) — `ConstantMap`, `DimensionMap`,
-  and `ArrayMap`: three representations of a set of integer coordinates, one
-  per storage dimension
+  and `ArrayMap`: three coordinate mappings that preserve order and duplicates,
+  one per storage dimension
 - [`zarr_indexing.transform`](transform.md) — `IndexTransform`, which pairs a
   domain with output maps, plus the indexing (`[...]`, `.oindex`, `.vindex`),
-  `intersect`, and `translate` operations, and `selection_to_transform`
-  transforms into one
+  `intersect`, `translate`, and `compose` operations, and `selection_to_transform`
 
 **Chunk resolution**