Read remote Zarr v2 stores via .zmetadata (0.1.1)#29
Merged
Conversation
examples/demo.zarr is a small (time,lat,lon) temperature grid written with consolidated metadata so it can be read remotely via read_zarr() over HTTPS (e.g. raw.githubusercontent.com).
Remote (HTTP/object) stores can't list directories, so read_zarr enumerates
arrays from consolidated metadata. list_array_names_remote only understood the
Zarr v3 consolidated_metadata block in zarr.json, so every remote v2 store
errored ("no consolidated metadata in zarr.json") — including public stores
like Pangeo GPCP and ARCO-ERA5. Fall back to the v2 .zmetadata object when the
v3 block is absent.
- meta.rs: parse .zmetadata (keys ending in /.zarray) to enumerate v2 arrays
- tests: test/test_http_integration.py covers both v2 .zmetadata and v3
consolidated_metadata over a loopback HTTP server (make test_http_debug);
adds consolidated_{v2,v3}_http fixtures
- docs: dims= is a string ('time,lat,lon'), not a SQL list — fix README and
description.yml; document the v2/v3 consolidated requirement for remote stores
- Cargo.toml, pyproject.toml, description.yml: 0.1.0 -> 0.1.1 - Cargo.lock, uv.lock: synced to the new version - CHANGELOG: split the accumulated [Unreleased] history into [0.1.0] (what shipped in the v0.1.0 tag) and a new [0.1.1] for the remote v2 .zmetadata fix and the dims= docs correction
- test/test_http_integration_real.py: read the live public Pangeo GPCP v2 store end-to-end (array enumeration from .zmetadata, coordinate pivot, precip decode). Network-marked; skipped when unreachable. Runs as part of the HTTP suite: make test_http / test_http_debug run both the loopback and real-data files (17 tests); make test_http_real runs only the real-data file. - docs: README, docs/README.md, docs/ome-zarr.md now use real, runnable examples (the public GPCP store and this repo's fixtures) instead of placeholder paths; each verified against the build. - pyproject: register the network pytest marker. - CHANGELOG: note the real-data coverage and docs refresh under 0.1.1.
Real public OME-Zarr stores can't be read remotely (no consolidated metadata, so object stores can't be listed, plus no dimension metadata), so the repo's synthetic OME-NGFF fixture is now written with consolidated metadata and read over the loopback HTTP server. - generate_fixtures.py: consolidate synthetic_multichannel.ome.zarr (idempotent; local reads scan directories and ignore it, so the SQLLogicTest is unaffected) - test/test_http_integration.py: OME-Zarr HTTP tests — metadata enumeration, array_path='0' per-channel means, and the nested label image — run with the v2/v3 tests via make test_http - CHANGELOG: note the OME-Zarr HTTP coverage
Real public OME-Zarr images (e.g. the IDR) carry no consolidated metadata and no
per-array dimension metadata, so they could not be read remotely. Two changes let
them read by array_path:
- read_zarr.rs: array_path/array opens the requested array directly; store listing
(which needs consolidated metadata on remote stores) is now best-effort and used
only to resolve coordinate arrays when available.
- meta.rs: dimension_names falls back to the parent group's OME multiscales.axes
(matched by dataset path and rank) when the array has neither dimension_names nor
_ARRAY_DIMENSIONS.
read_zarr('https://.../6001240.zarr', array_path='0') now returns c/z/y/x columns
and uint16 intensities. Covered by test_idr_ome_zarr_array_path (network-gated);
docs/ome-zarr.md and CHANGELOG updated.
d33bs
marked this pull request as ready for review
July 11, 2026 21:53
alxmrs
reviewed
Jul 12, 2026
Per review feedback: a list is the idiomatic SQL form, and duckdb-rs supports LIST-typed named parameters, so there was no reason for the VARCHAR string. read_zarr(store, dims=['time','lat','lon']) -- was dims='time,lat,lon' - read_zarr.rs: register dims as LIST(VARCHAR); read it via Value::to_list() - tests + docs: update all dims= usages to the list form
alxmrs
approved these changes
Jul 12, 2026
alxmrs
left a comment
alxmrs
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, thanks for the fix.
This was referenced Jul 13, 2026
Selecting a single array with array_path= named the value column after the array's store-relative path, so numeric levels (0) and nested label paths (labels/nuclei/0) had to be double-quoted. Register that column as `value` instead; multi-variable groups keep their real names. Decoding is unchanged (it keys off the array path, not the display name). - read_zarr.rs: value alias for a single numeric/nested array_path column - tests + docs updated (raised by alxmrs in review of xqlsystems#29)
Member
Author
|
Thanks @alxmrs - merging this in after making some changes and creating issues to track future progress! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Public v2 Zarr stores (Pangeo GPCP, ARCO-ERA5, most CMIP6) couldn't be read
over HTTP — the remote reader only understood the Zarr v3
consolidated_metadatablock. Object stores can't list directories, so this adds a fallback to the v2
.zmetadataobject, and those stores now read straight from a URL.list_array_names_remotereads v2.zmetadataarray_path=opens a single array without listing the whole store,and dimension names fall back to OME
multiscales.axes— so real public bioimages(e.g. the IDR) read via
read_zarr(url, array_path='0')even without consolidated metadatalive public stores (GPCP + IDR), run together via
make test_http