Read remote Zarr v2 stores via .zmetadata (0.1.1)

CHANGELOG.md

@@ -6,6 +6,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [Unreleased]
 
+## [0.1.1] - 2026-07-11
+
+### Fixed
+- Read Zarr **v2** stores over HTTP/S3/GCS/Azure by consuming their consolidated `.zmetadata`. `read_zarr`, `read_zarr_metadata`, and `read_zarr_groups` previously errored on *every* remote v2 store (`remote Zarr store has no consolidated metadata in zarr.json`) because `list_array_names_remote` only understood the Zarr v3 `consolidated_metadata` block. Object stores can't list directories, so v2's separate `.zmetadata` object is the only way to enumerate arrays remotely; the reader now falls back to it. Public v2 stores such as Pangeo GPCP and ARCO-ERA5 read directly from their URL. Covered by `test/test_http_integration.py` (v2 `.zmetadata`, v3 `consolidated_metadata`, and an OME-Zarr bioimage — `array_path` selection plus a nested label image — over a loopback HTTP server; `make test_http_debug`) and by `test/test_http_integration_real.py`, which reads the live public Pangeo GPCP v2 store end-to-end (`make test_http_real`, network-gated). The OME-Zarr fixture is now written with consolidated metadata so it is readable over HTTP as well as locally.
+- Read OME-Zarr images by `array_path` even when the store has no consolidated metadata. `array_path=` now opens the requested array directly instead of first listing the whole store, and dimension names fall back to the OME `multiscales.axes` when the array carries neither `dimension_names` nor `_ARRAY_DIMENSIONS`. Real public bioimage stores (e.g. the IDR) now read via `read_zarr(url, array_path='0')` — covered by a network-gated test in `test/test_http_integration_real.py`.
+- An array selected by `array_path` exposes its data as a `value` column, so numeric levels (`0`) and nested paths (`labels/nuclei/0`) no longer need to be double-quoted as SQL identifiers.
+- `dims=` is now a `LIST(VARCHAR)` — `read_zarr(store, dims=['time','lat','lon'])`, the idiomatic SQL form. It was previously a `VARCHAR` that accepted only a comma-separated or JSON-array *string* (and the SQL-list form errored at bind with `expected ident`).
+- Docs now use real, runnable examples verified against the extension — the public GPCP store in `README.md`, plus this repo's fixtures in `docs/README.md` and `docs/ome-zarr.md` — replacing the `path/to/...` and `image.ome.zarr` placeholders.
+
+## [0.1.0] - 2026-07-07
+
 ### Fixed
 - Rename the crate/extension from `duckdb_zarr` to `zarr` throughout (Cargo package name, `Makefile` `EXTENSION_NAME`, `MainDistributionPipeline.yml`, tests, docs). The prior partial rename only updated `description.yml` and added a `zarr_init_c_api` alias; native community builds still produced and looked for `duckdb_zarr` artifacts because the Rust crate name (and thus the compiled library filename) didn't match, and the `Makefile`'s plain `EXTENSION_NAME=duckdb_zarr` assignment overrode the `EXTENSION_NAME` env var the CI distribution workflow passes in.
 - CI: align DuckDB to v1.5.4 across all three version sites — crate `=1.10504.0`, workflow `duckdb_version`, and `Makefile` `TARGET_DUCKDB_VERSION` (the last stamps the extension metadata `duckdb_version`; with `USE_UNSTABLE_C_API=1` the loader requires an *exact* match) — so the built extension loads in the `duckdb_sqllogictest` test runner, which had moved to 1.5.4. Fixes macOS-arm64/Windows test-load version-mismatch failures. Wrapped now-`unsafe` `FlatVector::as_mut_ptr` calls per the 1.10504.0 API.

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "zarr"
-version = "0.1.0"
+version = "0.1.1"
 edition = "2021"
 
 [lib]

Makefile

@@ -1,4 +1,4 @@
-.PHONY: clean clean_all clippy fmt fmt-check generate_fixtures lint release-check render-community-descriptor
+.PHONY: clean clean_all clippy fmt fmt-check generate_fixtures lint release-check render-community-descriptor test_http test_http_debug test_http_release test_http_real
 
 PROJ_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
 
@@ -39,6 +39,28 @@ test: test_debug
 test_debug: generate_fixtures test_extension_debug
 test_release: generate_fixtures test_extension_release
 
+# HTTP integration tests. Build the extension first (make debug / make release).
+# pytest runs via uv, so no venv setup is needed; duckdb is pinned to the target
+# DuckDB version. Both files run together as one suite:
+#   test_http_integration.py      — loopback http.server + synthetic fixtures
+#                                   (deterministic; v2 .zmetadata + v3 consolidated_metadata)
+#   test_http_integration_real.py — reads a real public v2 store over the internet
+#                                   (network-marked; skips when the store is unreachable)
+# test_http_real runs only the real-data file.
+test_http: test_http_debug
+test_http_debug: generate_fixtures
+	uv run --with pytest --with 'duckdb==$(TARGET_DUCKDB_VERSION:v%=%)' \
+		pytest test/test_http_integration.py test/test_http_integration_real.py \
+		--extension build/debug/$(EXTENSION_NAME).duckdb_extension -v
+test_http_release: generate_fixtures
+	uv run --with pytest --with 'duckdb==$(TARGET_DUCKDB_VERSION:v%=%)' \
+		pytest test/test_http_integration.py test/test_http_integration_real.py \
+		--extension build/release/$(EXTENSION_NAME).duckdb_extension -v
+test_http_real:
+	uv run --with pytest --with 'duckdb==$(TARGET_DUCKDB_VERSION:v%=%)' \
+		pytest test/test_http_integration_real.py \
+		--extension build/debug/$(EXTENSION_NAME).duckdb_extension -v
+
 fmt:
 	cargo fmt --all
 

README.md

@@ -5,30 +5,45 @@
 A Rust DuckDB extension that lets you query [Zarr](https://zarr.dev/) stores with SQL — in the same spirit as [xarray-sql](https://github.com/alxmrs/xarray-sql) and [zarr-datafusion](https://lib.rs/crates/zarr-datafusion), but as a first-class DuckDB extension with no external query engine.
 
 ```sql
--- Local store: path ending in .zarr is intercepted automatically
+-- Query a public Zarr store straight from its URL. GPCP is a multi-group store
+-- (a precip variable plus CF bounds), so pick the group with dims= — a list of
+-- dimension names.
+SELECT time, latitude, longitude, precip
+FROM read_zarr(
+  'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr',
+  dims=['time','latitude','longitude']
+)
+LIMIT 10;
+
+-- Inspect a store's arrays — a local path or a URL
+SELECT name, role, dtype, shape
+FROM read_zarr_metadata('https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr');
+
+-- Single-group stores need no dims=; a path/URL ending in .zarr is intercepted
+-- automatically. (These use this repo's fixtures — run `make generate_fixtures` first.)
 SELECT lat, lon, AVG(temperature)
-FROM 'era5.zarr'
+FROM 'test/fixtures/xarray_tutorial/float_baseline.zarr'
 GROUP BY lat, lon;
 
--- HTTP/HTTPS stores also work via replacement scan
-SELECT * FROM 'https://example.com/data.zarr';
+-- Select one array by its store-relative path (e.g. an OME-Zarr resolution level or nested label)
+SELECT * FROM read_zarr('test/fixtures/bioimage/ome_zarr/synthetic_multichannel.ome.zarr', array_path='0');
 
--- Explicit table function with optional dims= for multi-group stores
-SELECT * FROM read_zarr('path/to/store.zarr', dims=['time', 'lat', 'lon']);
-
--- Select one array by its store-relative path (including nested arrays)
-SELECT * FROM read_zarr('image.ome.zarr', array_path='labels/nuclei/0');
-
--- Inspect arrays in a store
-SELECT name, role, dtype, shape FROM read_zarr_metadata('path/to/store.zarr');
-
--- List dimension groups
-SELECT * FROM read_zarr_groups('path/to/store.zarr');
+-- List a store's dimension groups
+SELECT dims, shape, data_vars FROM read_zarr_groups('test/fixtures/xarray_tutorial/multi_dim_group.zarr');
 ```
 
 See [docs/design.md](docs/design.md) for the full design and
 [docs/ome-zarr.md](docs/ome-zarr.md) for a small bioimage example.
 
+## Remote stores
+
+Read directly from a URL — HTTP/HTTPS, S3, GCS, or Azure. Object stores can't list
+directories, so a remote store must carry **consolidated metadata**: a Zarr v2
+`.zmetadata` object or a Zarr v3 `consolidated_metadata` block. Most published
+datasets already do (e.g. anything written by xarray with `consolidated=True`), so
+they read straight from their URL as shown above. S3/GCS/Azure credentials come from
+DuckDB's secrets manager (`CREATE SECRET ... TYPE S3`).
+
 ## Status
 
 Active development. Phases 1–3 are implemented:
@@ -52,10 +67,14 @@ Requires: Rust toolchain, Python 3.11+, make, git.
 ## Testing
 
 ```shell
-make test_debug   # or make test_release
+make test_debug   # SQLLogicTest suite over local stores (or make test_release)
+make test_http    # HTTP integration tests: synthetic loopback + a real public store
 ```
 
-Tests are in `test/sql/` (SQLLogicTest format). Fixtures are generated automatically by `make test_*`. To regenerate manually:
+SQLLogicTest cases live in `test/sql/`. The HTTP suite (`test/test_http_integration*.py`)
+reads stores over HTTP: synthetic v2 `.zmetadata` and v3 `consolidated_metadata` fixtures
+via a loopback server, plus the live public GPCP store (network-gated — skipped when
+unreachable). Fixtures are generated automatically by `make test_*`. To regenerate manually:
 
 ```shell
 make generate_fixtures

description.yml

@@ -1,7 +1,7 @@
 extension:
   name: zarr
   description: Query Zarr array stores (v2 and v3) with SQL — read xarray-convention datasets as tables
-  version: 0.1.0
+  version: 0.1.1
   language: Rust
   build: cargo
   requires_toolchains: "rust;python3"
@@ -29,7 +29,8 @@ docs:
     **Functions**
 
     - `read_zarr(path)` — scan all rows from a single-group Zarr store.
-    - `read_zarr(path, dims=['time','lat','lon'])` — select one dim group from a multi-group store.
+    - `read_zarr(path, dims=['time','lat','lon'])` — select one dim group from a multi-group store
+      (`dims` is a list of dimension names).
     - `read_zarr_metadata(path)` — inspect array names, dtypes, shapes, and roles.
     - `read_zarr_groups(path)` — enumerate distinct dimension groups in a store.
 
@@ -47,6 +48,7 @@ docs:
     - Zarr v2 and v3
     - Codecs: gzip, zstd, blosc, crc32c, sharding, transpose, bytes (big/little endian)
     - CF conventions: `scale_factor`/`add_offset` packed integers, `_FillValue`/`missing_value` null masking
-    - Local filesystem, HTTP/HTTPS, S3, GCS, and Azure Blob Storage
-      (consolidated metadata required for remote array listing; S3/GCS/Azure credentials
-      are read from DuckDB's secrets manager — `CREATE SECRET ... TYPE S3` etc.)
+    - Local filesystem, HTTP/HTTPS, S3, GCS, and Azure Blob Storage. Remote stores must carry
+      consolidated metadata — a Zarr v2 `.zmetadata` object or a Zarr v3 `consolidated_metadata`
+      block — because object stores cannot list directories. S3/GCS/Azure credentials are read
+      from DuckDB's secrets manager — `CREATE SECRET ... TYPE S3` etc.

docs/README.md

@@ -41,15 +41,16 @@ This project is related to [xarray-sql](https://github.com/alxmrs/xarray-sql) an
 -- Load the extension
 LOAD 'zarr';
 
--- Read metadata from a Zarr array
-SELECT * FROM read_zarr_metadata('/path/to/my/array.zarr');
+-- Read metadata from a Zarr store (a local path or a URL)
+SELECT * FROM read_zarr_metadata('test/fixtures/xarray_tutorial/float_baseline.zarr');
 
--- Read array data as a table
-SELECT * FROM read_zarr('/path/to/my/array.zarr');
+-- Read a store as a table
+SELECT * FROM read_zarr('test/fixtures/xarray_tutorial/float_baseline.zarr');
 
--- Query specific dimensions or filter data
-SELECT * FROM read_zarr('/path/to/my/array.zarr') 
-WHERE dimension_0 > 100 AND dimension_1 < 50;
+-- Filter with plain SQL on the coordinate columns
+SELECT time, lat, lon, temperature
+FROM read_zarr('test/fixtures/xarray_tutorial/float_baseline.zarr')
+WHERE lat > 0 AND lon < 180;
 ```
 
 For a small bioimage walkthrough, see [Querying OME-Zarr](ome-zarr.md).
@@ -63,19 +64,19 @@ Use metadata discovery first to see the available store-relative paths:
 
 ```sql
 SELECT name, dims, shape, dtype
-FROM read_zarr_metadata('/path/to/image.ome.zarr');
+FROM read_zarr_metadata('test/fixtures/bioimage/ome_zarr/synthetic_multichannel.ome.zarr');
 ```
 
 Then pass `array_path=` when you want a specific image or label array:
 
 ```sql
-SELECT c, AVG("0") AS mean_intensity
-FROM read_zarr('/path/to/image.ome.zarr', array_path='0')
+SELECT c, AVG(value) AS mean_intensity
+FROM read_zarr('test/fixtures/bioimage/ome_zarr/synthetic_multichannel.ome.zarr', array_path='0')
 GROUP BY c;
 
-SELECT "labels/nuclei/0" AS label, COUNT(*) AS pixels
-FROM read_zarr('/path/to/image.ome.zarr', array_path='labels/nuclei/0')
-WHERE "labels/nuclei/0" > 0
+SELECT value AS label, COUNT(*) AS pixels
+FROM read_zarr('test/fixtures/bioimage/ome_zarr/synthetic_multichannel.ome.zarr', array_path='labels/nuclei/0')
+WHERE value > 0
 GROUP BY label;
 ```
 

docs/ome-zarr.md

@@ -1,36 +1,58 @@
 # Querying OME-Zarr
 
 OME-Zarr images commonly contain several resolution levels and nested label
-arrays. Start by listing the available arrays:
+arrays. The examples below run against this repo's small synthetic fixture
+(`make generate_fixtures` first). Start by listing the available arrays:
 
 ```sql
 SELECT name, dims, shape, dtype
-FROM read_zarr_metadata('image.ome.zarr');
+FROM read_zarr_metadata('test/fixtures/bioimage/ome_zarr/synthetic_multichannel.ome.zarr');
 ```
 
 The `array_path` argument is optional. Use it when a store has multiple
 resolution levels, nested labels, or otherwise ambiguous array groups. The path
 is the same store-relative path reported by `read_zarr_metadata`.
 
-Then select a resolution level and aggregate a region by channel:
+Then select a resolution level and aggregate by channel:
 
 ```sql
-SELECT c, AVG("0") AS mean_intensity
-FROM read_zarr('image.ome.zarr', array_path='0')
-WHERE y BETWEEN 100 AND 199
-  AND x BETWEEN 200 AND 299
+SELECT c, AVG(value) AS mean_intensity
+FROM read_zarr('test/fixtures/bioimage/ome_zarr/synthetic_multichannel.ome.zarr', array_path='0')
 GROUP BY c;
 ```
 
-Here, `0` is the conventional path for the highest-resolution level. Xarray
-dimension names such as `c`, `y`, and `x` become SQL columns. Numeric and nested
-array names must be double-quoted when referenced as columns.
+Here, `0` is the conventional path for the highest-resolution level. When you
+select a single array with `array_path`, its data is exposed as a `value` column
+and its xarray dimension names (`c`, `y`, `x`) become the other columns — add a
+`WHERE y BETWEEN ... AND x BETWEEN ...` clause to aggregate a sub-region.
 
 Nested label arrays use the same selector:
 
 ```sql
-SELECT "labels/nuclei/0" AS label, COUNT(*) AS pixels
-FROM read_zarr('image.ome.zarr', array_path='labels/nuclei/0')
-WHERE "labels/nuclei/0" > 0
+SELECT value AS label, COUNT(*) AS pixels
+FROM read_zarr('test/fixtures/bioimage/ome_zarr/synthetic_multichannel.ome.zarr', array_path='labels/nuclei/0')
+WHERE value > 0
 GROUP BY label;
 ```
+
+## Remote OME-Zarr
+
+Public OME-Zarr images can be read straight from a URL. Bioimage stores usually
+lack consolidated metadata, so whole-store operations (`read_zarr_metadata`, or
+`read_zarr` without `array_path`) aren't available remotely — but a specific
+resolution level reads by `array_path`, with `c`/`z`/`y`/`x` recovered from the
+OME `multiscales.axes`:
+
+```sql
+-- A public image from the Image Data Resource (IDR)
+SELECT c, z, y, x, value AS intensity
+FROM read_zarr(
+  'https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr',
+  array_path='0'
+)
+LIMIT 10;
+```
+
+Filters and aggregates over a full pyramid level scan every chunk (there is no
+sub-chunk filter pushdown yet), so keep remote exploratory queries bounded with
+`LIMIT` or use a coarse resolution level.

examples/demo.zarr/lat/zarr.json

@@ -0,0 +1,45 @@
+{
+  "shape": [
+    4
+  ],
+  "data_type": "float64",
+  "chunk_grid": {
+    "name": "regular",
+    "configuration": {
+      "chunk_shape": [
+        4
+      ]
+    }
+  },
+  "chunk_key_encoding": {
+    "name": "default",
+    "configuration": {
+      "separator": "/"
+    }
+  },
+  "fill_value": "NaN",
+  "codecs": [
+    {
+      "name": "bytes",
+      "configuration": {
+        "endian": "little"
+      }
+    },
+    {
+      "name": "zstd",
+      "configuration": {
+        "level": 0,
+        "checksum": false
+      }
+    }
+  ],
+  "attributes": {
+    "_FillValue": "AAAAAAAA+H8="
+  },
+  "dimension_names": [
+    "lat"
+  ],
+  "zarr_format": 3,
+  "node_type": "array",
+  "storage_transformers": []
+}

examples/demo.zarr/lon/zarr.json

@@ -0,0 +1,45 @@
+{
+  "shape": [
+    5
+  ],
+  "data_type": "float64",
+  "chunk_grid": {
+    "name": "regular",
+    "configuration": {
+      "chunk_shape": [
+        5
+      ]
+    }
+  },
+  "chunk_key_encoding": {
+    "name": "default",
+    "configuration": {
+      "separator": "/"
+    }
+  },
+  "fill_value": "NaN",
+  "codecs": [
+    {
+      "name": "bytes",
+      "configuration": {
+        "endian": "little"
+      }
+    },
+    {
+      "name": "zstd",
+      "configuration": {
+        "level": 0,
+        "checksum": false
+      }
+    }
+  ],
+  "attributes": {
+    "_FillValue": "AAAAAAAA+H8="
+  },
+  "dimension_names": [
+    "lon"
+  ],
+  "zarr_format": 3,
+  "node_type": "array",
+  "storage_transformers": []
+}