diff --git a/posts/2026-06-05-lumen-xarray-source/images/diagram-1-hero.png b/posts/2026-06-05-lumen-xarray-source/images/diagram-1-hero.png new file mode 100644 index 0000000..e1c0769 Binary files /dev/null and b/posts/2026-06-05-lumen-xarray-source/images/diagram-1-hero.png differ diff --git a/posts/2026-06-05-lumen-xarray-source/images/diagram-2-before-after.png b/posts/2026-06-05-lumen-xarray-source/images/diagram-2-before-after.png new file mode 100644 index 0000000..f9de745 Binary files /dev/null and b/posts/2026-06-05-lumen-xarray-source/images/diagram-2-before-after.png differ diff --git a/posts/2026-06-05-lumen-xarray-source/images/diagram-3-architecture.png b/posts/2026-06-05-lumen-xarray-source/images/diagram-3-architecture.png new file mode 100644 index 0000000..700d385 Binary files /dev/null and b/posts/2026-06-05-lumen-xarray-source/images/diagram-3-architecture.png differ diff --git a/posts/2026-06-05-lumen-xarray-source/images/diagram-4-metadata.png b/posts/2026-06-05-lumen-xarray-source/images/diagram-4-metadata.png new file mode 100644 index 0000000..4b7a9eb Binary files /dev/null and b/posts/2026-06-05-lumen-xarray-source/images/diagram-4-metadata.png differ diff --git a/posts/2026-06-05-lumen-xarray-source/images/nb-cell-03.png b/posts/2026-06-05-lumen-xarray-source/images/nb-cell-03.png new file mode 100644 index 0000000..2d565d5 Binary files /dev/null and b/posts/2026-06-05-lumen-xarray-source/images/nb-cell-03.png differ diff --git a/posts/2026-06-05-lumen-xarray-source/images/nb-cell-07.png b/posts/2026-06-05-lumen-xarray-source/images/nb-cell-07.png new file mode 100644 index 0000000..825c6fd Binary files /dev/null and b/posts/2026-06-05-lumen-xarray-source/images/nb-cell-07.png differ diff --git a/posts/2026-06-05-lumen-xarray-source/images/nb-cell-09.png b/posts/2026-06-05-lumen-xarray-source/images/nb-cell-09.png new file mode 100644 index 0000000..22a6d1d Binary files /dev/null and b/posts/2026-06-05-lumen-xarray-source/images/nb-cell-09.png differ diff --git a/posts/2026-06-05-lumen-xarray-source/images/result-map.png b/posts/2026-06-05-lumen-xarray-source/images/result-map.png new file mode 100644 index 0000000..3e3810d Binary files /dev/null and b/posts/2026-06-05-lumen-xarray-source/images/result-map.png differ diff --git a/posts/2026-06-05-lumen-xarray-source/index.qmd b/posts/2026-06-05-lumen-xarray-source/index.qmd new file mode 100644 index 0000000..51c07bb --- /dev/null +++ b/posts/2026-06-05-lumen-xarray-source/index.qmd @@ -0,0 +1,312 @@ +--- +title: "Lumen now speaks SQL on N-dimensional scientific data" +description: | + XArraySQLSource brings xarray, NetCDF and cloud Zarr into Lumen + pipelines. Coordinate metadata is exposed through the source + schema, and Dask-backed laziness carries through to + multi-million-row queries. +author: + - name: Aman Kumar + url: https://github.com/ghostiee-11 +date: 2026-06-05 +date-format: "MMM D, YYYY" +categories: [lumen, xarray, sql, scientific-python, gsoc, zarr, netcdf] +image: images/diagram-1-hero.png +image-alt: "An isometric chunked Zarr cube on the left, a thick vermillion arrow labelled XArraySQLSource in the middle, and an xarray.Dataset summary card on the right." +toc: true +toc-depth: 2 +--- + +![](images/diagram-1-hero.png){fig-align="center"} + +Lumen is a SQL-first framework. That makes it natural for tabular +data and awkward for the multi-dimensional scientific datasets that +live in xarray. Until now, feeding an `xarray.Dataset` into a Lumen +pipeline meant flattening the cube to a pandas DataFrame and +watching the coordinates, units, and attributes evaporate on the way +in. The Dask graph behind a lazy Zarr disappeared with them. + +Lumen has a proper Source for this: `XArraySQLSource`. It takes any +`xarray.Dataset`, exposes its data variables as SQL tables with +full coordinate columns, and routes queries through +[xarray-sql](https://github.com/alxmrs/xarray-sql) and +[Apache DataFusion](https://github.com/apache/datafusion) under the +hood. NetCDF, Zarr, HDF5 and GRIB all flow through +`xr.open_dataset` automatically. Cloud stores work through the same +anonymous fsspec credentials xarray already accepts. Coordinate +metadata is exposed through the source schema, and Dask-backed +laziness carries through to multi-million-row queries without +materialising the grid. + +This post walks through what landed, how the source is shaped, what +it preserves, and what is next on the xarray integration roadmap. + +## A two-minute recap of Lumen + +If you have not used Lumen before: +[Lumen](https://lumen.holoviz.org) is the declarative data +application framework in the HoloViz stack. You describe a `Source` +(where data comes from), a `Pipeline` (how to filter and transform +it), and a `View` (how to render it). Lumen wires the three together +into a deployable Panel app, an embeddable component, or, since the +[Lumen AI 1.0 release earlier this year](https://blog.holoviz.org/posts/lumen_1.0/), +a chat-driven exploration surface. + +The whole framework is built around SQL. Sources expose tables, the +Pipeline composes SQL transforms, and the AI agent reasons about +columns and joins. That is a great model for relational data and it +is exactly the model that scientific N-dimensional data sits +awkwardly inside. + +## The mismatch + +A climate scientist has a NetCDF file: `2m_temperature` over latitude, +longitude and time. In xarray, that variable is a 4D array with +named dims (`time`, `latitude`, `longitude`), coordinate arrays with +units (`degrees_north`, `K`), and an `attrs` dict carrying things +like `long_name` and provenance. In SQL, the same data is a flat +table with columns `time, latitude, longitude, 2m_temperature` and +no awareness that those columns describe a regular grid. + +Until now, the only way to feed an xarray dataset into a Lumen +`SQLTransform` was the lossy path: open with xarray, call +`.to_dataframe()`, hand the resulting `pandas.DataFrame` to Lumen. +The dims become regular columns. The coordinate units are gone. The +`attrs` are gone. The Dask graph that backed your lazy Zarr is gone. +Every downstream consumer now sees a flat table with no idea it +ever knew about coordinates. + +![Before vs after, the data path through Lumen.](images/diagram-2-before-after.png){fig-align="center"} + +The XArraySQLSource replaces that flattening step with a translator +that keeps the structure intact. + +## What XArraySQLSource is + +It is a Lumen `Source` subclass that wraps any `xarray.Dataset`, +either by URI: + +```python +from lumen.sources.xarray_sql import XArraySQLSource + +source = XArraySQLSource( + uri="path/to/data.nc", # or .zarr, .h5, .grib, or a gs://... URL + engine="zarr", # auto-detected from extension if omitted + open_kwargs={"consolidated": True}, + chunks="auto", +) +``` + +or, when you already hold an opened `xarray.Dataset` (for example +from `xr.tutorial`, an intake catalog, or a `xarray-beam` pipeline), +through the `from_dataset` classmethod: + +```python +import xarray as xr +ds = xr.tutorial.open_dataset("air_temperature") +source = XArraySQLSource.from_dataset(ds) +``` + +The two share the same downstream behaviour. On construction, the +source walks the dataset and registers every data variable as its +own SQL table with `xarray-sql`'s `XarrayContext`. Coordinate arrays +become regular columns of that table. Chunks are forwarded to +DataFusion so each query can process data lazily. + +![The architecture, layer by layer.](images/diagram-3-architecture.png){fig-align="center"} + +Once the dataset is registered, the rest of Lumen treats the source +like any other SQL source. Schema introspection works the same way. +SQLTransforms compose against the registered table names. Filters +push predicates down. The Lumen UI sees the variables as queryable +tables and presents them through the same controls it uses for +DuckDB or Snowflake or BigQuery. + +## A first look + +Wrap a dataset, list the tables, run a query. Here is the standard +NCEP/NCAR reanalysis end to end: + +![Standard NCEP/NCAR `air_temperature` reanalysis: schema, a SELECT, the result table, and the full metadata block.](images/nb-cell-03.png){fig-align="center"} + +Two things to notice. First, the source reports its schema as a +real schema, with dtypes and human-readable descriptions pulled +from the xarray `attrs`. The Lumen agent and the SQLTransform layer +both consume that. Second, the variable name `air` is now a SQL +identifier; the query is plain DataFusion SQL with no Lumen-specific +syntax to learn. + +Multi-variable datasets register everything in one pass. ERA-Interim's +bundled three-variable subset of zonal wind, meridional wind, and +geopotential becomes three SQL tables you can query independently: + +![ERA-Interim u, v, z. Three variables, three SQL tables, no extra wiring.](images/nb-cell-07.png){fig-align="center"} + +Cross-variable queries through joins do not work yet (each variable +is registered separately), but per-variable queries with their own +coordinate filters are direct, and that is the common case for +scientific exploration. + +## Scale and shape + +The next question every scientific user asks is: does it actually +hold up on real data, or does the source materialise the whole grid +and run out of memory? + +DataFusion processes the registered Xarray-backed tables lazily. +Aggregations, filters and projections push down through the chunk +graph. A 10 million row synthetic dataset on a laptop runs three +back-to-back queries (count, aggregation, filtered region) without +materialising: + +![10M-row synthetic dataset: SELECT COUNT, AVG/MIN/MAX aggregations, and a filtered region query, all running lazily.](images/nb-cell-09.png){fig-align="center"} + +On the cloud, the same code points at a Zarr URL with anonymous +credentials. ARCO-ERA5, the official Analysis-Ready Cloud-Optimized +ERA5 reanalysis hosted by ECMWF and Google Public Datasets, opens +in a few seconds: + +```python +source = XArraySQLSource( + uri="gs://gcp-public-data-arco-era5/co/single-level-reanalysis.zarr-v2", + engine="zarr", + open_kwargs={ + "consolidated": True, + "storage_options": {"token": "anon"}, + }, + variables=["2m_temperature"], +) +``` + +A query for the monthly tropical mean over January through March 2020 +is a plain SQL statement. DataFusion only reads the chunks that +intersect the WHERE clause: + +```sql +SELECT + DATE_TRUNC('month', "time") AS month, + AVG("2m_temperature") AS mean_t2m_k +FROM "2m_temperature" +WHERE "latitude" BETWEEN -10 AND 10 + AND "time" BETWEEN '2020-01-01' AND '2020-04-01' +GROUP BY month +ORDER BY month +``` + +The result comes back as a tidy table with the coordinate columns +intact, and projecting it onto a map is the same one-liner you would +write against any tabular result: + +![The query result, projected on a global map. Colours follow the latitude band the NCEP/NCAR demo grid actually covers.](images/result-map.png){fig-align="center"} + +The full pipeline (load, query, transform, visualize) was validated +across [seven dataset types and chunking edge cases](https://gist.github.com/ghostiee-11/42f42d6cfb2ce2c1ed4f68e7c1fe7671) +before the PR landed: NCEP, RASM with non-linear `xc/yc` coordinates, +ERA-Interim multi-variable, a 10M-row synthetic stress test, Zarr, +HDF5, and one documented upstream limit on mixed-dimension chunking. +Each was executed and verified. + +## Where the metadata lives + +The result of `source.execute(sql)` is, today, a pandas DataFrame. +Coordinate metadata does not ride along on that result. What does +get preserved, and where Lumen actually consumes it, is the source +schema. + +![Where the metadata lives.](images/diagram-4-metadata.png){fig-align="center"} + +In practice this means: + +- **Dims** are exposed as named SQL columns, so a WHERE clause on + `latitude` filters along the latitude axis. No need to remember + which integer index corresponded to which physical coordinate. +- **Coordinate units** (`degrees_north`, `K`, `datetime64[ns]`), + variable `dtype`, and `attrs` like `long_name` and + `standard_name` are all accessible through + `source.get_schema(table)`. The Lumen AI agent and downstream + views consume them from there, not from the result DataFrame. +- **Dask laziness** carries from the underlying Zarr or NetCDF + through xarray-sql into DataFusion's query plan. Filters push + down, aggregations stream. The result is materialised to pandas + only at the very end of the query. +- **What is not on the result today**: the `dims` object, + coordinate units, and `attrs`. Those live on the source. The full + Dataset round-trip, where a SQL result comes back as an + `xarray.Dataset` with all coordinate metadata preserved on the + result itself, is tracked upstream at + [xarray-sql#58](https://github.com/alxmrs/xarray-sql/issues/58). + +The deliberate non-goal here is to make xarray and SQL look like +the same thing. They are not. SQL is unaware of grid structure, +`Dataset.sel(time=..., lat=...)` is. The XArraySQLSource gives +Lumen a faithful SQL view of the data, with coordinate metadata +introspectable through the source, without pretending the +underlying representation is flat. + +## Design decisions + +Three choices in the source are worth calling out because each +opens or closes a future direction. + +**Per-variable table registration.** A `Dataset` with three +variables becomes three SQL tables, not one wide one. DataFusion +expects each registered table to be rectangular, and a `Dataset` +can hold variables with different dims, so a single combined table +is not always definable. Splitting per variable also keeps the +schema readable and makes pushdown work cleanly. The trade-off is +that joins across variables are a separate problem to solve, and +right now you handle them in application code rather than in SQL. + +**xarray-sql plus DataFusion under the hood.** xarray-sql is the +existing Apache project for SQL-on-xarray. DataFusion is a Rust +query engine with a Python binding, good predicate pushdown, and a +mature relational planner. The alternative was to lean on +DuckDB-on-pandas, which would have meant materialising every query +result into pandas. That foreclosure on lazy compute felt worse +than the cost of an additional dependency. xarray-sql also gives +Lumen a direct path to contribute upstream when the round-trip +back to `xarray.Dataset` lands ([xarray-sql#58](https://github.com/alxmrs/xarray-sql/issues/58)). + +**`uri=` or `from_dataset()`, never both.** The constructor accepts +either a path the source opens itself, or a pre-opened Dataset +through the classmethod. The path keeps simple cases as a one-liner. +The classmethod keeps the door open for everything xarray ecosystem +people already do: pulling from intake catalogs, slicing in +xarray-beam pipelines, hand-constructing datasets. The two cases +share all the downstream code. + +## Get started + +The source is in `holoviz/lumen` main today: + +```bash +pip install lumen[xarray] +``` + +Then: + +```python +from lumen.sources.xarray_sql import XArraySQLSource +``` + +Source code lives at +[`lumen/sources/xarray_sql.py`](https://github.com/holoviz/lumen/blob/main/lumen/sources/xarray_sql.py). +The original PR with the full review history is +[holoviz/lumen#1741](https://github.com/holoviz/lumen/pull/1741). +The +[edge-case validation Gist](https://gist.github.com/ghostiee-11/42f42d6cfb2ce2c1ed4f68e7c1fe7671) +covers the seven dataset shapes used during development. + +Feedback, edge cases, and bug reports are welcome on the +[Lumen issue tracker](https://github.com/holoviz/lumen/issues), and +the next post in this series will cover the AI-agent side of the +integration, where the natural-language layer learns to ask +xarray-shaped questions. + +### Credits + +`XArraySQLSource` was contributed by [Aman Kumar](https://github.com/ghostiee-11) +during Google Summer of Code 2026, with HoloViz under the +[NumFOCUS](https://numfocus.org) umbrella. Design review and PR +feedback by [Andrew Huang](https://github.com/ahuang11) and +[Andy Maloney](https://github.com/amaloney).