Add post: Lumen now speaks SQL on N-dimensional scientific data#99
Closed
ghostiee-11 wants to merge 7 commits into
Closed
Add post: Lumen now speaks SQL on N-dimensional scientific data#99ghostiee-11 wants to merge 7 commits into
ghostiee-11 wants to merge 7 commits into
Conversation
added 7 commits
June 5, 2026 00:34
Writeup of XArraySQLSource (holoviz/lumen#1741) for the xarray, Zarr, and Pangeo audience. Covers the architecture, a real ARCO-ERA5 query end to end, the metadata that survives the SQL round-trip, and the design decisions behind per-variable table registration and the xarray-sql + DataFusion stack. Adds 9 files under posts/2026-06-05-lumen-xarray-source/ (index.qmd + 8 PNGs in images/). No project-level config or CSS changes.
…oken xarray-sql repo URL Section 5 was overclaiming a Dataset round-trip that does not exist in main today. source.execute(sql) calls .to_pandas() on the DataFusion result; the SQL result is always a pandas DataFrame. The coordinate metadata is only accessible via source.get_schema(table), not on the result itself. - Renamed section 'What stays intact' to 'Where the metadata lives' with honest prose: result type today is DataFrame, metadata is on the source schema, the full Dataset round-trip is upstream future work tracked at xqlsystems/xarray-sql#58. - Replaced diagram-4-roundtrip.png (input -> SQL -> Dataset out with green checks, depicting xarray-sql#58 design goal) with diagram-4-metadata.png (three-panel comparison showing input, result-today as DataFrame, and the vermillion-accented schema card where the metadata actually lives). - Fixed broken xarray-sql repo URLs from xarray-contrib/xarray-sql (which does not exist) to alxmrs/xarray-sql (the real Apache project by Alex Merose). - Softened opening summary and post description to match: 'metadata exposed through the source schema' instead of 'preserved through the SQL query'.
Re-rendered the 3 cell mockups and the D3 result map with CSS zoom:2 on the same HTML sources, then cropped to taste. Same compositions and data, sharper at typical blog content widths. - nb-cell-03.png (NOAA air_temperature, STANDARD) - nb-cell-07.png (ERA-Interim u/v/z, MULTI-VAR) - nb-cell-09.png (10M-row scale, SCALE) - result-map.png (D3 global map of the SELECT result)
Previous crops downsampled the 2x source back to ~800px wide, defeating the upscale. New pass element-targets .stage (caption + card, no surrounding cream-grid background), giving a tight crop at full 2x density with no manual cropping needed.
The auto-recapture at .stage + zoom:2 produced sharper pixels but the framing was off (cell-03 had excess trailing background). Going back to the user-cropped versions from one commit prior.
ghostiee-11
marked this pull request as ready for review
June 4, 2026 20:35
Collaborator
Author
|
Hey @ahuang11!! Can you please review this.. |
Contributor
|
Can you do me a favor and rebase onto holoviz-dev/blog instead of a fork so the build passes? |
Collaborator
Author
|
yeah sure, also got one more thing, the 4th diagram is bit blurry and not generated properly |
ahuang11
requested changes
Jun 4, 2026
ahuang11
left a comment
ahuang11
left a comment
Contributor
There was a problem hiding this comment.
Great start! I think it should be more geared towards Lumen AI, but you can mention the machinery underneath. Also some parts kind of read like AI. I'd like to share this https://matthewrocklin.com/blog/work/2020/07/13/brevity as what I usually like to do.
|
|
||
| {fig-align="center"} | ||
|
|
||
| Lumen is a SQL-first framework. That makes it natural for tabular |
Contributor
There was a problem hiding this comment.
Most people don't really know what Lumen is. Perhaps you can have a brief intro.
| watching the coordinates, units, and attributes evaporate on the way | ||
| in. The Dask graph behind a lazy Zarr disappeared with them. | ||
|
|
||
| Lumen has a proper Source for this: `XArraySQLSource`. It takes any |
Contributor
There was a problem hiding this comment.
I think you should highlight that you worked on this too on the way to support XArray.
| This post walks through what landed, how the source is shaped, what | ||
| it preserves, and what is next on the xarray integration roadmap. | ||
|
|
||
| ## A two-minute recap of Lumen |
Contributor
There was a problem hiding this comment.
Maybe link to this in above.
Comment on lines
+46
to
+50
| If you have not used Lumen before: | ||
| [Lumen](https://lumen.holoviz.org) is the declarative data | ||
| application framework in the HoloViz stack. You describe a `Source` | ||
| (where data comes from), a `Pipeline` (how to filter and transform | ||
| it), and a `View` (how to render it). Lumen wires the three together |
Contributor
There was a problem hiding this comment.
Most people will not interact with Lumen like this; let's highlight the AI of Lumen instead.
| Every downstream consumer now sees a flat table with no idea it | ||
| ever knew about coordinates. | ||
|
|
||
| {fig-align="center"} |
Collaborator
Author
|
Superseded: I rebased onto holoviz-dev/blog directly as you suggested @ahuang11, see the new PR with the same content (the deploy + auto-commit steps should pass now). |
Contributor
|
Oops just commented :D can you address those and tag me again? |
Collaborator
Author
|
Yeah sure, lemme fix all that and then will tag you |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New post: "Lumen now speaks SQL on N-dimensional scientific data". A writeup of the recently-landed
XArraySQLSource(lumen#1741).The post explains the new capability for an xarray / Zarr / Pangeo audience and walks through the architecture, a real query end-to-end on ARCO-ERA5, the metadata that survives the SQL round-trip, and the design decisions behind per-variable table registration and the xarray-sql plus DataFusion stack.
Assets