Add xarray-sql support#1434
Closed
ahuang11 wants to merge 3 commits into
Closed
Conversation
Member
|
verrrry interesting.. It would be worthwhile to scope out how well it could support anndata |
1 task
ghostiee-11
added a commit
to ghostiee-11/lumen
that referenced
this pull request
Mar 11, 2026
Adds XArraySQLSource(BaseSQLSource) that uses xarray-sql (Apache DataFusion) to provide full SQL query capabilities over xarray datasets. Each data variable is exposed as a separate SQL table with coordinate columns. Ref: holoviz#1434, holoviz#1508 - New source: lumen/sources/xarray_sql.py - SQL queries via DataFusion (postgres-compatible dialect) - Supports NetCDF, Zarr, HDF5, GRIB formats - Auto-engine detection from file extension - Dask-backed lazy chunking - Rich metadata from xarray attributes (units, dimensions, coordinates) - Dimension info API for UI controls and AI context - Async execute/get for non-blocking Lumen AI agents - create_sql_expr_source with shared DataFusion context - Full to_spec/from_spec serialization - Optional import -- graceful fallback when xarray not installed - Tests: 50 tests covering construction, SQL execution, schema, metadata, normalization, serialization, async, resource management - Optional dependency group: pip install lumen[xarray]
This was referenced Mar 11, 2026
5 tasks
Contributor
Author
|
Superseded #1741 |
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Just a proof of concept integrating https://github.com/alxmrs/xarray-sql
lumen-ai serve air.ncI wonder if this should be a lumen extension, like lumen-anndata. Part of me feels like it should be in core, so it works like hvplot.xarray, but at the same time, unsure about the limitations about the SQL & xarray.
I think if xarray-sql supported xarray-datatrees, then it's possible to rebuild AnndataSource on this since xarray is a generalized library supporting n-D dimension datasets.