Skip to content
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
176228e
Add ds.py wrapper, eager round-trip foundation (Phase 1)
ghostiee-11 May 12, 2026
0a8b15a
Add lazy SQLBackendArray, default to lazy round-trip (Phase 2)
ghostiee-11 May 12, 2026
18c6d1c
Sparse-extent handling and edge cases (Phase 3)
ghostiee-11 May 13, 2026
36dfb4b
Document SQL -> xarray round-trip in README (Phase 4)
ghostiee-11 May 13, 2026
479133c
Optimize lazy memory: skip DISTINCT scans + omit redundant WHERE clauses
ghostiee-11 May 13, 2026
f50ba86
Tighten type annotations and cover vectorized indexer path
ghostiee-11 May 13, 2026
ade544a
Rearchitect lazy backend on DataFusion DataFrame API + Arrow streaming
ghostiee-11 May 28, 2026
289bedb
Merge remote-tracking branch 'origin/main' into feat/lazy-sql-to-xarray
ghostiee-11 May 28, 2026
50b9f52
fix: changes
ghostiee-11 Jun 6, 2026
7b7518b
fix: address round-4 review
ghostiee-11 Jun 12, 2026
f5f0160
fix : conflicts
ghostiee-11 Jun 12, 2026
ac53d04
Improve performance: execute passes for to_dataframe _once_.
alxmrs Jun 14, 2026
9c48b53
Major fix: gutting the introspection path. Using output chunks as a k…
alxmrs Jun 14, 2026
6d50108
Documented the current feature in the README.
alxmrs Jun 14, 2026
8d58e82
dimension_columns --> dims
alxmrs Jun 14, 2026
f535f3c
Partition-snapped "auto" chunks implementation.
alxmrs Jun 14, 2026
221a28d
rm reverse docs, they can be documented elsewhere. Also, it currently…
alxmrs Jun 14, 2026
3d49c4a
unifiying "template_table" with just the "template" argument.
alxmrs Jun 14, 2026
1a29fa4
chore: ruff format
ghostiee-11 Jun 15, 2026
974b5e3
Attribution.
alxmrs Jun 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,59 @@ _(A runnable version of this example lives at
Succinctly, we "pivot" Xarray Datasets to treat them like tables so we can run
SQL queries against them.

## Round-tripping back to Xarray

`ctx.sql(...)` returns an `XarrayDataFrame` that exposes `.to_pandas()`
(unchanged) and a new `.to_dataset()` for converting the result back into
an `xr.Dataset`. The reverse path is **lazy by default**: the returned
Dataset is backed by an `xarray.backends.BackendArray` that translates
xarray indexers into DataFusion `filter` expressions and consumes the
filtered DataFrame via `execute_stream`. Arrow `RecordBatch` es scatter
directly into a preallocated numpy buffer with no pandas hop, so only
the slab actually accessed is materialized.

```python
out = ctx.sql('SELECT * FROM "air"').to_dataset()
# <xarray.Dataset>
# Dimensions: (time: 2920, lat: 25, lon: 53)
# Coordinates:
# * time (time) datetime64[ns] ...
# * lat (lat) float32 ...
# * lon (lon) float32 ...
# Data variables:
# air (time, lat, lon) float32 ...

# Slicing pushes down into DataFusion; only the requested slab is
# materialized.
slab = out["air"].isel(time=0).values

# For full eager materialization, call .compute().
eager = out.compute()
```

`dimension_columns` defaults to the dims of the single registered Dataset
on the context (or the one named via `template_table=` when several are
registered). Variable attrs, dataset attrs, non-dimension coordinates,
and dim-coordinate dtype are recovered from the registered Dataset
automatically.

For filtered queries that return only part of the original extent, pass
`sparse_extent="template"` to reindex back to the full grid with NaN

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These names seem unintuitive to me, but I like that we can make sparse representations of the Dataset.

fills:

```python
out = ctx.sql(
'SELECT * FROM "air" WHERE lat > 50'
).to_dataset(sparse_extent="template")
# Full lat range restored; cells with lat <= 50 are NaN.
```

Aggregation queries (e.g. `AVG(air) AS air_avg ... GROUP BY lat, lon`)
materialize once because their output does not align with the source dim
structure; the aggregation path is also Arrow-native (no pandas
intermediates). Pass `dimension_columns=[...]` explicitly when an
aggregation drops a dim.

## Why build this?

A few reasons:
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ module = [
"pyarrow.*",
"datafusion.*",
"xarray.*",
"pandas.*",
]
ignore_missing_imports = true

Expand Down
Loading
Loading