Skip to content

Commit 7ba9be0

Browse files
authored
Revised the README with recent context.
1 parent fbb5c60 commit 7ba9be0

1 file changed

Lines changed: 11 additions & 15 deletions

File tree

README.md

Lines changed: 11 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -94,11 +94,11 @@ That's it!
9494
_2025 update_: This library now implements a Dask-like `from_map` interface in
9595
pure DataFusion and PyArrow, but works with the same principle!
9696

97-
_2026 update_: Instead of `from_map()`, we make factory functions from blocks of
98-
Xarray datasets that return RecordBatchReaders. These feed into a Rust-based
99-
DataFusion `TableProvider`. Every chunk is uses the Arrow in memory format to
100-
translate between Python and Rust. Even still, the core of what makes this idea
101-
work is the core `pivot()` operation from where this project began!
97+
_2026 update_: Instead of `from_map()`, we create a way to translate Xarray chunks
98+
into Arrow RecordBatches. We pass a Python callback into a DataFusion `TableProvider`
99+
that lets the DB engine translate the underlying Dataset arrays into DataFusion partitions.
100+
Ultimately, the initial insight of the `pivot()` function -- that any ndarray can be
101+
translated into a 2D table -- underlies this performant query mechanism.
102102

103103
## Why does this work?
104104

@@ -116,11 +116,6 @@ early users – "tire kickers", if you will. We'd love your input to shape the d
116116
project! Please, give this a try and [file issues](https://github.com/alxmrs/xarray-sql/issues) as
117117
you see fit. Check out our [contributing guide](CONTRIBUTING.md), too 😉.
118118

119-
I can say that for now, the library is oriented towards making whole scans of
120-
Xarray Datasets. Common filter optimizations (even basic ones like an `.sel()` on
121-
core dimensions, let alone predicate push downs) are not fully implemented yet.
122-
However, these operations and more are on our roadmap.
123-
124119
## What would a deeper integration look like?
125120

126121
I have a few ideas so far. One approach involves applying operations directly on
@@ -138,15 +133,16 @@ _2025 update_: Something like this is being built across a few projects! The one
138133
- [CartoDB's Raquet](https://github.com/CartoDB/raquet)
139134
- The DataFusion community's [arrow-zarr](https://github.com/datafusion-contrib/arrow-zarr)
140135

141-
As of writing, this project is [amid integrating](https://github.com/alxmrs/xarray-sql/pull/69) a
142-
rust-based DataFusion backend provided by arrow-zarr.
136+
_2026 update_: A collegue and I are experimenting with native Zarr RDBMS engines. Check out:
137+
- [Zarr-Datafusion](https://lib.rs/crates/zarr-datafusion)
138+
- [DuckDB-Zarr](https://github.com/hobbes-bot/duckdb-zarr)
143139

144140
## Roadmap
145141

146142
- [x] ~Lazy evaluation via the pyarrow Dataset interface [#93](https://github.com/alxmrs/xarray-sql/issues/93).~ _Implemented in [#100](https://github.com/alxmrs/xarray-sql/pull/100)_
147-
- [ ] Support proper parallelism via proper partition handling on the rust/datafusion side. [#106](https://github.com/alxmrs/xarray-sql/issues/106)
148-
- [ ] Support core datafusion optimizations to scan less data, like [104](https://github.com/alxmrs/xarray-sql/issues/104), ...
149-
- [ ] Translate a single Zarr to a collection of tables via DataFusion's catalog interface [#85](https://github.com/alxmrs/xarray-sql/issues/85).
143+
- [x] Support proper parallelism via proper partition handling on the rust/datafusion side. [#106](https://github.com/alxmrs/xarray-sql/issues/106)
144+
- [x] Support core datafusion optimizations to scan less data, like [104](https://github.com/alxmrs/xarray-sql/issues/104), ...
145+
- [ ] Translate a single Zarr to a collection of tables [#85](https://github.com/alxmrs/xarray-sql/issues/85).
150146
- [ ] Distributed beyond a single node through the DataFusion integration with Ray Datasets [#68](https://github.com/alxmrs/xarray-sql/issues/68) or Apache Ballista [#98](https://github.com/alxmrs/xarray-sql/issues/98).
151147
- [ ] Demo: calculate Sea Surface Temperature from 1940 - Present in SQL [#36](https://github.com/alxmrs/xarray-sql/issues/36).
152148
- [ ] Provide an option to integrate DataFusion directly to Zarr via Rust [#4](https://github.com/alxmrs/xarray-sql/issues/4).

0 commit comments

Comments
 (0)