Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 23 additions & 7 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,17 +12,33 @@ reading [Xarray's contributing guide](https://docs.xarray.dev/en/stable/contribu

## Developer setup

We use [uv](https://docs.astral.sh/uv/) to manage the project.
We use [uv](https://docs.astral.sh/uv/) to manage the project. This project
also contains a Rust extension (built with [maturin](https://www.maturin.rs/)),
so a Rust toolchain is required.

0. Install uv: https://docs.astral.sh/uv/getting-started/installation/
1. Clone the repository (bonus: [via SSH](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account))
and `cd xarray_sql` (the project root).
2. Install dev dependencies: `uv sync --dev`
3. Install pre-commit hooks: `uvx pre-commit install`
0. Install Rust: https://rustup.rs/
1. Install uv: https://docs.astral.sh/uv/getting-started/installation/
2. Clone the repository (bonus: [via SSH](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account))
and `cd xarray-sql` (the project root).
3. Install Python dev dependencies (without building the Rust extension yet):
```shell
uv sync --dev --no-install-package xarray-sql
```
4. Build and install the Rust extension into the virtual environment:
```shell
uv run --no-project maturin develop --uv
```
This compiles the native code and links it so that `import xarray_sql` works.
Re-run this step whenever you modify any Rust source files under `src/`.
5. Run the test suite to verify your setup:
```shell
uv run --no-project pytest -v . -m "not integration"
```
6. Install pre-commit hooks: `uvx pre-commit install`

This will automatically run code formatting and type checking before each commit.
You can also run the hooks manually with: `uvx pre-commit run --all-files`
4. Build and serve docs locally: `uvx zensical serve`
7. Build and serve docs locally: `uvx zensical serve`


## Before submitting a pull request...
Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,10 +130,12 @@ and BigQuery. More thoughts on this
in [#4](https://github.com/alxmrs/xarray-sql/issues/4).

_2025 update_: Something like this is being built across a few projects! The ones I know about are:

- [CartoDB's Raquet](https://github.com/CartoDB/raquet)
- The DataFusion community's [arrow-zarr](https://github.com/datafusion-contrib/arrow-zarr)

_2026 update_: A collegue and I are experimenting with native Zarr RDBMS engines. Check out:
_2026 update_: A colleague and I are experimenting with native Zarr RDBMS engines. Check out:

- [Zarr-Datafusion](https://lib.rs/crates/zarr-datafusion)
- [DuckDB-Zarr](https://github.com/hobbes-bot/duckdb-zarr)

Expand Down
2 changes: 1 addition & 1 deletion xarray_sql/df.py
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ def iter_record_batches(
) -> Iterator[pa.RecordBatch]:
"""Yield RecordBatches of at most *batch_size* rows from a partition Dataset.

Unlike :func:`dataset_to_record_batch`, which materialises the entire
Unlike `dataset_to_record_batch`, which materialises the entire
partition as one batch, this generator emits smaller batches so that
DataFusion can begin filtering and aggregating before the full partition
is loaded. Peak memory per batch is O(batch_size) for coordinate columns
Expand Down
9 changes: 5 additions & 4 deletions xarray_sql/reader.py
Original file line number Diff line number Diff line change
Expand Up @@ -203,14 +203,15 @@ def read_xarray_table(
Each chunk becomes a separate partition, enabling DataFusion's parallel
execution across multiple cores.

Filter Pushdown:
Note:
SQL queries with WHERE clauses on dimension columns (time, lat, lon, etc.)
automatically prune partitions that can't contain matching rows. For example:
automatically prune partitions that can't contain matching rows — this is
called *filter pushdown*. For example:

# This query will skip loading partitions with time < '2020-02-01'
result = ctx.sql('SELECT * FROM air WHERE time > \"2020-02-01\"').collect()
result = ctx.sql('SELECT * FROM air WHERE time > "2020-02-01"').collect()

Supported operators: =, <, >, <=, >=, BETWEEN, IN, AND, OR.
Supported operators: `=`, `<`, `>`, `<=`, `>=`, `BETWEEN`, `IN`, `AND`, `OR`.

Args:
ds: An xarray Dataset. All data_vars must share the same dimensions.
Expand Down
Loading