diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e5ef523..cc4799d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -12,17 +12,33 @@ reading [Xarray's contributing guide](https://docs.xarray.dev/en/stable/contribu ## Developer setup -We use [uv](https://docs.astral.sh/uv/) to manage the project. +We use [uv](https://docs.astral.sh/uv/) to manage the project. This project +also contains a Rust extension (built with [maturin](https://www.maturin.rs/)), +so a Rust toolchain is required. -0. Install uv: https://docs.astral.sh/uv/getting-started/installation/ -1. Clone the repository (bonus: [via SSH](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account)) - and `cd xarray_sql` (the project root). -2. Install dev dependencies: `uv sync --dev` -3. Install pre-commit hooks: `uvx pre-commit install` +0. Install Rust: https://rustup.rs/ +1. Install uv: https://docs.astral.sh/uv/getting-started/installation/ +2. Clone the repository (bonus: [via SSH](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account)) + and `cd xarray-sql` (the project root). +3. Install Python dev dependencies (without building the Rust extension yet): + ```shell + uv sync --dev --no-install-package xarray-sql + ``` +4. Build and install the Rust extension into the virtual environment: + ```shell + uv run --no-project maturin develop --uv + ``` + This compiles the native code and links it so that `import xarray_sql` works. + Re-run this step whenever you modify any Rust source files under `src/`. +5. Run the test suite to verify your setup: + ```shell + uv run --no-project pytest -v . -m "not integration" + ``` +6. Install pre-commit hooks: `uvx pre-commit install` This will automatically run code formatting and type checking before each commit. You can also run the hooks manually with: `uvx pre-commit run --all-files` -4. Build and serve docs locally: `uvx zensical serve` +7. Build and serve docs locally: `uvx zensical serve` ## Before submitting a pull request... diff --git a/README.md b/README.md index b6169e7..f3408b6 100644 --- a/README.md +++ b/README.md @@ -130,10 +130,12 @@ and BigQuery. More thoughts on this in [#4](https://github.com/alxmrs/xarray-sql/issues/4). _2025 update_: Something like this is being built across a few projects! The ones I know about are: + - [CartoDB's Raquet](https://github.com/CartoDB/raquet) - The DataFusion community's [arrow-zarr](https://github.com/datafusion-contrib/arrow-zarr) -_2026 update_: A collegue and I are experimenting with native Zarr RDBMS engines. Check out: +_2026 update_: A colleague and I are experimenting with native Zarr RDBMS engines. Check out: + - [Zarr-Datafusion](https://lib.rs/crates/zarr-datafusion) - [DuckDB-Zarr](https://github.com/hobbes-bot/duckdb-zarr) diff --git a/xarray_sql/df.py b/xarray_sql/df.py index 353a3db..4b5dafc 100644 --- a/xarray_sql/df.py +++ b/xarray_sql/df.py @@ -228,7 +228,7 @@ def iter_record_batches( ) -> Iterator[pa.RecordBatch]: """Yield RecordBatches of at most *batch_size* rows from a partition Dataset. - Unlike :func:`dataset_to_record_batch`, which materialises the entire + Unlike `dataset_to_record_batch`, which materialises the entire partition as one batch, this generator emits smaller batches so that DataFusion can begin filtering and aggregating before the full partition is loaded. Peak memory per batch is O(batch_size) for coordinate columns diff --git a/xarray_sql/reader.py b/xarray_sql/reader.py index 1460242..93db2d6 100644 --- a/xarray_sql/reader.py +++ b/xarray_sql/reader.py @@ -203,14 +203,15 @@ def read_xarray_table( Each chunk becomes a separate partition, enabling DataFusion's parallel execution across multiple cores. - Filter Pushdown: + Note: SQL queries with WHERE clauses on dimension columns (time, lat, lon, etc.) - automatically prune partitions that can't contain matching rows. For example: + automatically prune partitions that can't contain matching rows — this is + called *filter pushdown*. For example: # This query will skip loading partitions with time < '2020-02-01' - result = ctx.sql('SELECT * FROM air WHERE time > \"2020-02-01\"').collect() + result = ctx.sql('SELECT * FROM air WHERE time > "2020-02-01"').collect() - Supported operators: =, <, >, <=, >=, BETWEEN, IN, AND, OR. + Supported operators: `=`, `<`, `>`, `<=`, `>=`, `BETWEEN`, `IN`, `AND`, `OR`. Args: ds: An xarray Dataset. All data_vars must share the same dimensions.