From 4e831e913c15e7003a27220e16a7e9f426f543a6 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 28 Mar 2026 21:12:38 +0000 Subject: [PATCH] Fix documentation formatting issues - README: add blank lines before markdown list items in "deeper integration" section so they render as proper lists; fix "collegue" typo. - CONTRIBUTING: expand developer setup with step-by-step instructions for installing Rust and building the native maturin extension, adapted from the ci.yml workflow. - xarray_sql/df.py: replace Sphinx-style :func: cross-reference with plain backtick syntax understood by mkdocstrings. - xarray_sql/reader.py: convert non-standard "Filter Pushdown:" section to a proper Google-style Note: section; remove spurious backslash escapes before double-quotes in the embedded SQL example. https://claude.ai/code/session_01TVBHrvmi2vxMJxpLt3U1vB --- CONTRIBUTING.md | 30 +++++++++++++++++++++++------- README.md | 4 +++- xarray_sql/df.py | 2 +- xarray_sql/reader.py | 9 +++++---- 4 files changed, 32 insertions(+), 13 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e5ef523..cc4799d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -12,17 +12,33 @@ reading [Xarray's contributing guide](https://docs.xarray.dev/en/stable/contribu ## Developer setup -We use [uv](https://docs.astral.sh/uv/) to manage the project. +We use [uv](https://docs.astral.sh/uv/) to manage the project. This project +also contains a Rust extension (built with [maturin](https://www.maturin.rs/)), +so a Rust toolchain is required. -0. Install uv: https://docs.astral.sh/uv/getting-started/installation/ -1. Clone the repository (bonus: [via SSH](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account)) - and `cd xarray_sql` (the project root). -2. Install dev dependencies: `uv sync --dev` -3. Install pre-commit hooks: `uvx pre-commit install` +0. Install Rust: https://rustup.rs/ +1. Install uv: https://docs.astral.sh/uv/getting-started/installation/ +2. Clone the repository (bonus: [via SSH](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account)) + and `cd xarray-sql` (the project root). +3. Install Python dev dependencies (without building the Rust extension yet): + ```shell + uv sync --dev --no-install-package xarray-sql + ``` +4. Build and install the Rust extension into the virtual environment: + ```shell + uv run --no-project maturin develop --uv + ``` + This compiles the native code and links it so that `import xarray_sql` works. + Re-run this step whenever you modify any Rust source files under `src/`. +5. Run the test suite to verify your setup: + ```shell + uv run --no-project pytest -v . -m "not integration" + ``` +6. Install pre-commit hooks: `uvx pre-commit install` This will automatically run code formatting and type checking before each commit. You can also run the hooks manually with: `uvx pre-commit run --all-files` -4. Build and serve docs locally: `uvx zensical serve` +7. Build and serve docs locally: `uvx zensical serve` ## Before submitting a pull request... diff --git a/README.md b/README.md index b6169e7..f3408b6 100644 --- a/README.md +++ b/README.md @@ -130,10 +130,12 @@ and BigQuery. More thoughts on this in [#4](https://github.com/alxmrs/xarray-sql/issues/4). _2025 update_: Something like this is being built across a few projects! The ones I know about are: + - [CartoDB's Raquet](https://github.com/CartoDB/raquet) - The DataFusion community's [arrow-zarr](https://github.com/datafusion-contrib/arrow-zarr) -_2026 update_: A collegue and I are experimenting with native Zarr RDBMS engines. Check out: +_2026 update_: A colleague and I are experimenting with native Zarr RDBMS engines. Check out: + - [Zarr-Datafusion](https://lib.rs/crates/zarr-datafusion) - [DuckDB-Zarr](https://github.com/hobbes-bot/duckdb-zarr) diff --git a/xarray_sql/df.py b/xarray_sql/df.py index 353a3db..4b5dafc 100644 --- a/xarray_sql/df.py +++ b/xarray_sql/df.py @@ -228,7 +228,7 @@ def iter_record_batches( ) -> Iterator[pa.RecordBatch]: """Yield RecordBatches of at most *batch_size* rows from a partition Dataset. - Unlike :func:`dataset_to_record_batch`, which materialises the entire + Unlike `dataset_to_record_batch`, which materialises the entire partition as one batch, this generator emits smaller batches so that DataFusion can begin filtering and aggregating before the full partition is loaded. Peak memory per batch is O(batch_size) for coordinate columns diff --git a/xarray_sql/reader.py b/xarray_sql/reader.py index 1460242..93db2d6 100644 --- a/xarray_sql/reader.py +++ b/xarray_sql/reader.py @@ -203,14 +203,15 @@ def read_xarray_table( Each chunk becomes a separate partition, enabling DataFusion's parallel execution across multiple cores. - Filter Pushdown: + Note: SQL queries with WHERE clauses on dimension columns (time, lat, lon, etc.) - automatically prune partitions that can't contain matching rows. For example: + automatically prune partitions that can't contain matching rows — this is + called *filter pushdown*. For example: # This query will skip loading partitions with time < '2020-02-01' - result = ctx.sql('SELECT * FROM air WHERE time > \"2020-02-01\"').collect() + result = ctx.sql('SELECT * FROM air WHERE time > "2020-02-01"').collect() - Supported operators: =, <, >, <=, >=, BETWEEN, IN, AND, OR. + Supported operators: `=`, `<`, `>`, `<=`, `>=`, `BETWEEN`, `IN`, `AND`, `OR`. Args: ds: An xarray Dataset. All data_vars must share the same dimensions.