Differentiate grad() as a SQL rewrite, dropping the Substrait bridge

Make grad()/jvp()/vjp() work inside any query shape (recursive CTEs, DML,
subqueries) by rewriting the calls as SQL text before planning, rather than
round-tripping the logical plan through Substrait (which could not represent
those shapes). Closes the gap tracked in #197.

XarrayContext.sql() now hands a query containing a marker to the native
rewrite_grad_sql, which parses the statement with sqlparser, differentiates
each marker call with the existing engine, and renders the derivative back
into the SQL in place. Because it runs before planning, every query shape the
parser accepts is supported, and the result is ordinary SQL the stock
datafusion-python context plans and executes directly.

This removes the Substrait round-trip entirely: the datafusion-substrait and
prost dependencies, the grad_rewrite/_sql_with_autograd/_table_schemas plumbing,
the marker-UDF registration, and the protoc steps in CI. Unlike the FFI
alternative, it needs no datafusion fork and no custom datafusion-python wheel.

The grad surface is unchanged (same SQL, same results); marker arguments use
unqualified column names, matching existing usage, since differentiation is
syntactic and runs before binding.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: alxmrs

.github/workflows/ci-build.yml

@@ -31,13 +31,6 @@ jobs:
 
       - uses: dtolnay/rust-toolchain@stable
 
-      # The `substrait` crate (a datafusion-substrait dependency) generates
-      # code from .proto files at build time and requires protoc.
-      - name: Install Protoc
-        uses: arduino/setup-protoc@v3
-        with:
-          repo-token: ${{ secrets.GITHUB_TOKEN }}
-
       - name: Setup sccache
         uses: mozilla-actions/sccache-action@v0.0.9
 

.github/workflows/ci-rust.yml

@@ -27,13 +27,6 @@ jobs:
         with:
           components: clippy
 
-      # The `substrait` crate (a datafusion-substrait dependency) generates
-      # code from .proto files at build time and requires protoc.
-      - name: Install Protoc
-        uses: arduino/setup-protoc@v3
-        with:
-          repo-token: ${{ secrets.GITHUB_TOKEN }}
-
       - name: Setup sccache
         uses: mozilla-actions/sccache-action@v0.0.9
 

.github/workflows/ci.yml

@@ -43,13 +43,6 @@ jobs:
 
       - uses: dtolnay/rust-toolchain@stable
 
-      # The `substrait` crate (a datafusion-substrait dependency) generates
-      # code from .proto files at build time and requires protoc.
-      - name: Install Protoc
-        uses: arduino/setup-protoc@v3
-        with:
-          repo-token: ${{ secrets.GITHUB_TOKEN }}
-
       - name: Setup sccache
         uses: mozilla-actions/sccache-action@v0.0.9
 

.github/workflows/publish.yml

@@ -54,13 +54,6 @@ jobs:
 
       - uses: dtolnay/rust-toolchain@stable
 
-      # The `substrait` crate (a datafusion-substrait dependency) generates
-      # code from .proto files at build time and requires protoc.
-      - name: Install Protoc
-        uses: arduino/setup-protoc@v3
-        with:
-          repo-token: ${{ secrets.GITHUB_TOKEN }}
-
       - name: Setup sccache
         uses: mozilla-actions/sccache-action@v0.0.9
 
@@ -98,18 +91,6 @@ jobs:
           manylinux: 2_28
           rustup-components: rust-std rustfmt
           sccache: 'true'
-          # protoc is required by the substrait crate build and must be
-          # installed inside the manylinux container.
-          before-script-linux: |
-            PROTOC_VERSION=29.3
-            case "$(uname -m)" in
-              x86_64) PROTOC_ARCH=x86_64 ;;
-              aarch64) PROTOC_ARCH=aarch_64 ;;
-              *) echo "unsupported arch $(uname -m)"; exit 1 ;;
-            esac
-            curl -L -o /tmp/protoc.zip "https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-linux-${PROTOC_ARCH}.zip"
-            unzip -o /tmp/protoc.zip -d /usr/local bin/protoc 'include/*'
-            protoc --version
           args: --release --strip --out dist -i python3.10 python3.11 python3.12 python3.13
 
       - uses: actions/upload-artifact@v6
@@ -132,18 +113,6 @@ jobs:
           manylinux: 2_28
           rustup-components: rust-std rustfmt
           sccache: 'true'
-          # protoc is required by the substrait crate build and must be
-          # installed inside the manylinux container.
-          before-script-linux: |
-            PROTOC_VERSION=29.3
-            case "$(uname -m)" in
-              x86_64) PROTOC_ARCH=x86_64 ;;
-              aarch64) PROTOC_ARCH=aarch_64 ;;
-              *) echo "unsupported arch $(uname -m)"; exit 1 ;;
-            esac
-            curl -L -o /tmp/protoc.zip "https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-linux-${PROTOC_ARCH}.zip"
-            unzip -o /tmp/protoc.zip -d /usr/local bin/protoc 'include/*'
-            protoc --version
           args: --release --strip --out dist -i python3.10 python3.11 python3.12 python3.13
 
       - uses: actions/upload-artifact@v6