Download pre-built DuckDB libraries (#628)

How this works:

When `DUCKDB_DOWNLOAD_LIB` is set, we map `CARGO_TARGET_DIR` to
`target/duckdb-download/<target>/<version>`, download the matching
DuckDB release archive, unzip it, and add the directory to the linker
search path.

We also copy the downloaded library into `target/<profile>/deps` so test
binaries can load it without extra env vars, matching the
`DUCKDB_LIB_DIR` flow.

This will make building much easier - no toolchain to compile DuckDB, or
manual downloads and fiddling with `DUCKDB_LIB_DIR`, etc.

Inspired by

-
https://github.com/vortex-data/vortex/blob/develop/vortex-duckdb/build.rs
-
https://github.com/nbigaouette/onnxruntime-rs/blob/master/onnxruntime-sys/build.rs

Author: mlafeldt

.github/workflows/rust.yaml

@@ -14,6 +14,9 @@ concurrency:
 
 jobs:
   test:
+    # - Linux uses DUCKDB_DOWNLOAD_LIB (non-bundled)
+    # - Windows uses pre-downloaded archives via DUCKDB_LIB_DIR
+    # - bundled feature tested by --all-features
     name: Test ${{ matrix.target }}
     strategy:
       fail-fast: true
@@ -30,38 +33,28 @@ jobs:
               duckdb: libduckdb-linux-amd64.zip,
             }
     runs-on: ${{ matrix.os }}
+    env:
+      DUCKDB_DOWNLOAD_LIB: "1"
     steps:
       - uses: actions/checkout@v2
       - uses: actions-rust-lang/setup-rust-toolchain@v1
         with:
           target: ${{ matrix.target }}
 
-      # download libduckdb
       - uses: robinraju/[email protected]
         name: Download duckdb
+        if: matrix.os == 'windows-latest'
         with:
           repository: "duckdb/duckdb"
           tag: "v1.4.2"
           fileName: ${{ matrix.duckdb }}
           out-file-path: .
 
-      # For Linux
-      - name: Linux extract duckdb
-        if: matrix.os == 'ubuntu-latest'
-        uses: ihiroky/extract-action@v1
-        with:
-          file_path: ${{ github.workspace }}/${{ matrix.duckdb }}
-          extract_dir: libduckdb
-
       - run: cargo fmt --all -- --check
         if: matrix.os == 'ubuntu-latest'
 
       - name: run cargo clippy
         if: matrix.os == 'ubuntu-latest'
-        env:
-          DUCKDB_LIB_DIR: ${{ github.workspace }}/libduckdb
-          DUCKDB_INCLUDE_DIR: ${{ github.workspace }}/libduckdb
-          LD_LIBRARY_PATH: ${{ github.workspace }}/libduckdb
         run: cargo clippy --all-targets --all-features --locked -- -D warnings
 
       - name: Dry-run release of crates
@@ -91,13 +84,23 @@ jobs:
           DUCKDB_INCLUDE_DIR: ${{ github.workspace }}/libduckdb
 
       - name: Build loadable extension
+        if: matrix.os == 'ubuntu-latest'
+        run: cargo build --example hello-ext --features="vtab-loadable"
+
+      - name: Build loadable extension (C API)
+        if: matrix.os == 'ubuntu-latest'
+        run: cargo build --example hello-ext-capi --features="vtab-loadable loadable-extension"
+
+      - name: Build loadable extension (windows)
+        if: matrix.os == 'windows-latest'
         run: cargo build --example hello-ext --features="vtab-loadable"
         env:
           DUCKDB_LIB_DIR: ${{ github.workspace }}/libduckdb
           DUCKDB_INCLUDE_DIR: ${{ github.workspace }}/libduckdb
           LD_LIBRARY_PATH: ${{ github.workspace }}/libduckdb
 
-      - name: Build loadable extension
+      - name: Build loadable extension C API (windows)
+        if: matrix.os == 'windows-latest'
         run: cargo build --example hello-ext-capi --features="vtab-loadable loadable-extension"
         env:
           DUCKDB_LIB_DIR: ${{ github.workspace }}/libduckdb

Cargo.lock

@@ -204,12 +204,20 @@ You can adjust this behavior in a number of ways:
    cargo build --examples
    ```
 
-3. Installing the duckdb development packages will usually be all that is required, but
+3. _Experimental:_ Setting `DUCKDB_DOWNLOAD_LIB=1` makes the build script download pre-built DuckDB binaries from GitHub Releases. This always links against the dynamic library in the archive (setting `DUCKDB_STATIC` has no effect), and it effectively automates the manual steps above. The archives are cached in `target/duckdb-download/<target>/<version>` and that directory is automatically added to the linker search path. The downloaded version always matches the `libduckdb-sys` crate version.
+
+   ```shell
+   DUCKDB_DOWNLOAD_LIB=1 cargo test
+   ```
+
+4. Installing the duckdb development packages will usually be all that is required, but
    the build helpers for [pkg-config](https://github.com/alexcrichton/pkg-config-rs)
    and [vcpkg](https://github.com/mcgoo/vcpkg-rs) have some additional configuration
    options. The default when using vcpkg is to dynamically link,
    which must be enabled by setting `VCPKGRS_DYNAMIC=1` environment variable before build.
 
+When none of the options above are used, the build script falls back to this discovery path and will emit the appropriate `cargo:rustc-link-lib` directives if DuckDB is found on your system.
+
 ### ICU extension and the bundled feature
 
 When using the `bundled` feature, the ICU extension is not included due to crates.io's 10MB package size limit. This means some date/time operations (like `now() - interval '1 day'` or `ts::date` casts) will fail. You can load ICU at runtime:

README.md

@@ -32,11 +32,16 @@ flate2 = { workspace = true }
 pkg-config = { workspace = true, optional = true }
 prettyplease = { workspace = true, optional = true }
 quote = { workspace = true, optional = true }
+reqwest = { version = "0.12", default-features = false, features = [
+    "blocking",
+    "rustls-tls",
+] }
 serde = { workspace = true, features = ["derive"] }
 serde_json = { workspace = true }
 syn = { workspace = true, optional = true }
 tar = { workspace = true }
 vcpkg = { workspace = true, optional = true }
+zip = { version = "6", default-features = false, features = ["deflate"] }
 
 [dev-dependencies]
 arrow = { workspace = true, features = ["ffi"] }