When writing to R2 via DuckDB's COPY, the parquet file skips all custom metadata that local storage includes:
- bloom filter (fast rejection for full-hash queries)
- source content hashes (dedup with `--force`)
- algorithm/source/record-count metadata (fast `info` without scanning)
This means R2-stored databases:
- always do a full scan for queries that could be rejected instantly
- can't detect already-processed sources
- `shaha info --r2` has to scan every row for stats instead of reading metadata
The root cause is that DuckDB's `COPY ... TO` writes its own parquet, while local storage uses ArrowWriter where we can inject key-value metadata. Fixing this likely means writing to a temp local parquet first (with all metadata), then uploading to R2 — or finding a way to set parquet KV metadata through DuckDB.
When writing to R2 via DuckDB's COPY, the parquet file skips all custom metadata that local storage includes:
This means R2-stored databases:
The root cause is that DuckDB's `COPY ... TO` writes its own parquet, while local storage uses ArrowWriter where we can inject key-value metadata. Fixing this likely means writing to a temp local parquet first (with all metadata), then uploading to R2 — or finding a way to set parquet KV metadata through DuckDB.