Skip to content

Commit df67d98

Browse files
alxmrsclaude
andcommitted
README: add "Does it work?" section (addresses #46); drop "honest"
- Move the geospatial-suite pointer out of "What is this?" into a new "Does it work?" section, placed between "How does it work?" and "Why does this work?", reframed to answer that question with a findings summary (validated operations, real datasets, the GraphCast-beats-Pangu result). Addresses #46. - Reword the two remaining uses of "honest" in docs/geospatial.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AWvrZYAT2NbuETBqNAN3o9
1 parent 3bd17a7 commit df67d98

2 files changed

Lines changed: 32 additions & 11 deletions

File tree

README.md

Lines changed: 27 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -108,12 +108,6 @@ _(A runnable version of this example lives at
108108
Succinctly, we "pivot" Xarray Datasets to treat them like tables so we can run
109109
SQL queries against them.
110110

111-
For worked examples of common geospatial operations expressed in SQL — NDVI,
112-
climatology, anomalies, zonal statistics, forecast skill (Pangu-Weather and
113-
GraphCast vs ERA5), reprojection, and regridding, each checked against an xarray
114-
reference — see [`benchmarks/geospatial/`](benchmarks/geospatial/) and the
115-
[Geospatial operations are relational operations](docs/geospatial.md) write-up.
116-
117111
## Why build this?
118112

119113
A few reasons:
@@ -148,6 +142,33 @@ that lets the DB engine translate the underlying Dataset arrays into DataFusion
148142
Ultimately, the initial insight of the `pivot()` function -- that any ndarray can be
149143
translated into a 2D table -- underlies this performant query mechanism.
150144

145+
## Does it work?
146+
147+
Yes. The recurring worry is that the SQL interface is a toy — fine for `SELECT`s,
148+
but not for the operations geoscience actually runs. So we wrote a suite that
149+
takes the staples of geospatial and climate analysis — the ones we assume *need*
150+
an array library — and expresses each one in SQL, then **checks the SQL answer
151+
against an xarray/array reference** to floating-point tolerance:
152+
153+
* **Spectral indices** (NDVI) — column arithmetic over a real Sentinel-2 scene.
154+
* **Climatology, anomalies, zonal means**`GROUP BY` and self-`JOIN` over the
155+
full 0.25° **ARCO-ERA5** archive (≈1.3M hourly steps), read lazily with
156+
partition pruning and column pushdown.
157+
* **Forecast skill** — scoring the **Pangu-Weather** and **GraphCast** ML models
158+
against ERA5 (WeatherBench 2) as a `JOIN` on `valid_time = init + lead`; it
159+
reproduces the published result that GraphCast beats Pangu at every lead.
160+
* **Raster × vector zonal stats** — a range `JOIN` of the ERA5 grid against a
161+
table of regions.
162+
* **Reprojection and regridding** — a scalar PROJ UDF (validated against Earth
163+
Engine's own geodesy via [Xee](https://github.com/google/Xee)) and a
164+
sparse-weight-table `JOIN` (regridding real SRTM terrain).
165+
166+
Every case matches its array reference. The headline finding: these operations
167+
are not really "array" operations at all — they are `GROUP BY`, `JOIN`, window
168+
functions, and `CASE` in disguise, and a query engine runs them at scale. See
169+
[`benchmarks/geospatial/`](benchmarks/geospatial/) and the write-up,
170+
[Geospatial operations are relational operations](docs/geospatial.md).
171+
151172
## Why does this work?
152173

153174
Underneath Xarray, Dask, and Pandas, there are NumPy arrays. These are paged in

docs/geospatial.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -192,10 +192,10 @@ this against **Earth Engine itself**: it opens a UTM grid through
192192
[Xee](https://github.com/google/Xee) carrying `ee.Image.pixelLonLat()`, so EE's
193193
own geodesy engine reports the true lon/lat of every pixel — an *independent*
194194
reprojection reference, not PROJ-vs-PROJ. The SQL UDF and EE agree to sub-metre
195-
precision. Two honest caveats, both documented in the script: PROJ's context is
196-
not thread-safe (so the UDF returns both coordinates from *one* call and runs on
197-
a single partition), and reprojection moves coordinates without resampling onto
198-
a grid — which is the next operation.
195+
precision. Two caveats, both documented in the script: PROJ's context is not
196+
thread-safe (so the UDF returns both coordinates from *one* call and runs on a
197+
single partition), and reprojection moves coordinates without resampling onto a
198+
grid — which is the next operation.
199199

200200
**Regridding is not** row-independent: each output cell is a weighted blend of
201201
several input cells. That is a *many-to-many* relationship — and a many-to-many
@@ -216,7 +216,7 @@ it is the most relational operation of all.
216216

217217
## Where the array paradigm still earns its keep
218218

219-
The honest boundary is **weight generation**. Applying a regridding is a join;
219+
The boundary is **weight generation**. Applying a regridding is a join;
220220
*computing* the weights — cell overlaps for conservative remapping, stencils and
221221
spherical geometry for bilinear, the whole machinery of xESMF/ESMF — is genuinely
222222
geometric work that arrays (and specialized libraries) do well. The relational

0 commit comments

Comments
 (0)