Skip to content

Commit 39aed08

Browse files
alxmrsclaude
andcommitted
Case 05: one partition each (both chunks=100); document the chunk finding
A 2×2 sweep of (forecasts, era5) time-chunks ∈ {6,100} shows the truth/era5 chunk is what matters — small splits it into many partitions and costs ~3× — while the forecasts chunk is in the noise and a mismatch costs nothing. So set both to one partition (time:100) for these small windows, and document why. Also: cases 07 and 08 now validated end-to-end against real Earth Engine (08's symmetric-lazy regrid matches xarray .interp over 4,320 target cells; 07's PROJ UDF matches EE pixelLonLat). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AWvrZYAT2NbuETBqNAN3o9
1 parent 22d38ef commit 39aed08

1 file changed

Lines changed: 8 additions & 4 deletions

File tree

benchmarks/geospatial/05_forecast_skill.py

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -145,10 +145,14 @@ def main() -> None:
145145
)
146146

147147
ctx = xql.XarrayContext()
148-
# chunks here is the Arrow batch (partition) size the table streams in, not a
149-
# filter — no data is dropped. truth spans only the valid-time window, so
150-
# time:100 makes it a single partition; forecasts stream a few inits at a time.
151-
ctx.from_dataset("forecasts", forecasts, chunks={"time": 6})
148+
# chunks here is the Arrow batch (partition) size each table streams in, not a
149+
# filter — no data is dropped. Both windows are small, so one partition each is
150+
# fastest (fewer partitions = fewer Python→Arrow round-trips for the same
151+
# rows); time:100 covers both the ~40 forecast inits and the ~79 truth steps.
152+
# Empirically the truth chunk is what matters — splitting it small costs ~3×,
153+
# while the forecasts chunk is in the noise — and a chunk *mismatch* costs
154+
# nothing, so there is no need to keep them different.
155+
ctx.from_dataset("forecasts", forecasts, chunks={"time": 100})
152156
ctx.from_dataset("era5", truth, chunks={"time": 100})
153157

154158
sql = """

0 commit comments

Comments
 (0)