fix: prevent OOM from dense custom AOIs and rework sub-AOI handling - #273
Merged
Conversation
A dense AOI (e.g. an Indonesia-scale asset, ~2.45M edges) was materialised client-side, growing the solara process to ~2.5GB and OOM-killing it on the 4GB SEPAL box. Root cause: importing/selecting an AOI pulled the full FeatureCollection geometry into Python (get_info -> GeoDataFrame -> __geo_interface__ -> ee.serializer). Geometry handling - aoi_geometry: add `simplify_fc` (per-feature server-side simplification, no dissolve) and `fc_from_source` (rebuild the EXACT geometry server-side from a small descriptor: asset id / admin code). Nothing dense reaches the client. - import/admin AOI: async via create_task; pull only a SIMPLIFIED outline for display, store a `source` descriptor on each feature for analysis + tiles. - Decouple display from analysis: `get_ee_features` rebuilds the exact geometry from `source` (asset/admin) so dashboard stats stay exact while display stays cheap. Custom sub-AOI rendering - Draw the exact outline as a single merged EE tile layer (FeatureCollection.style); no client-side GeoJSON layer. The simplified geometry collapsed small features into Point/Line/GeometryCollection parts that ipyleaflet rendered as stray markers and that broke hover. - Hover name label is driven from the cursor against a cached geometry, so it no longer depends on a transparent GeoJSON hit-area. - Containment check uses the exact geometry (via `source`) and a relative tolerance (1% of area) instead of a fixed 1 m^2 — independent datasets never align perfectly, which produced false "outside" verdicts. UX - Remove the "Vector file" (SHAPE) upload option from the main AOI and the import dialog; users supply geometry as a GEE asset. - Zoom to a freshly added sub-AOI (async, exact bounds). - Add an eye/zoom action to the custom-geometries table (zoom + close). Diagnostics - Add opt-in memory diagnostics (RSS / tracemalloc / session counts), DISABLED by default; enable with SEPLAN_MEM_DIAG=1. Requires pysepal>=3.6.2 (the column dropdown now reads propertyNames instead of pulling the whole first feature with its geometry).
Async AOI tasks could apply results after the user cancelled or made a newer
selection. Add a monotonic generation token per path (captured at task start,
re-checked before any state mutation) and carry per-selection data in the task
result instead of reading mutable self.* fields on completion; cancel the prior
task as a courtesy.
- admin_aoi_dialog: _resolve_admin_async now takes/returns code+text+gen;
_on_resolved rejects a stale gen and reads code/text from the result (was
reading self._admin_code on completion -> could pair selection A's geometry
with selection B's source). _cancel bumps the token + cancels.
- custom_aoi_dialog: _validate_async returns {gen, outside_count};
_on_validate_done / _on_validate_error reject a stale gen (no commit/alert
after cancel or a superseding save). on_cancel bumps + cancels.
- import_aoi_dialog: _build_async takes a gen and skips the on_new_geom tail when
superseded (the tail has no await, so cancel() alone can't stop it); dialog
_cancel/open_dialog invalidate via cancel_import().
- map: outline-tile refresh is single-flight — bump token + cancel prior before
any layer mutation (incl. the empty/delete-all remove), drop the pre-add
remove (add_layer self-removes), and add a delete-all guard so a late getMapId
can't resurrect deleted outlines.
zoom_to_custom is left as-is (last-writer-wins viewport, captures its features,
no resurrection/clobber).
Unit-tests the sync completion handlers (no GEE): a result whose generation token was superseded by a cancel or newer selection is dropped — admin _on_resolved does not forward a geometry, custom-geometry _on_validate_done does not commit, and a stale _on_validate_error does not reset the UI. Also asserts the fresh-token paths still act, and that admin reads source/name from the result (not mutable self.*).
_on_map_interaction runs on the shared kernel for every mousemove; hoist the hardcoded 0.1s into _HOVER_THROTTLE_S and raise to 0.2s (~5 Hz) to halve the per-user hover work ahead of multi-user load. Label stays responsive.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A dense AOI (e.g. an Indonesia-scale asset, ~2.45M edges) was materialised
client-side, growing the
solaraprocess to ~2.5 GB and OOM-killing it on the4 GB SEPAL box. Root cause: importing/selecting an AOI pulled the full
FeatureCollection geometry into Python (
get_info→ GeoDataFrame →__geo_interface__→ee.serializer).Changes
Geometry handling
aoi_geometry:simplify_fc(per-feature server-side simplification, no dissolve) andfc_from_source(rebuild the exact geometry server-side from a small descriptor — asset id / admin code). Nothing dense reaches the client.create_task; pull only a simplified outline for display and store asourcedescriptor per feature.get_ee_featuresrebuilds the exact geometry fromsource, so dashboard stats stay exact while display stays cheap.Custom sub-AOI rendering
FeatureCollection.style). No client-side GeoJSON layer — the simplified geometry collapsed small features into Point/Line/GeometryCollection parts that ipyleaflet drew as stray markers and that broke hover.source) and a relative tolerance (1% of area) instead of a fixed 1 m² — independent datasets never align perfectly, which produced false "outside" verdicts.UX
Diagnostics
SEPLAN_MEM_DIAG=1(e.g. via.env).Dependency
Requires
pysepal>=3.6.2(separate PR) — the column dropdown now readspropertyNames()instead of pulling the whole first feature with its geometry.pyproject.tomlpin bumped accordingly.Testing
ruff+blackclean;pytest tests/test_aoi_*/test_custom_aoi_*pass.fc_from_sourcerebuilds exact geometry (full asset 312 feats / filtered 1 / draw fallback 1); containment now passes WestSumatra_SF (0.0053% outside ≪ 1%); reproduced the marker cause (simplify produces Point/MultiPoint/LineString/GeometryCollection).