Skip to content

fix(liveuamap): restore resilient optional incident enrichment - #518

Merged
BigBodyCobain merged 16 commits into
mainfrom
fix/liveuamap-resilience-516-517
Aug 19, 2026
Merged

fix(liveuamap): restore resilient optional incident enrichment#518
BigBodyCobain merged 16 commits into
mainfrom
fix/liveuamap-resilience-516-517

Conversation

@BigBodyCobain

@BigBodyCobain BigBodyCobain commented Aug 18, 2026

Copy link
Copy Markdown
Owner

Summary

Rehabilitates LiveUAMap as optional enrichment for Global Incidents without making the paid API a requirement or degrading the GDELT baseline.

Fixes #516.
Fixes #517.

UX contract

  • Global Incidents always remains usable through GDELT. LiveUAMap is never a prerequisite for turning the layer on.
  • Linux/macOS/Docker preserve the existing behavior: the browser provider can enrich Global Incidents unless the operator explicitly disables it.
  • Windows keeps a one-time browser-contact choice, but declining now means GDELT-only, not “Global Incidents stays off.” The decline is remembered instead of nagging on every toggle.
  • A paid/supported LiveUAMap API is optional. If an operator has one, it is preferred; everyone else can continue using the existing best-effort browser path.

#516 — Docker Playwright packaging

Root cause was the browser-cache ownership/location boundary, not a missing Python Playwright dependency: browser installation runs as root, while the container executes as backenduser with /app as HOME.

  • Sets PLAYWRIGHT_BROWSERS_PATH=/ms-playwright before browser installation.
  • Keeps the browser bundle in one image-wide read/execute location.
  • Adds a build-time assertion after USER backenduser that the runtime user resolves an executable Chromium and that the matching chromium_headless_shell-* binary exists and is executable.
  • Does not launch Chromium during the image build, avoiding unnecessary multi-arch/QEMU fragility.

#517 — upstream payload drift

The old parser assumed json.loads(ovens) was always list[dict] and called .get() on every item. The reported current payload reaches the parser as strings, so that assumption crashes every region.

New liveuamap_parser.py separates representation decoding from provider/network logic and defensively supports bounded forms including:

  • direct marker arrays
  • JSON-encoded marker strings / arrays of JSON strings
  • double-encoded JSON
  • keyed marker dictionaries
  • common wrapper objects (data, events, markers, etc.)
  • legacy URL/base64-wrapped JSON
  • GeoJSON FeatureCollections / Point Features

Malformed/scalar entries are skipped, point coordinates are validated/range-checked, parser depth and candidate counts are bounded, and drift logs describe only payload shape rather than dumping raw data.

Provider resilience

  • Supported API provider is preferred when LIVEUAMAP_API_URL is configured.
  • API configuration is generic because endpoint/auth details come from the operator's LiveUAMap agreement; no paid service is required by Shadowbroker.
  • API URL must be HTTPS; header name/scheme are configurable and API secrets are not returned in status responses or logged.
  • API failure falls back to the browser provider only when that provider is independently allowed.
  • Browser imports are lazy, so API-only operation does not need to import the browser stack.
  • Existing Playwright/stealth behavior is retained for compatibility, but this PR does not add or strengthen anti-bot bypass behavior.
  • Obvious access-challenge pages fail soft.
  • One broken region no longer kills the other regions.
  • Three complete browser-provider failures open a bounded in-memory circuit breaker to avoid repeated expensive Chromium attempts.

Docker/ops

docker-compose.yml now actually forwards the documented browser override and the optional API provider settings:

  • SHADOWBROKER_ENABLE_LIVEUAMAP_SCRAPER
  • LIVEUAMAP_API_URL
  • LIVEUAMAP_API_KEY
  • LIVEUAMAP_API_AUTH_HEADER
  • LIVEUAMAP_API_AUTH_SCHEME
  • LIVEUAMAP_API_TIMEOUT_S

Added docs/LIVEUAMAP.md covering provider order, failure semantics, Docker behavior, and the non-blocking UX contract.

Tests / review

Focused coverage now gates CI for:

A pre-PR diff review also caught and removed a superseded regression-test draft that referenced an earlier helper shape, preventing a latent full-suite failure.

Final CI status

Current head 9924079f5331a13f887d4f16d527758418154c48 is green:

  • CI - Lint & Test #615: success
    • backend secret scan, Ruff, imports, and focused LiveUAMap tests passed
    • frontend lint, formatting, Vitest, production build, and bundle report passed
  • Docker Publish #770: success
    • CI gate passed
    • backend Docker builds passed on linux/amd64 and linux/arm64
    • frontend Docker builds passed on linux/amd64 and linux/arm64

The successful backend multi-arch Docker builds exercise the runtime-user Playwright packaging assertion added for #516.

Upstream limitation

I could not capture the reporter's exact current ovens payload from LiveUAMap because automated access from the review environment is currently answered with HTTP 403, and #517 has no payload attachment/comment yet. The fix therefore does not pretend one guessed schema is canonical; it normalizes the reported string case plus several bounded historical/common JSON shapes and fails safely on unknown shapes.

Non-goals

  • No Infonet/store protocol changes.
  • No requirement to buy LiveUAMap API access.
  • No removal of GDELT or other Global Incidents value.
  • No new anti-bot-evasion techniques.

Ready for review.

@BigBodyCobain
BigBodyCobain marked this pull request as ready for review August 18, 2026 23:45
@BigBodyCobain
BigBodyCobain force-pushed the fix/liveuamap-resilience-516-517 branch from b850a7b to 9924079 Compare August 19, 2026 05:50

Copy link
Copy Markdown
Owner Author

Cleaned this PR back to the known-good LiveUAMap-only head 9924079f5331a13f887d4f16d527758418154c48.

The 10 later commits that introduced the unrelated WorldviewLeftPanel/WorldviewLeftPanelCore refactor and associated frontend churn have been removed from the PR branch. The diff is back to the 16 LiveUAMap/ops/frontend-boundary files that passed CI and multi-arch Docker builds previously.

GitHub has re-triggered CI and Docker Publish on the restored head; I am treating those fresh runs as the final merge gate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant