Distinguish outages from data absence; add failure taxonomy

[13w13]

Summary

This PR introduces a failure taxonomy to distinguish between provider outages (network failures, HTTP 5xx errors) and legitimate data absence (empty results, missing coverage). Previously, all failures were silently caught and returned empty lists, making outages indistinguishable from "no data for this country," which caused the pipeline to report zero records as if they were facts.

Key Changes

• New exception hierarchy in config.py:

  • SourceUnavailable: Provider unreachable or returned server error (outage)
  • MissingCredential: Free API key not configured (skip, not zero)
  • NotCovered: Source has no coverage for this country (absence of coverage, not absence of data)
  • is_outage(): Classifies exceptions as outages vs. bugs (HTTP 5xx/429, timeouts, connection errors vs. 4xx/schema changes)

• Pipeline status tracking:

  • Added status field to fetch results with seven states: ok, empty, partial, unavailable, error, skipped, not_covered
  • _attempt() helper: Runs individual source requests, records failures without swallowing them
  • _state(): Determines final source status from collected errors and record count
  • Failed requests now return None (printed as "n/a") instead of fake zeros

• Output improvements:

  • fetch_summary.csv now includes status column and detailed error notes
  • data_inventory.csv tracks actual files on disk with row counts
  • SOURCE_FILES registry: Defines exact filenames each source produces; previous outputs are cleared before each run to prevent stale data

• Client-side changes:

  • All clients now raise SourceUnavailable, MissingCredential, or NotCovered instead of returning empty lists
  • raise_unavailable() helper for consistent error handling
  • IDMC, ACLED, ACAPS, IFRC GO, WFP, HPC, INFORM, DTM, ReliefWeb updated to use new exceptions
  • get_credential() centralized in config; missing keys raise MissingCredential

• Robustness fixes:

  • sys.stdout.reconfigure() now guarded with hasattr() check (absent in Jupyter, IDLE, captured output)
  • Period filtering: filter_by_period() keeps records with overlapping date ranges, not just start dates
  • HDX CKAN: format parameter renamed to format_filter to avoid shadowing builtin
  • HAPI: app_identifier now properly URL-encoded (base64 contains unsafe characters)
  • ReliefWeb: Appname validation; 403 errors raise MissingCredential with setup instructions
  • Liveuamap: Icon 51 corrected from "election" to "power_infrastructure" (electricity, not elections)

• Test suite enhancements:

  • test_hardening.py: Tests 7–8 verify outage taxonomy and that network failures don't produce fake zeros
  • test_skill_50.py: rec_exc() distinguishes outages (SKIP) from bugs (GAP)
  • test_p0_e2e.py, test_dtm_e2e.py: Updated to use is_outage() for consistent failure classification
  • GitHub Actions workflow added for offline test suite

• Documentation:

  • README updated: ReliefWeb now requires pre-approved appname (free, request at apidoc.reliefweb.int)
  • Exit codes documented: 0 (all sources answered), 1 (partial/unavailable/error), 2 (invalid arguments)
  • SECURITY.md: wgss_probe.py now uses tempfile.mkdtemp() instead of hardcoded path (was relative on Linux/macOS)

Notable Implementation Details

• Period filtering: Checks for overlap between record's [date_start, date_end] and requested [date_from, date_to], not just

https://claude.ai/code/session_01SQ5UN2pMiamMm4HGGqKA7j