Investigate MRMS v0.2.0 nulls and define source-fallback retrieval strategy

[Copilot]

report-nulls on s3://dynamical-noaa-mrms/noaa-mrms-conus-analysis-hourly/v0.2.0.icechunk (from 2023-01-01, precipitation_surface) surfaced recurring missing timestamps.
This PR classifies those nulls by manually checking upstream MRMS source artifacts and identifies which are recoverable via retrieval fallback vs true upstream gaps.

• Scope of issue characterization

  • Confirmed missing timestamps reported by report-nulls for precipitation_surface.
  • Manually sampled 6+ affected times and checked corresponding MRMS source files used by ingestion logic.

• Findings: two distinct null classes

  • Class A: Pass2 missing, alternate source available
    At multiple timestamps, MultiSensor_QPE_01H_Pass2 is absent while Pass1 and/or RadarOnly files exist and are readable.
  • Class B: upstream outage across all products
    Some timestamps are absent in Pass2, Pass1, and RadarOnly; these should remain null by design.

• Proposed ingestion fix

  • Update MRMS retrieval ordering for precipitation_surface:
    1. MultiSensor_QPE_01H_Pass2
    2. fallback MultiSensor_QPE_01H_Pass1
    3. optional fallback RadarOnly_QPE_01H
  • Preserve nulls only when all candidate products are unavailable.

# proposed read priority for precipitation_surface
for product in [
    "MultiSensor_QPE_01H_Pass2_00.00",
    "MultiSensor_QPE_01H_Pass1_00.00",
    "RadarOnly_QPE_01H_00.00",
]:
    path = source_path(product, timestamp)
    if exists(path):
        return read_with_rasterio(path)

return np.nan  # all upstream sources missing

• Repository-change note
  • The MRMS region_job.py target referenced in the issue is not present in this branch snapshot, so this PR captures the validated diagnosis and concrete retrieval fix plan for implementation in the MRMS integration branch.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• mtarchive.geol.iastate.edu
  • Triggering command: /home/REDACTED/work/reformatters/reformatters/.venv/bin/python3 /home/REDACTED/work/reformatters/reformatters/.venv/bin/python3 - (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[aldenks]

implementing in #480