SWE data for OCONUS, CADWR, ENVCA

[mkarim-rtx]

OCONUS, CADWR, ENVCA
https://jira.nextgenwaterprediction.com/browse/NGWPC-8754

Objective
A replacement workflow for the lost SNODAS SWE processing script was reconstructed by extending Matt’s unfinished swe_gage.py implementation and leveraging the completed SMAP processing workflows (smap_gage.py and smap_data_archiving.py) as reference patterns. The updated workflow now supports processing daily SNODAS NetCDF grids into basin-averaged SWE CSV files for individual gages/catchments using hydrofabric geopackage (.gpkg) polygons.

The implementation:

Reads gages-<gage_id>.gpkg files and extracts the divides layer geometry for each basin.
Opens daily SNODAS NetCDF SWE grids from S3.
Computes area-weighted basin-average SWE values using polygon/grid overlays.

Generates per-gage CSV outputs such as: gages-02236500_swe.csv containing: timestamp, basin_avg_swe
Adds CSV validation utilities to detect:
missing files
unsorted timestamps
non-daily records
duplicate timestamps
sparse or corrupted SWE data
Adds manifest/report generation for tracking missing or failed gages.
Adds helper scripts to locate/download required gage geopackage (.gpkg) files from local directories or S3.

Current findings/issues:

Several CONUS SWE CSVs appear incomplete or corrupted, with records only every 4–8 days and timestamps not sorted.
The original SNODAS NetCDF-to-CSV extraction script written by Kyle is unavailable, so this workflow serves as a reconstructed replacement.
ENVCA gages cannot currently be processed because the available SNODAS NetCDF datasets on S3 are clipped to CONUS and do not include Canadian coverage.
CADWR gages should be processable once missing/corrupted CSVs are regenerated.
The workflow depends on hydrofabric geopackage files (gages-<gage_id>.gpkg), which are currently missing locally for several target gages and must be fetched/copied from existing hydrofabric datasets or S3 storage.

Remaining work / next steps:

Locate or download missing hydrofabric .gpkg files for target gages.
Re-run SNODAS basin SWE extraction for affected CONUS and CADWR gages.
Validate regenerated CSV outputs for complete daily coverage and sorted timestamps.
Re-download and reprocess unclipped SNODAS datasets for ENVCA coverage if Canadian SWE support is required.

NHF PROCESSING

https://jira.nextgenwaterprediction.com/browse/NGWPC-10760

The SWE reprocessing workflow was updated to support the NextGen Hydrofabric (NHF) instead of the older hydrofabric datasets. The implementation reconstructs and extends the previously incomplete SNODAS SWE aggregation workflow by integrating NHF geopackage retrieval directly from the Icefabric API and reprocessing basin-average SWE outputs for NHF gages.

The new workflow:

Retrieves NHF gage geopackages dynamically from the Icefabric API using source=nhf and id_type=gage_id.
Supports CONUS gage processing using the USGS_gages.txt list.
Downloads daily SNODAS NetCDF SWE datasets from S3.
Aggregates SWE values over NHF catchment polygons using area-weighted basin averaging.

Generates per-gage SWE CSV outputs such as: gages-<gage_id>_swe.csv
Adds validation and manifest reporting to verify:

• successful geopackage retrieval
• valid SWE CSV generation
• missing/corrupted outputs
• zero-SWE/no-snow basins
• missing SNODAS overlap

Additional utilities were implemented to:

• validate SWE CSV completeness and timestamp ordering
• support smoke tests and large-scale processing
• classify valid zero-snow basins separately from failed processing

Testing successfully demonstrated:

NHF API geopackage retrieval for multiple CONUS gages
successful SNODAS NetCDF access from S3
successful SWE aggregation and CSV generation for 50 NHF CONUS gages
correct validation classification (valid vs valid_zero_snow)

Issues encountered during development included:

missing hydrofabric geopackages locally
incorrect SNODAS S3 path handling
NetCDF backend dependency (h5netcdf) issues
validation reports incorrectly flagging unprocessed gages as missing during limited smoke tests

These issues were resolved by:

dynamically downloading NHF geopackages from the Icefabric API
improving SNODAS path handling and S3 download logic
adding fallback NetCDF loading logic
improving validation/reporting behavior

Current status:

CONUS NHF SWE processing workflow is functioning successfully in smoke tests.
Alaska and Hawaii processing are pending availability of NHF gage lists and/or domain-specific SNODAS verification.
Full historical reprocessing and S3 publication remain the next execution step for final production generation.