Refine MRMS validators and reprocess window for pass_2 fix#495
Merged
Conversation
The validators were incorrectly failing due to: 1. precipitation_radar_only_surface has ~34% structural NaN from radar coverage gaps, exceeding the 15% threshold. The old random 2x2 spatial sampling was also unstable on this large geographic domain. 2. precipitation_pass_2_surface is 100% NaN right after an update due to very high source data latency, exceeding the 50% threshold. Changes: - Use quarter spatial sampling for stable estimates on the 3500×7000 grid - precipitation_surface only in the non-lagged NaN check (max 35%): worst-case quarter is ~30% NaN when the most recent timestamp falls back to radar-only; radar_only's structural NaN is captured indirectly here - pass_1 + categorical only in lagged NaN check (max 50%): pass_1 worst-case quarter is ~46% NaN (most recent timestamp unavailable); pass_2 excluded as it has very high latency and is 100% NaN after updates https://claude.ai/code/session_01JuCMa8C61QUGxsr8KLgXQk
- Add precipitation_radar_only_surface to non-lagged NaN check (35% threshold covers ~34% structural coverage gaps) - Add precipitation_pass_2_surface to lagged NaN check alongside pass_1 (50% threshold handles gauge-collection latency) https://claude.ai/code/session_01JuCMa8C61QUGxsr8KLgXQk
Validators: - Move categorical_precipitation_type_surface to non-lagged check (PrecipFlag is radar-derived with no gauge latency) - Tighten lagged check threshold from 50% to 40% (worst-case with gauge latency is ~38%: 1 of 3 timestamps at 100% NaN, rest at ~6%) Operational update: - Start reprocessing 3 hours before dataset max time so precipitation_surface gets rewritten with pass_2 data once it becomes available, overwriting any radar-only fallback written by the previous update run https://claude.ai/code/session_01JuCMa8C61QUGxsr8KLgXQk
Define gauge_latency_vars once and use it as exclude_vars in the non-lagged check, so new variables are automatically covered by the tighter threshold without needing to update both validators. https://claude.ai/code/session_01JuCMa8C61QUGxsr8KLgXQk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.