Allow expected nan area in MRMS deaccumulation#496
Merged
Conversation
The backfill and operational update test was explicitly filtering out pass1 and pass2 variables, which masked a production issue where shard 153 had all-NaN values for pass_2_surface after the variable was added post-backfill. Including these variables in the test and asserting they have the expected deaccumulation NaN pattern will catch similar gaps. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
… expected_invalid_fraction The numba deaccumulation function was raising ValueError internally when encountering negative values below threshold. For MRMS data, ~6.2% of pixels have -3.0 no-data sentinels (over-ocean areas) that become invalid after deaccumulation, triggering this error on every operational update. The old code caught this with a bare except ValueError that silently skipped writing data, causing pass2 (and likely other vars) to remain all-NaN. Changes: - Move error counting from numba (raise) to Python wrapper (return counts + fraction check) - Add expected_invalid_fraction parameter so callers can declare expected no-data levels - MRMS region_job uses expected_invalid_fraction=0.07 and removes the try/except swallow - Add tests for expected_invalid_fraction suppressing/still-raising errors Co-Authored-By: Claude Opus 4.6 <[email protected]>
…st corrupt files The try/except guards against individual corrupt files breaking the entire operational update. The root cause fix (expected_invalid_fraction=0.07) prevents the -3.0 no-data sentinels from triggering the error under normal conditions. Co-Authored-By: Claude Opus 4.6 <[email protected]>
- Atomic decompression: _decompress_gzip writes to temp file then renames, preventing corruption when precipitation_surface and precipitation_pass_2_surface concurrently decompress the same MultiSensor_QPE_01H_Pass2 file. Also deletes .gz after decompression to avoid stale cache. - Add test_download_and_read_pass_1 and test_download_and_read_pass_2 slow tests for individual file download+read coverage. - Revert backfill integration test to precipitation_surface + categorical only, avoiding the concurrent download collision in test. Co-Authored-By: Claude Opus 4.6 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
About 5% of pass 1 & pass 2 data is expected to be a negative missing value sentinel outside radar coverage. This was triggering our default deaccumulation nan check.
Add argument to set expected allowed nan fraction to deaccumulation.
Also address race condition as both precipitation_surface and pass 2 download, decompress, read and delete the same file.