fix(reorg): break phantom depth feedback loop in reorg models#226
Open
fix(reorg): break phantom depth feedback loop in reorg models#226
Conversation
The existing_depths CTE reads back all distinct depths from the output table, including rows with reorg_count=0. These zero-count rows feed back into the CROSS JOIN, causing every depth that ever existed to propagate to every day/hour forever. Filter existing_depths to only consider rows where reorg_count > 0, breaking the self-reinforcing cycle while still allowing one pass of zero-overwrite when a reorg legitimately disappears from the data.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
existing_depthsCTE in both reorg models reads back all distinct depths from the output table for the current batch's target days, including rows withreorg_count=0. These zero-count rows self-reinforce on every reprocessing of those days, keeping phantom depth entries alive indefinitely.fct_block_proposerhadn't fully populated, so missing slots created false runs of consecutive orphaned slots that were detected as deep reorgs. Once the dependency data settled, the reorgs disappeared but the zero-count rows persisted via the feedback loop.WHERE e.reorg_count > 0toexisting_depthsin bothfct_reorg_dailyandfct_reorg_hourly, breaking the cycle while preserving the one-pass zero-overwrite behavior needed for eventually-consistent data corrections.