-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
T&D Queries should analyze less data where possible by only considering recently emitted rows #28380
Comments
potentially relevant past issue about legacy normalization partition pruning #14070 |
^ tldr - save the results of |
putting this down for now to focus on async standard inserts. there's a loose pr where I sketched out the interface diff (#31191), but I didn't get time to modify/add any test cases for this. |
@edgao I think you started this story perhaps? |
Datawarehouses bill based on how much data is analyzed in the query. If we can add more WHERE clauses to limit which rows are considered, that will help.
See https://github.com/airbytehq/typing-and-deduping-sql/pull/23/files for some ideas of what to do. Also, read this slack thread.
Add a test case for this:
The text was updated successfully, but these errors were encountered: