-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf: Prune unused operations from sql #1365
Conversation
99523e5
to
4280f6c
Compare
4280f6c
to
e01d2ce
Compare
I'll take a look, but one high-level thought: we should exclude operations from this that might have side effects. Specifically, remote function calls could do all sorts of things (send email, create files in GCS, etc.) without the return value being used. |
There is a lot that needs to be done to properly model intended side-effects. Lazy evaluation can result in side effects being repeated, while filter pushdown (client side or bigquery side) can remove calls. I think side-effects should probably be explicitly indicated through a special api (maybe with no-output consumer udfs?), and we can then model as its own node type. |
bigframes/core/rewrite/pruning.py
Outdated
T = TypeVar("T") | ||
|
||
|
||
def dedupe(items: Iterable[T]) -> Iterable[T]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This function seems to have no caller. Remove it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
Fixes #<issue_number_goes_here> 🦕