Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: Prune unused operations from sql #1365

Merged
merged 6 commits into from
Feb 6, 2025
Merged

perf: Prune unused operations from sql #1365

merged 6 commits into from
Feb 6, 2025

Conversation

TrevorBergeron
Copy link
Contributor

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

@product-auto-label product-auto-label bot added size: l Pull request size is large. api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. labels Feb 5, 2025
@TrevorBergeron TrevorBergeron force-pushed the reprune branch 3 times, most recently from 99523e5 to 4280f6c Compare February 6, 2025 01:20
@TrevorBergeron TrevorBergeron marked this pull request as ready for review February 6, 2025 07:49
@TrevorBergeron TrevorBergeron requested review from a team as code owners February 6, 2025 07:49
@TrevorBergeron TrevorBergeron requested a review from shobsi February 6, 2025 07:49
@TrevorBergeron TrevorBergeron requested a review from sycai February 6, 2025 07:49
@tswast
Copy link
Collaborator

tswast commented Feb 6, 2025

I'll take a look, but one high-level thought: we should exclude operations from this that might have side effects. Specifically, remote function calls could do all sorts of things (send email, create files in GCS, etc.) without the return value being used.

@TrevorBergeron
Copy link
Contributor Author

I'll take a look, but one high-level thought: we should exclude operations from this that might have side effects. Specifically, remote function calls could do all sorts of things (send email, create files in GCS, etc.) without the return value being used.

There is a lot that needs to be done to properly model intended side-effects. Lazy evaluation can result in side effects being repeated, while filter pushdown (client side or bigquery side) can remove calls. I think side-effects should probably be explicitly indicated through a special api (maybe with no-output consumer udfs?), and we can then model as its own node type.

T = TypeVar("T")


def dedupe(items: Iterable[T]) -> Iterable[T]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function seems to have no caller. Remove it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

@TrevorBergeron TrevorBergeron requested a review from sycai February 6, 2025 22:13
@TrevorBergeron TrevorBergeron merged commit 923da03 into main Feb 6, 2025
23 checks passed
@TrevorBergeron TrevorBergeron deleted the reprune branch February 6, 2025 23:33
shuoweil pushed a commit that referenced this pull request Feb 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. size: l Pull request size is large.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants