Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid relying on barrier_interval_ms to increase operator cache eviction watermark #19403

Closed
hzxa21 opened this issue Nov 15, 2024 · 3 comments · Fixed by #19494
Closed

Avoid relying on barrier_interval_ms to increase operator cache eviction watermark #19403

hzxa21 opened this issue Nov 15, 2024 · 3 comments · Fixed by #19494
Assignees
Milestone

Comments

@hzxa21
Copy link
Collaborator

hzxa21 commented Nov 15, 2024

After #16087, we have switched to using sequence-based memory eviction policy instead of an epoch-based policy. This means we no longer need to rely on barrier to increase the watermark. However, it seems that we are still use barrier_interval_ms as the ticker interval here. This may result in delay in cache eviction when barrier_interval_ms is increased.

@github-actions github-actions bot added this to the release-2.2 milestone Nov 15, 2024
@yuhao-su
Copy link
Contributor

This is a long existing issue. We should first make sure all executors are not evicting cache on barrier basis.

@hzxa21
Copy link
Collaborator Author

hzxa21 commented Nov 18, 2024

This is a long existing issue. We should first make sure all executors are not evicting cache on barrier basis.

I think we already did? Just checked hash join and hash agg.

@MrCroxx
Copy link
Contributor

MrCroxx commented Nov 20, 2024

This is a long existing issue. We should first make sure all executors are not evicting cache on barrier basis.

It had been done before the sequence-based operator cache eviction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants