Skip to content

ref(kafka): Remove incremental rebalancing #101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 16, 2024
Merged

Conversation

john-z-yang
Copy link
Member

@john-z-yang john-z-yang commented Dec 16, 2024

Overview

Removes incremental rebalancing logic from the kafka consumer state machine.
We currently have a working implementation here. However, taking advantage of incremental rebalancing requires a little further work:

  • The rdkafka library currently has a bug with committing consumer offset manually during incremental rebalance, causing the consumer to lose its assignment when encountering an illegal generation error. We generally don't use manual offset commit and utilize the auto commit from rdkafka, but we still manually commit offset on shutdown signals. The way the PR above works around this issue is to shutdown the actors, poll the consumer for a predetermined amount of time before actually shutting down. But to use this in production we should wait for the tracking issue to be resolved and see what course of action the rdkfaka team decides to take before trying again.
  • We currently re-create all actors even on partition assignment, we need to just increase actors without shutting down to actually reduce the time spent rebalancing. (This should be a relatively straightforward change).

@john-z-yang john-z-yang force-pushed the john/remove-coop-rebl branch from 27656b8 to fa789d6 Compare December 16, 2024 21:39
@john-z-yang john-z-yang marked this pull request as ready for review December 16, 2024 22:07
@john-z-yang john-z-yang requested a review from a team December 16, 2024 22:07
@john-z-yang john-z-yang merged commit e8cf0f7 into main Dec 16, 2024
6 checks passed
@john-z-yang john-z-yang deleted the john/remove-coop-rebl branch December 16, 2024 23:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants