Skip to content

Conversation

LZD-PratyushBhatt
Copy link

@LZD-PratyushBhatt LZD-PratyushBhatt commented Aug 11, 2025

Issues

  • My PR addresses the following Helix issues and references them in the PR description:

(#200 - Link your issue number here: You can write "Fixes #XXX". Please use the proper keyword so that the issue gets closed automatically. See https://docs.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue
Any of the following keywords can be used: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved)
This PR fixes #3063 : Legacy IZkStateListenerI0ItecImpl causes zombie participants by discarding session IDs

Description

  • Here are some details about my PR, including screenshots of any UI changes:

(Write a concise description including what, why, how)
Helix participants can enter a "zombie" state where they appear healthy in ZooKeeper but are functionally disconnected from cluster operations, unable to process state transition messages. This occurs due to a critical flaw in the legacy compatibility wrapper IZkStateListenerI0ItecImpl that completely ignores session ID parameters when processing ZooKeeper session events.
When the single-threaded ZkEventThread becomes blocked (e.g., during long-running FINALIZE operations), multiple SyncConnected events accumulate in the event queue. Once processing resumes, these events are handled in FIFO order, but the legacy wrapper discards the original session ID from each queued event and falls back to using the current active session ID via getSessionId(). This causes session events originally queued for older sessions to be processed using the context of the most recent session, violating the intended event processing timeline.
The result is catastrophic: the first misprocessed event successfully creates a LiveInstance for the current session, but all subsequent events in the backlog attempt to create LiveInstances for the same current session and fail with "already has a live-instance" exceptions. Each failed attempt partially resets the participant's message handlers (setting _ready = false) but never completes the re-initialization process, leaving the participant in a broken state where it cannot process any state transition messages despite appearing active to the Helix controller.

The fix we are proposing is to remove the legacy deprecated listeners altogether.

Tests

  • The following tests are written for this issue:

(List the names of added unit/integration tests)
New test file: TestStaleSessionEvents.java

  • The following is the result of the "mvn test" command on the appropriate module:

(If CI test fails due to known issue, please specify the issue and test PR locally. Then copy & paste the result of "mvn test" to here.)

Changes that Break Backward Compatibility (Optional)

  • My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:

(Consider including all behavior changes for public methods or API. Also include these changes in merge description so that other developers are aware of these changes. This allows them to make relevant code changes in feature branches accounting for the new method/API behavior.)

Documentation (Optional)

  • In case of new functionality, my PR adds documentation in the following wiki page:

(Link the GitHub wiki you added)

Commits

  • My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Code Quality

  • My diff has been formatted using helix-style.xml
    (helix-style-intellij.xml if IntelliJ IDE is used)

@LZD-PratyushBhatt LZD-PratyushBhatt marked this pull request as draft August 11, 2025 11:57
@LZD-PratyushBhatt LZD-PratyushBhatt marked this pull request as ready for review August 21, 2025 07:20
@LZD-PratyushBhatt
Copy link
Author

Hey @junkaixue @xyuanlu can you review this one?
This PR fixes #3063 : Legacy IZkStateListenerI0ItecImpl causes zombie participants by discarding session IDs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Legacy IZkStateListenerI0ItecImpl causes zombie participants by discarding session IDs
1 participant