Skip to content

RATIS-2721. Notify the StateMachine when a server shutdown starts - #1647

Merged
szetszwo merged 12 commits into
apache:masterfrom
amaliujia:robus_server_and_ratis
Oct 25, 2026
Merged

szetszwo merged 12 commits into
apache:masterfrom
amaliujia:robus_server_and_ratis

Conversation

@amaliujia

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Notify the StateMachine when a server shutdown starts

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-2721

Please replace this section with the link to the Apache JIRA)

How was this patch tested?

Unit test

@amaliujia
amaliujia force-pushed the robus_server_and_ratis branch from 0f1bd8b to b55f83b Compare October 17, 2026 14:10

@szetszwo szetszwo left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amaliujia , thanks for the update!

  • RATIS-2645 added a close(cause) method. We could notify INITIATED there
  • We only need to notify COMPLETED when the close future completes.

See https://issues.apache.org/jira/secure/attachment/13085559/1647_review.patch

Cc @ivandika3

@ivandika3 ivandika3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @amaliujia for the patch. Agreed with @szetszwo patch suggestion to unify the close logic.

@szetszwo

Copy link
Copy Markdown
Contributor

@amaliujia , thanks for the update!

  • RATIS-2645 added a close(cause) method. We could notify INITIATED there
  • We only need to notify COMPLETED when the close future completes.

See https://issues.apache.org/jira/secure/attachment/13085559/1647_review.patch

We only need to notify INITIATED and COMPLETED respectively in close(..) and closeFuture.whenComplete(..). Please take a look at 1647_review.patch

// A division closed before its role is initialized has no role info to report.
if (role.getCurrentRole() != null) {
stateMachine.event().notifyServerShutdown(getRoleInfoProto(), false);
// allServers is null when the close has failed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@szetszwo do you think if we actually need a

if (allServers == null) || e != null) {
   // add a status EventStatus.FAILED and post this status?
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current code in some cases simply call close() but not close(cause) when there is an exception. If we add EventStatus.FAILED, we should also fix these cases.

It seems to me that it does not worth the effort for now. If we really want to do it, we can change it in a separated JIRA.

Comment on lines 622 to 626
if (cause == null) {
closeFuture.complete(null);
closeFuture.complete(allServers);
} else {
closeFuture.completeExceptionally(cause);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amaliujia , For completeExceptionally(..), we will lose the allServers information. How about calling
complete(allServers) even for exceptions? The exception information currently is unused.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True. We in fact does nothing in the whenComplete clause for exceptional cases.

I changed to just call closeFuture.complete(allServers); here but also added a TODO to document that we do not actually handle cause yet.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually reverted this suggestion. Found test failures which indicates that, this closeFuture.completeExceptionally is used by somewhere else: https://github.com/apache/ratis/blob/master/ratis-server/src/main/java/org/apache/ratis/server/impl/RaftServerImpl.java#L486-L495

test failures from this job: https://github.com/apache/ratis/actions/runs/39903654392/job/120250580159?pr=1647

So basically there is other code depending on this closeFuture.completeExceptionally so it cannot be dropped. Then if we do not want to lose allServers here, probably one way is to wrap it into our exception. WDYT?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on the problem!

The closeFuture was newly added by RATIS-2645. For simplicity, let's keep the current change.

@amaliujia
amaliujia force-pushed the robus_server_and_ratis branch from 02dd6a0 to b579c79 Compare October 23, 2026 05:44

@szetszwo szetszwo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amaliujia , thanks for the update!

Please see the comments inlined.

BTW, I see some other bugs

+++ b/ratis-server/src/main/java/org/apache/ratis/server/impl/RaftServerImpl.java
@@ -490,8 +490,7 @@ class RaftServerImpl implements RaftServer.Division,
       closeFuture.get();
     } catch (InterruptedException e) {
       Thread.currentThread().interrupt();
-      LOG.warn("{}: Waiting closing interrupted, will not continue to remove group locally", getMemberId());
-      return;
+      throw new CompletionException(getMemberId() + ": Interrupted groupRemove", e);
     } catch (ExecutionException e) {
       throw new CompletionException(getMemberId() + ": Failed to close this division", e);
     }
@@ -611,7 +610,11 @@ class RaftServerImpl implements RaftServer.Division,
         LOG.warn("{}: Failed to shutdown serverExecutor", getMemberId(), e);
       }
     } catch (RuntimeException | Error e) {
-      failure = e;
+      if (failure == null) {
+        failure = e;
+      } else  {
+        failure.addSuppressed(e);
+      }
       throw e;
     } finally {
       completeClose(failure, allServers);

* @param eventStatus {@link EventStatus#INITIATED} or {@link EventStatus#COMPLETED}
*/
default void notifyServerShutdown(RoleInfoProto roleInfo, boolean allServer) {}
default void notifyServerShutdown(RoleInfoProto roleInfo, boolean allServers, EventStatus eventStatus) {}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually need to implement it in order to preserve the existing behavior:

    default void notifyServerShutdown(RoleInfoProto roleInfo, boolean allServers, EventStatus eventStatus) {
      // preserve the existing behavior
      if (eventStatus == EventStatus.COMPLETED) {
        notifyServerShutdown(roleInfo, allServers);
      }
    }

@amaliujia amaliujia Oct 24, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the user does not implement these functions but happen to call notifyServerShutdown(roleInfo, allServers), it will turn into an infinite loop rather than a harmless no-op.

One approach is we actually throw unsupported exception in the default implementation but that would be a behavior change for the existing API.

As of now I feel like the better way is we do not migrate existing notifyServerShutdown(RoleInfoProto roleInfo, boolean allServer) calls to the EventStatus based calls, which then will preserve existing behavior:

  1. so the complete status report still uses notifyServerShutdown(RoleInfoProto roleInfo, boolean allServer)
  2. only the initialized status report uses default void notifyServerShutdown(RoleInfoProto roleInfo, boolean allServers, EventStatus eventStatus).

I will adopt this way for now.

@amaliujia amaliujia Oct 24, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah actually I guess the complete solution is this:

@Deprecated
default void notifyServerShutdown(RoleInfoProto roleInfo, boolean allServers) {}


default void notifyServerShutdown(RoleInfoProto roleInfo, boolean allServers, EventStatus eventStatus) {
  // preserve the existing behavior
  if (eventStatus == EventStatus.COMPLETED) {
    notifyServerShutdown(roleInfo, allServers);
  }
}

It became a bit confusing to me...

This might not be that perfect though if a user calls notifyServerShutdown(roleInfo, allServers), the user might expect the implementation for notifyServerShutdown(roleInfo, allServers, eventStatus) will be invoked, however it will not. The root cause of the confusion still is from we do not throw unsupported exception but choose to have a default no-op. Anyway the above is the best I can see now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah actually I guess the complete solution is this...

Agree.

... if a user calls notifyServerShutdown(roleInfo, allServers), ...

The notification methods should only be called by Ratis but not the users.

  • BTW, when we add the allServers parameter, we should implement it the other way as below. Let's also fix it here.
    default void notifyServerShutdown(RoleInfoProto roleInfo) {}

    default void notifyServerShutdown(RoleInfoProto roleInfo, boolean allServers) {
      notifyServerShutdown(roleInfo);
    }
  • Let's also change the javadoc: "please use/override" -> "please override". It is more clear.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Done

Comment on lines 622 to 626
if (cause == null) {
closeFuture.complete(null);
closeFuture.complete(allServers);
} else {
closeFuture.completeExceptionally(cause);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on the problem!

The closeFuture was newly added by RATIS-2645. For simplicity, let's keep the current change.

@szetszwo szetszwo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 the change looks good.

@szetszwo
szetszwo merged commit 020fabd into apache:master Oct 25, 2026
33 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants