Repository navigation
RATIS-2721. Notify the StateMachine when a server shutdown starts - #1647
Conversation
0f1bd8b to
b55f83b
Compare
There was a problem hiding this comment.
@amaliujia , thanks for the update!
- RATIS-2645 added a close(cause) method. We could notify INITIATED there
- We only need to notify COMPLETED when the close future completes.
See https://issues.apache.org/jira/secure/attachment/13085559/1647_review.patch
Cc @ivandika3
ivandika3
left a comment
There was a problem hiding this comment.
Thanks @amaliujia for the patch. Agreed with @szetszwo patch suggestion to unify the close logic.
|
@amaliujia , thanks for the update!
We only need to notify INITIATED and COMPLETED respectively in close(..) and closeFuture.whenComplete(..). Please take a look at 1647_review.patch |
| // A division closed before its role is initialized has no role info to report. | ||
| if (role.getCurrentRole() != null) { | ||
| stateMachine.event().notifyServerShutdown(getRoleInfoProto(), false); | ||
| // allServers is null when the close has failed. |
There was a problem hiding this comment.
@szetszwo do you think if we actually need a
if (allServers == null) || e != null) {
// add a status EventStatus.FAILED and post this status?
}
There was a problem hiding this comment.
The current code in some cases simply call close() but not close(cause) when there is an exception. If we add EventStatus.FAILED, we should also fix these cases.
It seems to me that it does not worth the effort for now. If we really want to do it, we can change it in a separated JIRA.
| if (cause == null) { | ||
| closeFuture.complete(null); | ||
| closeFuture.complete(allServers); | ||
| } else { | ||
| closeFuture.completeExceptionally(cause); | ||
| } |
There was a problem hiding this comment.
@amaliujia , For completeExceptionally(..), we will lose the allServers information. How about calling
complete(allServers) even for exceptions? The exception information currently is unused.
There was a problem hiding this comment.
True. We in fact does nothing in the whenComplete clause for exceptional cases.
I changed to just call closeFuture.complete(allServers); here but also added a TODO to document that we do not actually handle cause yet.
There was a problem hiding this comment.
I actually reverted this suggestion. Found test failures which indicates that, this closeFuture.completeExceptionally is used by somewhere else: https://github.com/apache/ratis/blob/master/ratis-server/src/main/java/org/apache/ratis/server/impl/RaftServerImpl.java#L486-L495
test failures from this job: https://github.com/apache/ratis/actions/runs/39903654392/job/120250580159?pr=1647
So basically there is other code depending on this closeFuture.completeExceptionally so it cannot be dropped. Then if we do not want to lose allServers here, probably one way is to wrap it into our exception. WDYT?
There was a problem hiding this comment.
Good catch on the problem!
The closeFuture was newly added by RATIS-2645. For simplicity, let's keep the current change.
02dd6a0 to
b579c79
Compare
szetszwo
left a comment
There was a problem hiding this comment.
@amaliujia , thanks for the update!
Please see the comments inlined.
BTW, I see some other bugs
+++ b/ratis-server/src/main/java/org/apache/ratis/server/impl/RaftServerImpl.java
@@ -490,8 +490,7 @@ class RaftServerImpl implements RaftServer.Division,
closeFuture.get();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
- LOG.warn("{}: Waiting closing interrupted, will not continue to remove group locally", getMemberId());
- return;
+ throw new CompletionException(getMemberId() + ": Interrupted groupRemove", e);
} catch (ExecutionException e) {
throw new CompletionException(getMemberId() + ": Failed to close this division", e);
}
@@ -611,7 +610,11 @@ class RaftServerImpl implements RaftServer.Division,
LOG.warn("{}: Failed to shutdown serverExecutor", getMemberId(), e);
}
} catch (RuntimeException | Error e) {
- failure = e;
+ if (failure == null) {
+ failure = e;
+ } else {
+ failure.addSuppressed(e);
+ }
throw e;
} finally {
completeClose(failure, allServers);| * @param eventStatus {@link EventStatus#INITIATED} or {@link EventStatus#COMPLETED} | ||
| */ | ||
| default void notifyServerShutdown(RoleInfoProto roleInfo, boolean allServer) {} | ||
| default void notifyServerShutdown(RoleInfoProto roleInfo, boolean allServers, EventStatus eventStatus) {} |
There was a problem hiding this comment.
We actually need to implement it in order to preserve the existing behavior:
default void notifyServerShutdown(RoleInfoProto roleInfo, boolean allServers, EventStatus eventStatus) {
// preserve the existing behavior
if (eventStatus == EventStatus.COMPLETED) {
notifyServerShutdown(roleInfo, allServers);
}
}There was a problem hiding this comment.
If the user does not implement these functions but happen to call notifyServerShutdown(roleInfo, allServers), it will turn into an infinite loop rather than a harmless no-op.
One approach is we actually throw unsupported exception in the default implementation but that would be a behavior change for the existing API.
As of now I feel like the better way is we do not migrate existing notifyServerShutdown(RoleInfoProto roleInfo, boolean allServer) calls to the EventStatus based calls, which then will preserve existing behavior:
- so the complete status report still uses
notifyServerShutdown(RoleInfoProto roleInfo, boolean allServer) - only the initialized status report uses
default void notifyServerShutdown(RoleInfoProto roleInfo, boolean allServers, EventStatus eventStatus).
I will adopt this way for now.
There was a problem hiding this comment.
Ah actually I guess the complete solution is this:
@Deprecated
default void notifyServerShutdown(RoleInfoProto roleInfo, boolean allServers) {}
default void notifyServerShutdown(RoleInfoProto roleInfo, boolean allServers, EventStatus eventStatus) {
// preserve the existing behavior
if (eventStatus == EventStatus.COMPLETED) {
notifyServerShutdown(roleInfo, allServers);
}
}
It became a bit confusing to me...
This might not be that perfect though if a user calls notifyServerShutdown(roleInfo, allServers), the user might expect the implementation for notifyServerShutdown(roleInfo, allServers, eventStatus) will be invoked, however it will not. The root cause of the confusion still is from we do not throw unsupported exception but choose to have a default no-op. Anyway the above is the best I can see now.
There was a problem hiding this comment.
Ah actually I guess the complete solution is this...
Agree.
... if a user calls notifyServerShutdown(roleInfo, allServers), ...
The notification methods should only be called by Ratis but not the users.
- BTW, when we add the allServers parameter, we should implement it the other way as below. Let's also fix it here.
default void notifyServerShutdown(RoleInfoProto roleInfo) {}
default void notifyServerShutdown(RoleInfoProto roleInfo, boolean allServers) {
notifyServerShutdown(roleInfo);
}- Let's also change the javadoc: "please use/override" -> "please override". It is more clear.
| if (cause == null) { | ||
| closeFuture.complete(null); | ||
| closeFuture.complete(allServers); | ||
| } else { | ||
| closeFuture.completeExceptionally(cause); | ||
| } |
There was a problem hiding this comment.
Good catch on the problem!
The closeFuture was newly added by RATIS-2645. For simplicity, let's keep the current change.
szetszwo
left a comment
There was a problem hiding this comment.
+1 the change looks good.
What changes were proposed in this pull request?
Notify the StateMachine when a server shutdown starts
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/RATIS-2721
Please replace this section with the link to the Apache JIRA)
How was this patch tested?
Unit test