Repository navigation
RATIS-2814. Fix replyPendingRequest() ignoring the replacement RetryCache entry - #1695
Conversation
amaliujia
left a comment
There was a problem hiding this comment.
Nice catch on the bug! Just have comments for the testing.
| cluster.start(); | ||
| waitForLeader(cluster); | ||
| // Give peer connections time to recover from connection failures during startup. | ||
| TimeDuration.valueOf(3, TimeUnit.SECONDS).sleep(); |
There was a problem hiding this comment.
Do we really need for this 3 second wait? What kind of connection failures it could happen during/after leader election?
There was a problem hiding this comment.
org.apache.ratis.grpc.TestLeaderElectionWithGrpc-output.txt
I reproduced this locally without the sleep and enabled gRPC debug logging. During cluster startup, s0’s connections to s1 and s2 were refused. Both outbound channels entered TRANSIENT_FAILURE, with reconnects scheduled about 949 ms and 1032 ms later.
s1 was elected leader, and the test immediately requested a transfer to s0. About 600 ms into the backoff, s0 tried to request votes from both peers. Both RPCs failed with UNAVAILABLE, and s1 was elected again in term 3. The sleep gave those channels time to reconnect. With the simpler test you suggested, we can remove it.
| } | ||
|
|
||
| @Test | ||
| public void testRetryCacheCompletesAfterLeaderTransfer() throws Exception { |
There was a problem hiding this comment.
I tried to read this test and failed to understand what does it do :) For a simple fix, this test seems to be too long.
So I asked AI and AI gives a short version of test:
In RetryCacheTestUtil.java, we can add
public static RetryCache.Entry createFailedEntry(RaftServer.Division server, ClientId clientId, long callId) {
final RetryCacheImpl.CacheEntry entry = ((RetryCacheImpl) server.getRetryCache())
.getOrCreateEntry(ClientInvocationId.valueOf(clientId, callId));
entry.failWithException(new IllegalStateException("Injected failure"));
return entry;
}In RetryCacheTests.java, we can have
@Test
public void testApplyCompletesReplacementOfFailedEntry() throws Exception {
runWithNewCluster(NUM_SERVERS, this::runTestApplyCompletesReplacementOfFailedEntry);
}
void runTestApplyCompletesReplacementOfFailedEntry(CLUSTER cluster) throws Exception {
final RaftPeerId leaderId = RaftTestUtil.waitForLeader(cluster).getId();
final RaftServer.Division follower = cluster.getFollowers().get(0);
try (final RaftClient client = cluster.createClient(leaderId)) {
final long callId = 999;
final RetryCache.Entry failed = RetryCacheTestUtil.createFailedEntry(follower, client.getId(), callId);
final RaftClientRequest r = cluster.newRaftClientRequest(client.getId(), leaderId,
callId, new SimpleMessage("message"));
assertReply(client.getClientRpc().sendRequest(r), client, callId);
JavaUtils.attempt(() -> {
final RetryCache.Entry entry = RetryCacheTestUtil.get(follower, client.getId(), callId);
Assertions.assertNotSame(failed, entry);
final RaftClientReply reply = entry.getReplyFuture().getNow(null);
Assertions.assertNotNull(reply, "The replacement entry is not completed");
Assertions.assertTrue(reply.isSuccess(), reply::toString);
}, 50, TimeDuration.valueOf(100, TimeUnit.MILLISECONDS), "wait for the follower to apply", LOG);
}
}There was a problem hiding this comment.
Thanks for the suggestion. The original test used a leadership transfer to leave a failed entry on the former leader, then retried the same write on the new leader and checked that the former leader completed the replacement entry when it applied the write.
I agree that creating the failed entry directly on a follower makes the test easier to follow. I’ll update the test to do that and keep the check that the replacement entry receives a successful reply.
|
Thanks @amaliujia for reviewing and merging! |
What changes were proposed in this pull request?
When applying a committed write,
RaftServerImpl#replyPendingRequest()replaces a failed RetryCache entry with a new pending entry. However, it ignores the entry returned byrefreshEntry(), and the reply callback still uses the original failed entry.refreshEntry().testRetryCacheCompletesAfterLeaderTransferto verify that the former leader completes the replacement cache entry after applying a write retried on the new leader.What is the link to the Apache JIRA
RATIS-2814
How was this patch tested?
Tested locally.
TestLeaderElectionWithGrpc