Skip to content

RATIS-2814. Fix replyPendingRequest() ignoring the replacement RetryCache entry - #1695

Merged
amaliujia merged 2 commits into
apache:masterfrom
lugency:RATIS-2814
Nov 18, 2026
Merged

amaliujia merged 2 commits into
apache:masterfrom
lugency:RATIS-2814

Conversation

@lugency

@lugency lugency commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

When applying a committed write, RaftServerImpl#replyPendingRequest() replaces a failed RetryCache entry with a new pending entry. However, it ignores the entry returned by refreshEntry(), and the reply callback still uses the original failed entry.

  • Complete the reply future of the replacement entry returned by refreshEntry().
  • Add testRetryCacheCompletesAfterLeaderTransfer to verify that the former leader completes the replacement cache entry after applying a write retried on the new leader.

What is the link to the Apache JIRA

RATIS-2814

How was this patch tested?

Tested locally.

  • TestLeaderElectionWithGrpc

@amaliujia amaliujia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch on the bug! Just have comments for the testing.

cluster.start();
waitForLeader(cluster);
// Give peer connections time to recover from connection failures during startup.
TimeDuration.valueOf(3, TimeUnit.SECONDS).sleep();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need for this 3 second wait? What kind of connection failures it could happen during/after leader election?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

org.apache.ratis.grpc.TestLeaderElectionWithGrpc-output.txt
I reproduced this locally without the sleep and enabled gRPC debug logging. During cluster startup, s0’s connections to s1 and s2 were refused. Both outbound channels entered TRANSIENT_FAILURE, with reconnects scheduled about 949 ms and 1032 ms later.

s1 was elected leader, and the test immediately requested a transfer to s0. About 600 ms into the backoff, s0 tried to request votes from both peers. Both RPCs failed with UNAVAILABLE, and s1 was elected again in term 3. The sleep gave those channels time to reconnect. With the simpler test you suggested, we can remove it.

}

@Test
public void testRetryCacheCompletesAfterLeaderTransfer() throws Exception {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to read this test and failed to understand what does it do :) For a simple fix, this test seems to be too long.

So I asked AI and AI gives a short version of test:

In RetryCacheTestUtil.java, we can add

  public static RetryCache.Entry createFailedEntry(RaftServer.Division server, ClientId clientId, long callId) {
    final RetryCacheImpl.CacheEntry entry = ((RetryCacheImpl) server.getRetryCache())
        .getOrCreateEntry(ClientInvocationId.valueOf(clientId, callId));
    entry.failWithException(new IllegalStateException("Injected failure"));
    return entry;
  }

In RetryCacheTests.java, we can have

  @Test
  public void testApplyCompletesReplacementOfFailedEntry() throws Exception {
    runWithNewCluster(NUM_SERVERS, this::runTestApplyCompletesReplacementOfFailedEntry);
  }

  void runTestApplyCompletesReplacementOfFailedEntry(CLUSTER cluster) throws Exception {
    final RaftPeerId leaderId = RaftTestUtil.waitForLeader(cluster).getId();
    final RaftServer.Division follower = cluster.getFollowers().get(0);

    try (final RaftClient client = cluster.createClient(leaderId)) {
      final long callId = 999;
      final RetryCache.Entry failed = RetryCacheTestUtil.createFailedEntry(follower, client.getId(), callId);

      final RaftClientRequest r = cluster.newRaftClientRequest(client.getId(), leaderId,
          callId, new SimpleMessage("message"));
      assertReply(client.getClientRpc().sendRequest(r), client, callId);

      JavaUtils.attempt(() -> {
        final RetryCache.Entry entry = RetryCacheTestUtil.get(follower, client.getId(), callId);
        Assertions.assertNotSame(failed, entry);
        final RaftClientReply reply = entry.getReplyFuture().getNow(null);
        Assertions.assertNotNull(reply, "The replacement entry is not completed");
        Assertions.assertTrue(reply.isSuccess(), reply::toString);
      }, 50, TimeDuration.valueOf(100, TimeUnit.MILLISECONDS), "wait for the follower to apply", LOG);
    }
  }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. The original test used a leadership transfer to leave a failed entry on the former leader, then retried the same write on the new leader and checked that the former leader completed the replacement entry when it applied the write.

I agree that creating the failed entry directly on a follower makes the test easier to follow. I’ll update the test to do that and keep the check that the replacement entry receives a successful reply.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, thanks!

@amaliujia
amaliujia merged commit 341c724 into apache:master Nov 18, 2026
17 checks passed
@lugency

lugency commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @amaliujia for reviewing and merging!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants