STOR-5615: Add configurable Durable Object retries - #7383
Open
apeacock1991 wants to merge 4 commits into
Open
apeacock1991 wants to merge 4 commits into
apeacock1991 wants to merge 4 commits into
Conversation
apeacock1991
force-pushed
the
apeacock/STOR-5615-do-retry-policy-config
branch
11 times, most recently
from
September 17, 2026 13:57
7b33183 to
c2a7334
Compare
apeacock1991
force-pushed
the
apeacock/STOR-5615-do-retry-policy-config
branch
from
September 17, 2026 14:28
c2a7334 to
652c1fd
Compare
Cap the randomized backoff between actor call retries at two seconds. Without a cap the doubling window reaches 800ms by the fifth attempt and grows past the retry budget once attempt counts become configurable, which would waste most of a caller's budget waiting. Add fetch tests for redirects through Durable Object stubs. Each redirect hop is already treated as a new actor call with a fresh retry token and first-attempt subrequest accounting, and a redirected streaming body becomes replayable and so retryable. Nothing today asserts any of that. Configurable retry counts and durations change how limits are measured across hops, so the current per-hop behavior needs to be pinned down first.
Add an optional `retryPolicy` group to Durable Object namespace bindings with a `maxAttempts` field, and carry it into minted stubs and jurisdictional subnamespaces as a UserDefinedRetryPolicy. It counts retries after the initial attempt, so zero disables retries. Reject values above ten at config load. Resolve the policy once per fetch into an ActorRetryPolicy. A binding without a retry policy always runs under the runtime's five-attempt default. A binding with one uses it when the new off-by-default DURABLE_OBJECT_RETRIES_USERLAND autogate and both fetch retry gates are on, and falls back to the default otherwise. The gates are checked before the stub's I/O objects are touched so a disabled rollout never reaches them. `maxAttempts` defaults to four so a policy that sets only other fields still yields five attempts in total, and enabling the gate changes nothing for bindings that have not opted in. ActorCallRetryState reads the resolved policy and never asks where it came from. Consolidate the two retries-exhausted exits into one helper that records the outcome and returns the original disconnect.
Add `maxDurationMs` to the Durable Object namespace binding's `retryPolicy`, capped at 60 seconds. When set, and the call can retry at all, it bounds the whole logical call, including backoff and redirects. Whatever is in flight when it passes is interrupted. The caller sees a timeout error, or the original disconnect if an attempt has already failed. When omitted, the runtime's 10-second window still applies. That window only decides whether a retry may start and never interrupts an attempt in flight. ActorCallTimeLimit carries which of the two limits is in force, so ActorCallRetryState reads the kind instead of inferring it from where the policy came from. A call that cannot retry, because the count is zero or the body is not replayable, runs under the default window regardless of any configured duration. Both limits start when the retry state is created, after the output-gate wait. Covering the wait would mean resolving the target stub's policy before the Request exists. Not worth it. The wait is bounded by storage, not by this feature. A redirect carries the original call's start so one deadline spans the chain, while the default window resets per hop. Add TimerChannel::atLimitTimeout() so a deadline can be expressed as an absolute point on the limit-timeout clock.
Forward namespace retry policy through ctx.exports loopback bindings so stubs minted from them retain the binding's retry limits.
apeacock1991
force-pushed
the
apeacock/STOR-5615-do-retry-policy-config
branch
from
September 18, 2026 11:41
652c1fd to
ff80881
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
retryPolicyto Durable Object namespace bindings and carry it into minted stubs and jurisdictional subnamespaces.fetch()calls.DURABLE_OBJECT_RETRIES_USERLANDautogate.Behavior
A retry policy has two fields:
maxAttemptssets the number of retries after the initial attempt. It defaults to four, matching the runtime default of five total attempts, and cannot exceed ten. Setting to 0 disables retries.maxDurationMslimits the call to at most 60 seconds.The binding policy applies only when
DURABLE_OBJECT_RETRIES_USERLANDand both existing fetch-retry gates are enabled. If any gate is disabled, or the binding has no policy, the call uses the runtime defaults.A configured duration starts after the output-gate wait. It covers the initial attempt, backoff, retries, and redirects, and interrupts an attempt still in flight when the deadline passes. It has no effect when the call cannot retry, either because
maxAttemptsis zero or because the request body is not replayable.Without a configured duration, each actor call has the existing 10-second window in which a retry may start. The window does not interrupt an attempt in flight.
A redirect starts a new actor call with a fresh retry allowance, retry token,
isRetry = false, subrequest accounting, and default retry window. A configured duration keeps its original start time across redirects.This PR does not enable the new gate.