Skip to content

fix(coordinator): retry storage failures and prepare grant redaction - #2847

Open
vincentkoc wants to merge 4 commits into
mainfrom
fix/coordinator-storage-reader-rollout-20261015
Open

vincentkoc wants to merge 4 commits into
mainfrom
fix/coordinator-storage-reader-rollout-20261015

Conversation

@vincentkoc

@vincentkoc vincentkoc commented Oct 15, 2026 •

Copy link
Copy Markdown
Member

Stage A: reader rollout. This PR includes R and A, ending at 7edfe614be987daf6bfb59082fa721a1d5b234cc. Deploy this stage and verify its production SHA/version and Fleet Durable Object convergence before stage B. Never deploy B directly from the pre-migration base. The separate writer PR is #2848.

What Problem This Solves

Fixes missing retry signals when transient coordinator storage errors are caught internally, and prepares a safe migration for an encrypted GitHub grant header visible in Cloudflare invocation logs.

User Impact

Transient Cloudflare storage failures return a retryable response without replaying a possibly committed operation. Provider error responses and Node runtime errors retain their existing behavior. Pairing and bridge authentication accept both header names while stage A still writes only the legacy header.

The observed header contained encrypted grant ciphertext. This is not a claim that plaintext credentials were logged. Redaction is completed only by the separately gated stage B deployment.

Why This Change Was Made

The inner Fleet catch converted storage failures to HTTP 500 before the Worker boundary could classify them. The Cloudflare runtime now records failures at actual storage operations, preserving each original exception. Transaction callback errors are not misclassified as storage failures. This matters because provider errors also use retryability flags.

A shared grant reader accepts both names and prefers the legacy value, preventing a caller-supplied new-name value forwarded by an old Worker from shadowing the authoritative grant. Stage A strips that new-name input and retains the legacy-only writer.

Commit sequence:

  • R: 658dba4b1f3ef3a774aff1b0d86e8c460f996dad — storage reliability repair.
  • Initial A: b92b0a4c67b33d04bf212ecccc4e3ad399ff8857 — compatible readers, legacy-only writer.
  • Storage provenance: 4b643154614c34f6dab664490eac7057591570c2 — preserve provider errors while classifying native storage failures.
  • Final A: 7edfe614be987daf6bfb59082fa721a1d5b234cc — refresh the existing AWS RDS trust-bundle checksum required by the Node container build.
  • B: fix(coordinator): redact grants after reader rollout #2848 — separately qualified writer migration after A production proof.

Production gate: deploy the squash-merged A revision and record its source SHA and version at 100%. The Worker receipt alone does not prove Fleet Durable Object convergence; verify an actual Fleet invocation runs the A version. Do not advance to B without that evidence. Preserve compatible rollback targets. See docs/operations.md for the sequence.

Evidence

At 4b643154614c34f6dab664490eac7057591570c2, whose runtime and tests are unchanged by the certificate-pin commit: the complete Worker suite passed 4,473 tests, with 15 skipped (90 test files passed, two skipped). Worker and Node TypeScript checks, touched lint/format checks and git diff --check passed.

Coverage includes both grant header forms through both actual consumers, authoritative legacy precedence, forged-header stripping, storage provenance through transactions and alarms, preservation of error identity, no operation replay, provider errors with the same flags, and unchanged Node error behavior. The full suite also preserves provider cleanup/cancellation and uncertain-commit recovery assertions.

The Node container build exposed an outdated pin for AWS's official RDS bundle. Two HTTPS downloads matched SHA-256 90a6f1a7978a1b1b096dca6c57511ca48d3b8cf052f8e704baf358f34ea2f21b; 114 unique Amazon RDS CA certificates parsed and were unexpired. Only the checksum, its matching test and changelog changed. Seven Node/Fargate contract tests passed. PostgreSQL-only trust scope and checksum enforcement remain intact.

Independent source review is required at the final SHA; hosted checks must pass before merge. Remaining live proof: the merged A deployment receipt and actual Fleet Durable Object convergence. No deployment has been performed for this change.

@vincentkoc
vincentkoc force-pushed the fix/coordinator-storage-reader-rollout-20261015 branch from fcf8120 to 4b64315 Compare October 15, 2026 03:17
@vincentkoc
vincentkoc marked this pull request as ready for review October 15, 2026 03:18
@clawsweeper

clawsweeper Bot commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper reviews are opt-in for maintainers. If a review is required, finish the work and validation first, then comment @clawsweeper review. Please avoid repeated requests while a review is queued or running; agents should not request reviews automatically.

@vincentkoc
vincentkoc force-pushed the fix/coordinator-storage-reader-rollout-20261015 branch from a74093e to 7edfe61 Compare October 15, 2026 03:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant