Skip to content

Commit df857f8

Browse files
BYKGPT-6 Sol
BYK
and
GPT-6 Sol
authored
Test the exact production Worker version before promotion (#1399)
## Summary - Upload one production Worker version, stage it at 0% beside the current 100% version, and smoke-test it through the production route using the Cloudflare version override and version metadata before promoting that same version. - Verify traffic changes against the captured live version and run marker. Restore only the captured version while the latest deployment remains owned by this run; refuse ambiguous or intervening state. - Add protected, serialized manual recovery for a stopped runner. It checks the requested run attempt and contiguous run-owned deployment history, waits for propagation, then checks the restored Worker. Older Workers without the version route still receive functional smoke tests and a Cloudflare active-version check. - Keep production JUnit publication after the automatic recovery boundary, and update the token-migration removal gate to require all successful upload, stage, smoke, promotion, and ownership steps in the new flow. This replaces #1395. It adds no journal branch, writer App, HMAC key, or repository variable. Production code deployments still require a successful push-triggered `Test` run on current `main`. ## Validation - `pnpm run tsc`, `pnpm run lint`, `pnpm run test`: passed on the final tree (CLI 10,255 passed/16 skipped; MCP core 1,743/6; MCP server 83; test client 80; Cloudflare 431). An earlier run had one CLI ZIP-test timeout; the test passed in isolation and on the full rerun. - `pnpm run test:ci-projects`: 27 passed. The new workflow and migration regressions failed before the fix and passed afterward. - `pnpm run docs:check` and `pnpm run pre-commit:generated`: passed. - Version override and zero-percent staging checked against Cloudflare documentation and Wrangler 4.80.0 commands. --------- Co-authored-by: GPT-6 Sol <agent@openai.com>
1 parent 9f1a7b2 commit df857f8

12 files changed

Lines changed: 816 additions & 104 deletions

File tree

‎.github/workflows/deploy.yml‎

Lines changed: 73 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -71,81 +71,109 @@ jobs:
7171
- name: Install dependencies
7272
run: pnpm install --frozen-lockfile
7373

74-
# === BUILD AND DEPLOY CANARY WORKER ===
74+
# Upload one production Worker version. A version includes code, assets,
75+
# bindings and compatibility settings; the upload does not move traffic.
7576
- name: Build
7677
run: pnpm --filter '@sentry/mcp-cloudflare...' run build
7778
env:
7879
SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
7980
VITE_SENTRY_DSN: ${{ secrets.VITE_SENTRY_DSN }}
8081
VITE_SENTRY_ENVIRONMENT: production
8182

82-
- name: Deploy to Canary Worker
83-
id: deploy_canary
83+
- name: Capture active production version
84+
env:
85+
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
86+
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
87+
TESTED_SHA: ${{ github.event.workflow_run.head_sha }}
88+
run: node scripts/cloudflare-deployment.mjs capture
89+
90+
- name: Upload production version without moving traffic
91+
id: upload
8492
working-directory: packages/mcp-cloudflare
8593
env:
8694
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
8795
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
88-
run: pnpm exec wrangler deploy --config wrangler.canary.jsonc
96+
TESTED_SHA: ${{ github.event.workflow_run.head_sha }}
97+
WRANGLER_OUTPUT_FILE_PATH: ${{ runner.temp }}/wrangler-upload.jsonl
98+
run: |
99+
pnpm exec wrangler versions upload --experimental-auto-create=false \
100+
--config wrangler.jsonc \
101+
--message "toolkit-mcp:$GITHUB_RUN_ID:$GITHUB_RUN_ATTEMPT:$TESTED_SHA:upload"
102+
103+
- name: Identify uploaded production version
104+
id: uploaded
105+
env:
106+
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
107+
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
108+
TESTED_SHA: ${{ github.event.workflow_run.head_sha }}
109+
WRANGLER_OUTPUT_FILE_PATH: ${{ runner.temp }}/wrangler-upload.jsonl
110+
run: node scripts/cloudflare-deployment.mjs uploaded
89111

90-
- name: Wait for Canary to Propagate
91-
if: success()
112+
- name: Require tested revision on main before staging
113+
env:
114+
GH_TOKEN: ${{ github.token }}
115+
EXPECTED_SHA: ${{ github.event.workflow_run.head_sha }}
92116
run: |
93-
echo "Waiting 30 seconds for canary deployment to propagate..."
94-
sleep 30
117+
current_sha="$(gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/main" --jq '.object.sha')"
118+
if [[ "$current_sha" != "$EXPECTED_SHA" ]]; then
119+
echo 'Main advanced before candidate staging; refusing deployment.' >&2
120+
exit 1
121+
fi
95122
96-
# === SMOKE TEST CANARY ===
97-
- name: Run Smoke Tests on Canary
98-
id: canary_smoke_tests
99-
if: success()
123+
- name: Stage exact candidate at zero percent
124+
id: stage
100125
env:
101-
PREVIEW_URL: https://sentry-mcp-canary.getsentry.workers.dev
126+
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
127+
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
128+
TESTED_SHA: ${{ github.event.workflow_run.head_sha }}
129+
run: node scripts/cloudflare-deployment.mjs stage
130+
131+
- name: Wait for staged candidate to propagate
132+
run: sleep 30
133+
134+
- name: Smoke test exact production Worker version
135+
id: candidate_smoke_tests
136+
env:
137+
PREVIEW_URL: https://mcp.sentry.dev
138+
CLOUDFLARE_WORKER_NAME: sentry-mcp
139+
CLOUDFLARE_VERSION_OVERRIDE: ${{ steps.uploaded.outputs.candidate_version }}
140+
EXPECTED_VERSION_ID: ${{ steps.uploaded.outputs.candidate_version }}
102141
run: |
103-
echo "Running smoke tests against canary worker..."
104142
cd packages/smoke-tests
105143
pnpm test:ci
106144
107-
- name: Publish Canary Smoke Test Report
145+
- name: Publish Candidate Smoke Test Report
108146
uses: mikepenz/action-junit-report@cf701569b05ccdd861a76b8607a66d76f6fd4857
109-
if: always() && steps.canary_smoke_tests.outcome != 'skipped'
147+
if: always() && steps.candidate_smoke_tests.outcome != 'skipped'
110148
with:
111149
report_paths: "packages/smoke-tests/tests.junit.xml"
112-
check_name: "Canary Smoke Test Results"
150+
check_name: "Candidate Smoke Test Results"
113151
fail_on_failure: false
114152

115-
# === DEPLOY PRODUCTION WORKER (only if canary tests pass) ===
116-
- name: Require tested revision on main before production
117-
if: steps.canary_smoke_tests.outcome == 'success'
153+
- name: Require tested revision on main before promotion
154+
if: steps.candidate_smoke_tests.outcome == 'success'
118155
env:
119156
GH_TOKEN: ${{ github.token }}
120157
EXPECTED_SHA: ${{ github.event.workflow_run.head_sha }}
121158
run: |
122159
current_sha="$(gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/main" --jq '.object.sha')"
123160
if [[ "$current_sha" != "$EXPECTED_SHA" ]]; then
124-
echo 'Main advanced during canary testing; refusing production deployment.' >&2
161+
echo 'Main advanced during candidate testing; refusing promotion.' >&2
125162
exit 1
126163
fi
127164
128-
- name: Capture active production version
129-
if: steps.canary_smoke_tests.outcome == 'success'
165+
- name: Promote tested version to production
166+
id: promote
167+
if: steps.candidate_smoke_tests.outcome == 'success'
130168
env:
131169
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
132170
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
133171
TESTED_SHA: ${{ github.event.workflow_run.head_sha }}
134-
run: node scripts/cloudflare-deployment.mjs capture
135-
136-
- name: Deploy to Production Worker
137-
id: deploy_production
138-
if: steps.canary_smoke_tests.outcome == 'success'
139-
working-directory: packages/mcp-cloudflare
140-
env:
141-
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
142-
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
143-
TESTED_SHA: ${{ github.event.workflow_run.head_sha }}
144-
run: pnpm exec wrangler deploy --message "toolkit-mcp:$GITHUB_RUN_ID:$GITHUB_RUN_ATTEMPT:$TESTED_SHA"
172+
run: node scripts/cloudflare-deployment.mjs promote
145173

146174
- name: Verify production deployment ownership
147175
id: verify_production
148-
if: steps.deploy_production.outcome == 'success'
176+
if: steps.promote.outcome == 'success'
149177
env:
150178
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
151179
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
@@ -164,29 +192,25 @@ jobs:
164192
if: steps.verify_production.outcome == 'success'
165193
env:
166194
PREVIEW_URL: https://mcp.sentry.dev
195+
EXPECTED_VERSION_ID: ${{ steps.uploaded.outputs.candidate_version }}
167196
run: |
168197
echo "Running smoke tests on production..."
169198
cd packages/smoke-tests
170199
pnpm test:ci
171200
201+
- name: Recover captured previous version if owned transition fails
202+
if: failure() && steps.uploaded.outcome == 'success' && steps.stage.outcome != 'skipped'
203+
env:
204+
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
205+
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
206+
TESTED_SHA: ${{ github.event.workflow_run.head_sha }}
207+
run: node scripts/cloudflare-deployment.mjs recover
208+
209+
# Report publication must not turn a verified promotion into a rollback.
172210
- name: Publish Production Smoke Test Report
173211
uses: mikepenz/action-junit-report@cf701569b05ccdd861a76b8607a66d76f6fd4857
174212
if: always() && steps.production_smoke_tests.outcome != 'skipped'
175213
with:
176214
report_paths: "packages/smoke-tests/tests.junit.xml"
177215
check_name: "Production Smoke Test Results"
178216
fail_on_failure: false
179-
180-
- name: Recover captured previous version after smoke failure
181-
if: failure() && steps.production_smoke_tests.outcome == 'failure' && steps.verify_production.outcome == 'success'
182-
env:
183-
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
184-
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
185-
TESTED_SHA: ${{ github.event.workflow_run.head_sha }}
186-
run: node scripts/cloudflare-deployment.mjs recover
187-
188-
- name: Fail Job if Production Smoke Tests Failed
189-
if: failure() && steps.production_smoke_tests.outcome == 'failure'
190-
run: |
191-
echo 'Production smoke tests failed. Inspect the deployment before changing traffic.' >&2
192-
exit 1

‎.github/workflows/migrate-cloudflare-token.yml‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -90,9 +90,14 @@ jobs:
9090
.status == "completed" and
9191
.conclusion == "success" and
9292
(. as $job | [
93-
"Deploy to Canary Worker",
94-
"Run Smoke Tests on Canary",
95-
"Deploy to Production Worker",
93+
"Capture active production version",
94+
"Upload production version without moving traffic",
95+
"Identify uploaded production version",
96+
"Require tested revision on main before staging",
97+
"Stage exact candidate at zero percent",
98+
"Smoke test exact production Worker version",
99+
"Require tested revision on main before promotion",
100+
"Promote tested version to production",
96101
"Verify production deployment ownership",
97102
"Run Smoke Tests on Production"
98103
] | all(.[]; . as $name | ($job.steps | map(select(.name == $name and .status == "completed" and .conclusion == "success")) | length) == 1))

‎.github/workflows/recover-cloudflare-deployment.yml‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
name: Recover Cloudflare Deployment
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
run_id:
7+
description: Deploy to Cloudflare run ID to recover
8+
required: true
9+
type: string
10+
run_attempt:
11+
description: Attempt of that deployment run
12+
required: true
13+
type: string
14+
15+
permissions:
16+
actions: read
17+
contents: read
18+
deployments: write
19+
20+
concurrency:
21+
group: mcp-production-deploy
22+
cancel-in-progress: false
23+
24+
jobs:
25+
recover:
26+
name: Recover exact owned Worker version
27+
if: ${{ github.ref == 'refs/heads/main' && github.event.repository.id == 957245447 }}
28+
runs-on: ubuntu-latest
29+
environment: production
30+
steps:
31+
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
32+
with:
33+
persist-credentials: false
34+
35+
- name: Require current main and validate source run
36+
env:
37+
GH_TOKEN: ${{ github.token }}
38+
SOURCE_RUN_ID: ${{ inputs.run_id }}
39+
SOURCE_RUN_ATTEMPT: ${{ inputs.run_attempt }}
40+
EXPECTED_SHA: ${{ github.sha }}
41+
run: |
42+
set -euo pipefail
43+
if [[ ! "$SOURCE_RUN_ID" =~ ^[1-9][0-9]*$ || ! "$SOURCE_RUN_ATTEMPT" =~ ^[1-9][0-9]*$ ]]; then
44+
echo 'Invalid deployment run identity' >&2
45+
exit 1
46+
fi
47+
current_sha="$(gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/main" --jq '.object.sha')"
48+
if [[ "$current_sha" != "$EXPECTED_SHA" ]]; then
49+
echo 'Main advanced before manual recovery; retry from current main.' >&2
50+
exit 1
51+
fi
52+
gh api "repos/$GITHUB_REPOSITORY/actions/runs/$SOURCE_RUN_ID/attempts/$SOURCE_RUN_ATTEMPT" \
53+
--jq '{id,run_attempt,head_sha,head_branch,event,status,path,repository_id:.repository.id}' \
54+
> "$RUNNER_TEMP/source-run.json"
55+
if ! jq -e --argjson id "$SOURCE_RUN_ID" --argjson attempt "$SOURCE_RUN_ATTEMPT" \
56+
--argjson repository "$GITHUB_REPOSITORY_ID" \
57+
'.id == $id and .run_attempt == $attempt and .repository_id == $repository and
58+
.event == "workflow_run" and .head_branch == "main" and .status == "completed" and
59+
.path == ".github/workflows/deploy.yml" and
60+
(.head_sha | test("^[0-9a-f]{40}$"))' \
61+
"$RUNNER_TEMP/source-run.json" > /dev/null; then
62+
echo 'Source run is not a completed Toolkit main deployment.' >&2
63+
exit 1
64+
fi
65+
jq -r '.head_sha | "SOURCE_SHA=\(.)"' "$RUNNER_TEMP/source-run.json" >> "$GITHUB_ENV"
66+
67+
- name: Setup Node.js
68+
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
69+
with:
70+
node-version: "22"
71+
72+
- uses: pnpm/action-setup@a7487c7e89a18df4991f7f222e4898a00d66ddda # v4
73+
with:
74+
run_install: false
75+
76+
- name: Install dependencies
77+
run: pnpm install --frozen-lockfile
78+
79+
- name: Restore only the last owned deployment
80+
id: restore
81+
env:
82+
GH_TOKEN: ${{ github.token }}
83+
EXPECTED_SHA: ${{ github.sha }}
84+
SOURCE_RUN_ID: ${{ inputs.run_id }}
85+
SOURCE_RUN_ATTEMPT: ${{ inputs.run_attempt }}
86+
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
87+
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
88+
run: |
89+
current_sha="$(gh api "repos/$GITHUB_REPOSITORY/git/ref/heads/main" --jq '.object.sha')"
90+
if [[ "$current_sha" != "$EXPECTED_SHA" ]]; then
91+
echo 'Main advanced before manual recovery; retry from current main.' >&2
92+
exit 1
93+
fi
94+
node scripts/cloudflare-deployment.mjs manual
95+
96+
- name: Wait for restored version to propagate
97+
run: sleep 30
98+
99+
- name: Verify recovered production Worker
100+
env:
101+
PREVIEW_URL: https://mcp.sentry.dev
102+
EXPECTED_VERSION_ID: ${{ steps.restore.outputs.previous_version }}
103+
ALLOW_LEGACY_VERSION_ENDPOINT: "1"
104+
run: pnpm --dir packages/smoke-tests test:ci

‎docs/operations/github-actions.md‎

Lines changed: 31 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -19,12 +19,28 @@ Package-specific exceptions live in `package.json#sentryCi`; the standalone
1919
smoke-test suite remains in its own workflow.
2020

2121
### deploy.yml
22-
Runs after a successful `Test` push run on `main`. Checks out the tested commit,
23-
requires that it is still the tip of `main`, then deploys and tests canary.
24-
Records the active production version before changing traffic, deploys the
25-
tested commit, and verifies the run-owned candidate before production smoke
26-
tests. If those fail, restores the captured version only while this run's
27-
candidate remains active. External changes stop recovery.
22+
Runs after a successful `Test` push run on `main`. Checks out the tested commit
23+
and requires that it is still the tip of `main`. Builds once, records the active
24+
production version, and uploads one new version of `sentry-mcp`. It stages the
25+
candidate at 0% alongside the old version at 100%, then runs smoke tests
26+
through the production route with a version override on every request. The
27+
version endpoint confirms the override reaches the candidate. Only then does
28+
it promotes the tested version to 100% and repeats the smoke tests. On failure
29+
it restores the exact captured prior version only if the live deployments still
30+
belong to this run.
31+
32+
### recover-cloudflare-deployment.yml
33+
Manual `workflow_dispatch` recovery accepts a deployment run ID and attempt.
34+
It runs trusted current-`main` code in the protected `production` environment,
35+
checks the completed source run, and derives the prior version from contiguous
36+
run-owned Cloudflare deployment history. It refuses an intervening deployment,
37+
split or ambiguous traffic, or a marker mismatch. A successful restore is
38+
verified against Cloudflare and the live Worker. Recovery never builds or
39+
deploys source code from the old run. The first restored version may predate
40+
`/_health/version`; in that case, Cloudflare's active-version check and the
41+
functional smoke tests verify recovery while the version-route test accepts
42+
only its 404 response. When the route exists, the smoke test compares its
43+
version ID with the restored version.
2844

2945
### migrate-cloudflare-token.yml
3046
Moves the Cloudflare API token from a repository secret into the protected
@@ -77,10 +93,11 @@ Other configuration:
7793

7894
### Workers
7995
- **`sentry-mcp`** - Production worker at `https://mcp.sentry.dev`
80-
- **`sentry-mcp-canary`** - Canary worker at `https://sentry-mcp-canary.getsentry.workers.dev`
96+
- The candidate is tested on the production Worker at 0% traffic before promotion.
8197

8298
### Resource Isolation
83-
Canary and production use separate resources for complete isolation:
99+
The existing canary Worker has separate resources; exact-version rollout does
100+
not deploy it. The production candidate uses the production bindings:
84101

85102
| Resource | Production | Canary |
86103
|----------|------------|---------|
@@ -96,9 +113,12 @@ confirm the restored version fails the job rather than guessing a recovery.
96113

97114
## Manual Deployment
98115

99-
Manual production dispatch is unavailable. Use a reviewed change and its
100-
passing `Test` run to deploy. Never run bare `wrangler rollback` against
101-
production; that command chooses from mutable history.
116+
Manual dispatch cannot deploy a new revision. Use a reviewed change and its
117+
passing `Test` run to deploy. To restore an owned deployment after its runner
118+
has stopped, dispatch `Recover Cloudflare Deployment` from `main` with the
119+
failed deployment run ID and attempt. Verify the active Cloudflare version and
120+
live smoke-test result; if ownership has changed, investigate rather than
121+
retrying against mutable history. Never run bare `wrangler rollback`.
102122

103123
## Cloudflare token migration
104124

0 commit comments

Comments
 (0)