Skip to content

Commit ebcfec3

Browse files
authored
fix(ci): count only automation-dispatched runs toward a workflow outage (#10266)
`escalate-workflow-outage.ts` counted every non-success at the head of a workflow's run history, so a maintainer retrying a publish by hand escalated as a standing outage. That is what #10171 was: six consecutive publish-miner.yml failures, every one a hand `gh workflow run` against main failing ETARGET on a @loopover/contract version that was not published yet. The next run after contract landed succeeded with no code change. Filtering on the run's `event` / `triggering_actor` / `head_branch` cannot work here, and the header records why so it is not tried again. The reconcile path in mcp-release-please.yml dispatches a bare `gh workflow run "$workflow"` under a PAT, so its runs land as `workflow_dispatch` / `main` / the PAT owner -- the identical triple a laptop produces. Verified against live history: publish-miner run #484, which the reconcile job's own log shows it dispatched, is indistinguishable on every one of those fields from the six manual #10171 failures, and `GET /actions/runs/:id` exposes no `inputs` key either. So provenance is stamped at dispatch instead. `run-name:` is rendered into `display_title`, which the runs API does return -- the same mechanism visual-capture-fallback.yml already uses to correlate a dispatch to its PR. The five publish workflows take a `dispatched_by_automation` input and stamp a marker into their run name; both dispatch sites pass it; the streak counts only runs carrying it. Manual runs are excluded rather than merely not resetting the streak: a run nobody automated is not evidence about the automated path in either direction. Non-dispatch triggers still count exactly as before, so selfhost.yml -- the other caller, and push-triggered -- is unchanged. Runs already in the history carry no stamp and so read as unattributable. The publish escalation therefore stays quiet until three stamped automated failures accumulate; under-alerting briefly is the fail-safe direction and beats re-creating the false alarm this removes. check-dispatch-provenance-stamped.ts asserts the lockstep, since both sides of the marker fail silently: a drifted `run-name:` leaves the escalation reading every automated run as manual, still green and still "wired" while covering nothing. It also rejects an unconditional marker, which would stamp a human's dispatch as automated and restore the original false alarm. Closes #10234
1 parent 7b6e2e2 commit ebcfec3

11 files changed

Lines changed: 533 additions & 30 deletions

.github/workflows/mcp-release-please.yml

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -178,31 +178,31 @@ jobs:
178178
if: ${{ steps.release.outputs['packages/loopover-mcp--release_created'] == 'true' }}
179179
env:
180180
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
181-
run: gh workflow run publish-mcp.yml --ref "${{ steps.release.outputs['packages/loopover-mcp--tag_name'] }}" -f released_by_release_please=true
181+
run: gh workflow run publish-mcp.yml --ref "${{ steps.release.outputs['packages/loopover-mcp--tag_name'] }}" -f released_by_release_please=true -f dispatched_by_automation=true
182182

183183
- name: Dispatch Engine publish
184184
if: ${{ steps.release.outputs['packages/loopover-engine--release_created'] == 'true' }}
185185
env:
186186
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
187-
run: gh workflow run publish-engine.yml --ref "${{ steps.release.outputs['packages/loopover-engine--tag_name'] }}" -f released_by_release_please=true
187+
run: gh workflow run publish-engine.yml --ref "${{ steps.release.outputs['packages/loopover-engine--tag_name'] }}" -f released_by_release_please=true -f dispatched_by_automation=true
188188

189189
- name: Dispatch Contract publish
190190
if: ${{ steps.release.outputs['packages/loopover-contract--release_created'] == 'true' }}
191191
env:
192192
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
193-
run: gh workflow run publish-contract.yml --ref "${{ steps.release.outputs['packages/loopover-contract--tag_name'] }}" -f released_by_release_please=true
193+
run: gh workflow run publish-contract.yml --ref "${{ steps.release.outputs['packages/loopover-contract--tag_name'] }}" -f released_by_release_please=true -f dispatched_by_automation=true
194194

195195
- name: Dispatch Miner publish
196196
if: ${{ steps.release.outputs['packages/loopover-miner--release_created'] == 'true' }}
197197
env:
198198
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
199-
run: gh workflow run publish-miner.yml --ref "${{ steps.release.outputs['packages/loopover-miner--tag_name'] }}" -f released_by_release_please=true
199+
run: gh workflow run publish-miner.yml --ref "${{ steps.release.outputs['packages/loopover-miner--tag_name'] }}" -f released_by_release_please=true -f dispatched_by_automation=true
200200

201201
- name: Dispatch UI Kit publish
202202
if: ${{ steps.release.outputs['packages/loopover-ui-kit--release_created'] == 'true' }}
203203
env:
204204
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
205-
run: gh workflow run publish-ui-kit.yml --ref "${{ steps.release.outputs['packages/loopover-ui-kit--tag_name'] }}" -f released_by_release_please=true
205+
run: gh workflow run publish-ui-kit.yml --ref "${{ steps.release.outputs['packages/loopover-ui-kit--tag_name'] }}" -f released_by_release_please=true -f dispatched_by_automation=true
206206

207207
# Self-heal for a known, reproducible, upstream googleapis/release-please limitation
208208
# (googleapis/release-please#1946, #1444, #1406 -- all the same "There are untagged, merged release
@@ -282,7 +282,7 @@ jobs:
282282
# `before_ts` only on the rare case the URL wasn't returned.
283283
local before_ts run_id dispatch_output
284284
before_ts="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
285-
dispatch_output="$(gh workflow run "$workflow" --repo "$GITHUB_REPOSITORY" 2>&1)"
285+
dispatch_output="$(gh workflow run "$workflow" --repo "$GITHUB_REPOSITORY" -f dispatched_by_automation=true 2>&1)"
286286
echo "$dispatch_output"
287287
run_id="$(printf '%s' "$dispatch_output" | grep -oE '/runs/[0-9]+' | grep -oE '[0-9]+' | tail -1 || true)"
288288
if [ -z "$run_id" ]; then

.github/workflows/publish-contract.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,18 @@ on:
2323
description: "Internal: set by the release automation's dispatch so this run skips re-creating the GitHub release it already made."
2424
type: boolean
2525
default: false
26+
dispatched_by_automation:
27+
description: "Internal: set by the release automation's dispatch so a hand retry is not counted as an outage. Informational -- it feeds run-name below and nothing else."
28+
type: boolean
29+
default: false
30+
31+
# PROVENANCE STAMP (#10234). `run-name` is the ONLY dispatch-time signal the runs API gives back (as
32+
# `display_title`): the reconcile path dispatches bare against main under a PAT, so `event`, `head_branch`
33+
# and `triggering_actor` are identical to a human's `gh workflow run`, and run objects carry no `inputs`.
34+
# scripts/escalate-workflow-outage.ts recovers "was this automated?" from the marker below -- see its
35+
# header for the full story. The marker must stay in lockstep with that script's
36+
# AUTOMATION_RUN_NAME_MARKER; scripts/check-dispatch-provenance-stamped.ts fails the build if they drift.
37+
run-name: "Publish Contract Package${{ inputs.dispatched_by_automation && ' [automated]' || '' }}"
2638

2739
permissions:
2840
contents: read

.github/workflows/publish-engine.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,18 @@ on:
1818
description: "Internal: set by the release automation's dispatch so this run skips re-creating the GitHub release it already made."
1919
type: boolean
2020
default: false
21+
dispatched_by_automation:
22+
description: "Internal: set by the release automation's dispatch so a hand retry is not counted as an outage. Informational -- it feeds run-name below and nothing else."
23+
type: boolean
24+
default: false
25+
26+
# PROVENANCE STAMP (#10234). `run-name` is the ONLY dispatch-time signal the runs API gives back (as
27+
# `display_title`): the reconcile path dispatches bare against main under a PAT, so `event`, `head_branch`
28+
# and `triggering_actor` are identical to a human's `gh workflow run`, and run objects carry no `inputs`.
29+
# scripts/escalate-workflow-outage.ts recovers "was this automated?" from the marker below -- see its
30+
# header for the full story. The marker must stay in lockstep with that script's
31+
# AUTOMATION_RUN_NAME_MARKER; scripts/check-dispatch-provenance-stamped.ts fails the build if they drift.
32+
run-name: "Publish Engine Package${{ inputs.dispatched_by_automation && ' [automated]' || '' }}"
2133

2234
permissions:
2335
contents: read

.github/workflows/publish-mcp.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,18 @@ on:
1212
description: "Internal: set by the release automation's dispatch so this run skips re-creating the GitHub release it already made."
1313
type: boolean
1414
default: false
15+
dispatched_by_automation:
16+
description: "Internal: set by the release automation's dispatch so a hand retry is not counted as an outage. Informational -- it feeds run-name below and nothing else."
17+
type: boolean
18+
default: false
19+
20+
# PROVENANCE STAMP (#10234). `run-name` is the ONLY dispatch-time signal the runs API gives back (as
21+
# `display_title`): the reconcile path dispatches bare against main under a PAT, so `event`, `head_branch`
22+
# and `triggering_actor` are identical to a human's `gh workflow run`, and run objects carry no `inputs`.
23+
# scripts/escalate-workflow-outage.ts recovers "was this automated?" from the marker below -- see its
24+
# header for the full story. The marker must stay in lockstep with that script's
25+
# AUTOMATION_RUN_NAME_MARKER; scripts/check-dispatch-provenance-stamped.ts fails the build if they drift.
26+
run-name: "Publish MCP Package${{ inputs.dispatched_by_automation && ' [automated]' || '' }}"
1527

1628
permissions:
1729
contents: read

.github/workflows/publish-miner.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,18 @@ on:
1818
description: "Internal: set by the release automation's dispatch so this run skips re-creating the GitHub release it already made."
1919
type: boolean
2020
default: false
21+
dispatched_by_automation:
22+
description: "Internal: set by the release automation's dispatch so a hand retry is not counted as an outage. Informational -- it feeds run-name below and nothing else."
23+
type: boolean
24+
default: false
25+
26+
# PROVENANCE STAMP (#10234). `run-name` is the ONLY dispatch-time signal the runs API gives back (as
27+
# `display_title`): the reconcile path dispatches bare against main under a PAT, so `event`, `head_branch`
28+
# and `triggering_actor` are identical to a human's `gh workflow run`, and run objects carry no `inputs`.
29+
# scripts/escalate-workflow-outage.ts recovers "was this automated?" from the marker below -- see its
30+
# header for the full story. The marker must stay in lockstep with that script's
31+
# AUTOMATION_RUN_NAME_MARKER; scripts/check-dispatch-provenance-stamped.ts fails the build if they drift.
32+
run-name: "Publish Miner Package${{ inputs.dispatched_by_automation && ' [automated]' || '' }}"
2133

2234
permissions:
2335
contents: read

.github/workflows/publish-ui-kit.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,18 @@ on:
1818
description: "Internal: set by the release automation's dispatch so this run skips re-creating the GitHub release it already made."
1919
type: boolean
2020
default: false
21+
dispatched_by_automation:
22+
description: "Internal: set by the release automation's dispatch so a hand retry is not counted as an outage. Informational -- it feeds run-name below and nothing else."
23+
type: boolean
24+
default: false
25+
26+
# PROVENANCE STAMP (#10234). `run-name` is the ONLY dispatch-time signal the runs API gives back (as
27+
# `display_title`): the reconcile path dispatches bare against main under a PAT, so `event`, `head_branch`
28+
# and `triggering_actor` are identical to a human's `gh workflow run`, and run objects carry no `inputs`.
29+
# scripts/escalate-workflow-outage.ts recovers "was this automated?" from the marker below -- see its
30+
# header for the full story. The marker must stay in lockstep with that script's
31+
# AUTOMATION_RUN_NAME_MARKER; scripts/check-dispatch-provenance-stamped.ts fails the build if they drift.
32+
run-name: "Publish UI Kit Package${{ inputs.dispatched_by_automation && ' [automated]' || '' }}"
2133

2234
permissions:
2335
contents: read

package.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,7 @@
101101
"regate-sort-key:check": "node --experimental-strip-types scripts/check-regate-sort-key.ts",
102102
"command-redelivery-guards:check": "node --experimental-strip-types scripts/check-command-redelivery-guards.ts",
103103
"dispatch-gate-reasons:check": "node --experimental-strip-types scripts/check-dispatch-gate-reasons.ts",
104+
"dispatch-provenance:check": "tsx scripts/check-dispatch-provenance-stamped.ts",
104105
"replay-runner-manifest": "tsx scripts/replay-runner-image-manifest.ts",
105106
"replay-runner-manifest:write": "tsx scripts/replay-runner-image-manifest.ts --write",
106107
"replay-runner-manifest:check": "tsx scripts/replay-runner-image-manifest.ts --check",
@@ -149,7 +150,7 @@
149150
"test:smoke:browser:install": "playwright install chromium",
150151
"test:smoke:browser": "node --experimental-strip-types scripts/smoke-ui-browser.ts",
151152
"pretest:ci": "npm run check-node-version",
152-
"test:ci": "git diff --check && npm run actionlint && npm run lint:composite-actions && npm run db:migrations:check && npm run db:migrations:immutable:check && npm run turbo-inputs:check && npm run workspace-dep-ranges:check && npm run db:schema-drift:check && npm run selfhost:env-reference:check && npm run miner:env-reference:check && npm run selfhost:validate-observability && npm run cf-typegen:check && npm run build --workspace @loopover/engine && npm run build --workspace @loopover/discovery-index && npm run build:mcp && npm run build:miner && npm run build --workspace @loopover/ui-kit && npm run typecheck && npm run test:coverage && npm run test:engine-parity && npm run test:live-gate-parity && npm run test:driver-parity && npm run validate:mcp && npm run test --workspace @loopover/engine && npm run test:workers && npm run test:mcp-pack && npm run test:contract-pack && npm run test:miner-pack && npm run test:engine-pack && npm run test:ui-kit-pack && npm run test:miner-deployment-docs-audit && npm run rees:test && npm run ui:openapi:check && npm run control-plane:contract:check && npm run control-plane:openapi:check && npm run ui:version-audit && npm run docs:drift-check && npm run coverage-boltons:check && npm run import-specifiers:check && npm run ui-derived-types:check && npm run server-manifest:check && npm run dead-source-files:check && npm run dead-exports:check && npm run publishable-deps:check && npm run fixture-clock-races:check && npm run typecheck-coverage:check && npm run test-wiring:check && npm run checkers-wired:check && npm run focus-manifest-fields:check && npm run release-commit-parsing:check && npm run releasable-commit-types:check && npm run regate-sort-key:check && npm run maintainer-associations:check && npm run command-redelivery-guards:check && npm run dispatch-gate-reasons:check && npm run validate:no-hand-written-js && npm run replay-runner-manifest:check && npm run coco-dev-versions:check && npm run branding-drift:check && npm run manifest:drift-check && npm run engine-parity:drift-check && npm run engines-nvmrc:check && npm run release-manifest:sync:check && npm run release-linked-versions:check && npm run command-reference:check && npm run mcp:tool-reference:check && npm run mcp:client-config:check && npm run contract:api-schemas:check && npm run ui:lint && npm run ui:typecheck && npm run ui:test && npm run ui:build",
153+
"test:ci": "git diff --check && npm run actionlint && npm run lint:composite-actions && npm run db:migrations:check && npm run db:migrations:immutable:check && npm run turbo-inputs:check && npm run workspace-dep-ranges:check && npm run db:schema-drift:check && npm run selfhost:env-reference:check && npm run miner:env-reference:check && npm run selfhost:validate-observability && npm run cf-typegen:check && npm run build --workspace @loopover/engine && npm run build --workspace @loopover/discovery-index && npm run build:mcp && npm run build:miner && npm run build --workspace @loopover/ui-kit && npm run typecheck && npm run test:coverage && npm run test:engine-parity && npm run test:live-gate-parity && npm run test:driver-parity && npm run validate:mcp && npm run test --workspace @loopover/engine && npm run test:workers && npm run test:mcp-pack && npm run test:contract-pack && npm run test:miner-pack && npm run test:engine-pack && npm run test:ui-kit-pack && npm run test:miner-deployment-docs-audit && npm run rees:test && npm run ui:openapi:check && npm run control-plane:contract:check && npm run control-plane:openapi:check && npm run ui:version-audit && npm run docs:drift-check && npm run coverage-boltons:check && npm run import-specifiers:check && npm run ui-derived-types:check && npm run server-manifest:check && npm run dead-source-files:check && npm run dead-exports:check && npm run publishable-deps:check && npm run fixture-clock-races:check && npm run typecheck-coverage:check && npm run test-wiring:check && npm run checkers-wired:check && npm run focus-manifest-fields:check && npm run release-commit-parsing:check && npm run releasable-commit-types:check && npm run regate-sort-key:check && npm run maintainer-associations:check && npm run command-redelivery-guards:check && npm run dispatch-gate-reasons:check && npm run dispatch-provenance:check && npm run validate:no-hand-written-js && npm run replay-runner-manifest:check && npm run coco-dev-versions:check && npm run branding-drift:check && npm run manifest:drift-check && npm run engine-parity:drift-check && npm run engines-nvmrc:check && npm run release-manifest:sync:check && npm run release-linked-versions:check && npm run command-reference:check && npm run mcp:tool-reference:check && npm run mcp:client-config:check && npm run contract:api-schemas:check && npm run ui:lint && npm run ui:typecheck && npm run ui:test && npm run ui:build",
153154
"test:release": "npm run test:ci && npm run changelog:check",
154155
"test:release:mcp": "npm run test:ci",
155156
"test:watch": "vitest",

0 commit comments

Comments
 (0)