ci(docs): autonomous docs-audit merge gated on an explicit confidence marker#187
Conversation
… marker Lets the docs-audit bot merge fully autonomously when it is confident, while still parking anything it flags for a human. - update-docs prompt: the auditing agent must end every PR body with exactly one marker — `<!-- DOCS-AUDIT: CLEAR -->` (everything verified, no open questions) or `<!-- DOCS-AUDIT: HOLD -->` (open questions / unverifiable claims / follow-ups). Decision rule + "when unsure, HOLD". - docs-auto-merge: drop the `!draft` job gate so the workflow evaluates drafts too; replace the brittle prose-regex guard with a marker check. CLEAR -> mark ready for review + enable squash auto-merge. HOLD or no marker -> leave as a draft (fail-safe). Draft now means "the agent wasn't sure", which is the human-in-the-loop signal we actually want. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploying automem-website with
|
| Latest commit: |
d61bc32
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://f2fe8f68.automem-website.pages.dev |
| Branch Preview URL: | https://ci-docs-autonomous-merge.automem-website.pages.dev |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f69cb57ab8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if printf '%s' "$BODY" | grep -Eiq '<!--[[:space:]]*docs-audit:[[:space:]]*hold[[:space:]]*-->'; then | ||
| echo "PR is marked HOLD; leaving it as a draft for maintainer review." | ||
| echo "eligible=false" >> "$GITHUB_OUTPUT" | ||
| exit 0 |
There was a problem hiding this comment.
If an earlier CLEAR run has already enabled auto-merge, a later synchronize where the bot changes the body to HOLD only writes eligible=false and exits, leaving the existing autoMergeRequest in place. GitHub only auto-disables auto-merge when someone without write access pushes new changes, so these same-repo docs/audit-* bot updates can still merge after CI despite the HOLD marker. Please disable any existing auto-merge request before exiting on HOLD or missing-marker paths.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in d61bc32. Added a "Cancel stale auto-merge when held" step on the eligible != 'true' path that runs gh pr merge --disable-auto when an autoMergeRequest is present, so a CLEAR→HOLD flip tears down the prior request instead of merging after CI. Confirmed the write-access cancellation behavior you described.
| # <!-- DOCS-AUDIT: CLEAR --> -> agent verified everything; mark ready + auto-merge. | ||
| # <!-- DOCS-AUDIT: HOLD --> -> agent flagged open questions; leave as a draft. | ||
| # Fail-safe: a HOLD marker OR no marker at all keeps the PR a draft for a human. | ||
| # To ship a HELD PR, a maintainer replaces the marker with CLEAR (or merges by hand); |
There was a problem hiding this comment.
This documented override path does not actually run the guard: the workflow only listens for opened, reopened, synchronize, and ready_for_review, so editing a PR body from HOLD to CLEAR does not fire this workflow or mark the PR ready. A held draft will remain parked unless the maintainer also causes another configured event, so add an edited trigger or document the required extra action.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in d61bc32. Added edited to the pull_request_target trigger types so replacing HOLD with CLEAR in the body re-runs the guard and ships the PR. The job if still gates to same-repo docs/audit-* branches and the steps only use the gh CLI (no PR checkout), so the added trigger doesn't widen the privileged surface.
- Cancel stale auto-merge on HOLD/no-marker: a prior CLEAR run enables auto-merge, and GitHub does not auto-cancel it for same-repo bot pushes (write access), so a CLEAR->HOLD flip would still merge after CI. The not-eligible path now runs `gh pr merge --disable-auto`. - Add the `edited` trigger so editing a PR body from HOLD to CLEAR actually re-runs the guard — the documented maintainer override. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
Summary
Makes the docs-audit pipeline run fully autonomously when the agent is confident, while still parking anything it flags for a human — replacing the brittle prose-regex gate added in #173.
The problem
anthropics/claude-code-action@v1opens docs-audit PRs as drafts by default. The previous auto-merge workflow (#173) skipped drafts entirely (!github.event.pull_request.draft) and only ran onready_for_review, so every docs PR sat in draft until a human marked it ready — they piled up unreviewed. Its "confidence" check also grepped the PR body prose for phrases likeopen question/maintainer input, which is phrasing-dependent: e.g. #177's "Follow-ups (… anchors still wrong)" section slipped past the regex.The change
.github/prompts/update-docs.md— the auditing agent must now end every PR body with exactly one machine marker:<!-- DOCS-AUDIT: CLEAR -->— every edit verified against a named SHA, no open questions.<!-- DOCS-AUDIT: HOLD -->— open questions, unverifiable claims, or follow-ups.Plus an explicit decision rule ("if you wrote an Open questions / Follow-ups section, you MUST use HOLD; when unsure, HOLD").
.github/workflows/docs-auto-merge.yml!github.event.pull_request.draftfrom the job condition so drafts are evaluated too.gh pr ready+ enable squash auto-merge.Net effect: "draft" stops meaning "nobody readied it yet" and starts meaning "the agent wasn't sure" — the human-in-the-loop signal we actually want.
Maintainer override
A HELD PR won't auto-merge just by clicking "Ready for review" (the body still says HOLD). To ship one, replace the marker with
CLEAR(asynchronizeevent re-evaluates it) or merge it by hand.Validation
yaml.safe_loadparses the workflow.CLEAR,HOLD, no-spaces<!--DOCS-AUDIT:CLEAR-->, and missing-marker inputs — all classify correctly.Note on existing open drafts
The four currently-open docs drafts (#177, #178, #179, #183) predate the marker convention, so under this workflow they correctly stay drafts (no marker → HOLD). They're being handled separately — their fixes are good; their open questions are being filed as tracking issues.
🤖 Generated with Claude Code