Pre-release: simplified README, blind-UAT'd docs, eleven bugs found and fixed - #48
Merged
Merged
Conversation
The README duplicated large parts of REFERENCE.md, LINTING.md and FLEET.md, often verbatim, and at 528 lines broke the project's own <200-line rule. It is now a ~190-line front door: pitch, install, routing table, a 60-second tour, core concepts, and one-paragraph pointers into the docs. The worked examples (bootstrap, modify, plan/apply review, refusals) move to the new docs/GUIDE.md, kept under 200 lines. Also: link the orphaned docs/CONCEPTS.md, SECURITY.md and CHANGELOG.md from the doc map, document add_owner/remove_owner's bracketed-list form (R-33) in the ops table, mention the except clause, fix the dead #audit---lint anchor in REFERENCE.md, and add lint's --policy flag to its synopsis. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lg69i7e9Z96dPzrW2Zfq3K
Each test states behavior the tool should have and currently does not, so each FAILS until its bug is fixed and then becomes the regression guard: - plan/apply skip sync's repo-root guard and write a CODEOWNERS GitHub never reads, reporting success (the "dead on arrival" outcome) - a \#-escaped pattern is written and "proven" although S-6 says GitHub honors no such escape, so the rule is dead on GitHub - a symlinked .github/CODEOWNERS inside the clone is written through silently, though the docs state GitHub does not follow it - the S-7 branch-mismatch refusal embeds raw "--end-of-options\n" from git rev-parse in its error text - --file ./.github/CODEOWNERS (uncleaned spelling) draws a false "governs nothing" warning that flows into records and summaries - set_owners authors a shadowed duplicate rule without disclosing it; the R-7 warning only fires on the next run - audit --format json prints "audit clean" after the JSON on the clean path, breaking every jq pipe exactly on healthy repos - positional arguments are silently dropped along with every flag after them, so "audit <repo> --checks a999" audits the cwd at exit 0 - audit accepts an unknown --format and silently falls back to text; sync, check and lint all refuse at exit 3 - co-own.sh un-anchors single-segment scopes (/docs/ becomes docs), so the op and its verify both run broader than the operator typed: out-of-scope paths change owners and verify reports all-within-scope docs/BEHAVIOR.md regenerated (make docs) for the new test doc comments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lg69i7e9Z96dPzrW2Zfq3K
There was a problem hiding this comment.
🟡 Changes recommended
The newly added prerelease tests are designed to fail and will keep the default test suite/CI red unless they’re gated (e.g., build tags or skipped by default).
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR prepares for a pre-release by restructuring the top-level documentation (moving worked examples into a new guide) and adding “known bug” e2e tests that encode currently failing behaviors as executable specs.
Changes:
- Simplifies
README.mdinto a shorter “front door” and updates the doc map to point readers to dedicated guides/reference docs. - Adds
docs/GUIDE.mdwith worked end-to-end walkthroughs and refreshes generateddocs/BEHAVIOR.md. - Introduces new prerelease “known bug” e2e test files for
internal/cliandtools/fleet.
File summaries
| File | Description |
|---|---|
| README.md | Slimmed/reshaped README with updated pointers and a short “tour” section. |
| docs/GUIDE.md | New worked-examples guide extracted from the README. |
| docs/REFERENCE.md | Fixes/updates references in command synopsis and anchor/linking around lint. |
| docs/BEHAVIOR.md | Regenerated behavior documentation to include the new prerelease test docs. |
| internal/cli/prerelease_bugs_test.go | Adds prerelease failing e2e tests capturing known CLI bugs. |
| tools/fleet/prerelease_bugs_test.go | Adds prerelease failing e2e test capturing a known co-own.sh scope bug. |
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…le gap A blind subagent completed a 9-task CODEOWNERS-editing suite using only the docs (9/9 verified by independent grading), but its log named real friction: - GUIDE's "When it refuses" gave advice with no runnable command; the two working escape hatches (narrow the scope spelling, or carve out paths with an except clause) are now shown with ops, and except.md is linked from the section where a stuck user actually stands. - snapshot reads the committed file and verify counts tree membership as change; the reviewing section now states both hygiene rules, and REFERENCE.md gains a snapshot-and-verify section (exit codes, --scope, [] vs null in the ownership map). - The [] vs null semantics are stated where snapshot output first appears in the README, along with per-command --help discoverability. - REFERENCE's Operations table now covers the R-33 bracketed-list form. - LINTING.md states plainly that offline dead-rule removal is unsupported. That last one is also a product gap, not just a docs one: a rule audit proves dead from the tree alone (A-4/A-5, no API involved) cannot be removed without a GitHub token — lint refuses the whole run at exit 5 citing owner-existence rules that don't apply to --remove-stale-paths. New failing e2e test TestKnownBug_OfflineStaleRuleRemovalReportable documents it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lg69i7e9Z96dPzrW2Zfq3K
A fresh blind agent completed all 14 CODEOWNERS-editing tasks (basic through adversarial) from the docs alone — 14/14 under independent grading. Its two remaining complaints were both one-line gaps: scope spellings with spaces (backslash escape) appeared only in except.md's grammar, and the README tour never said snapshot reads the committed file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lg69i7e9Z96dPzrW2Zfq3K
…ates (R-7) Two of the eleven pre-release findings, turning their formerly-failing e2e tests green: - pattern.Compile now rejects a pattern beginning with \#, mirroring the existing ! treatment: GitHub honors no such escape, so a written \#-rule is a comment there and the tool's proven-tree claim would be false in the one place it matters. Only the leading position is rejected — a mid-pattern hash needs no escape and keeps its oracle-compatible literal meaning. A pre-existing \# line degrades through the established invalid-line path (skipped in resolution, reported by A-8). Differential fuzz vs the hmarr oracle: 0 mismatches on two seeds. - synthSet now warns, in the same run, when its inserted rule shadows an earlier byte-equal pattern — previously the R-7 duplicate warning only fired on the NEXT run, leaving the run that authored the dead line silent. Written bytes are unchanged (R-1); only the disclosure is new. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lg69i7e9Z96dPzrW2Zfq3K
The script stripped the anchoring slash from the typed scope, so a single-segment /docs/ became the unanchored docs — matching at any depth — and BOTH the op it ran and the verify --scope that was supposed to prove the edit used that broadened spelling: out-of-scope paths changed owners and the proof blessed it. The op and verify now receive the scope exactly as typed (the tool's own matcher supplies real anchoring semantics); the branch name and awk line-match keep their stripped spelling; the jq state logic gains an anchored/unanchored flag so its decisions agree with what the op will do. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lg69i7e9Z96dPzrW2Zfq3K
- plan and apply now run the same checkRepoRoot guard as sync: a --repo below the repository root writes a CODEOWNERS GitHub never reads, so it is refused at exit 2 before any artifact is written. - audit --format json emits exactly one JSON document; the 'audit clean' verdict line is text-mode only. - Every verb rejects leftover positional arguments at exit 3, naming the stray argument — Go's flag parser silently dropped them and every flag after them. - audit validates --format like sync/check/lint do; never a silent fallback to text. - headLabel no longer passes --end-of-options to rev-parse (which echoes it as output in filter mode), fixing the garbled S-7 refusal and making the detached-HEAD bare-SHA label reachable. - The S-8 location check classifies the cleaned --file spelling, so ./.github/CODEOWNERS no longer draws a false 'governs nothing' warning in output, records, and summaries. - A symlinked write target is refused at exit 2 in sync, apply, and lint: GitHub does not follow a symlinked CODEOWNERS, so writing through one is the 'applied, dead on arrival' outcome the containment guard exists to prevent. Symlinks elsewhere in the repo stay irrelevant. Turns six formerly-failing KnownBug e2e tests green; adds nine hardening tests in prerelease_fixes_test.go. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lg69i7e9Z96dPzrW2Zfq3K
A rule the offline audit proves dead from the tree alone (A-4/A-5) could not be repaired without a GitHub token: lint refused the whole run at exit 5 citing R-12, whose rationale — owner existence is an API fact — does not apply to stale paths, a git-tree fact. The only offline remedy was the hand edit the tool exists to prevent. Offline (no token / no --github-repo) with --remove-stale-paths now runs only the dead-rule removal stage against the tree at --branch: owner checks are skipped entirely — never degraded — and the skip is disclosed in text output and as owner_checks_skipped in JSON. The A-5 case-only miss stays spared exactly as online (a typo, not a dead rule; exit 4). Offline without --remove-stale-paths keeps the exit-5 refusal, now naming the escape hatch. R-12 fail-closed behavior for owner repairs is untouched: no owner is looked up, repaired, or removed offline, and the Verifier is never constructed. Also carries the symlinked-target refusal wiring in the lint verb from the companion CLI commit. Turns the last KnownBug e2e test green; adds twelve offline-mode tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lg69i7e9Z96dPzrW2Zfq3K
All eleven formerly-failing tests now pass; the KnownBug_ prefix and the expected-to-fail headers are gone, the finding descriptions stay as the record of what each test guards. docs/BEHAVIOR.md regenerated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lg69i7e9Z96dPzrW2Zfq3K
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lg69i7e9Z96dPzrW2Zfq3K
Review findings from the fix-verification pass: - The OnZeroMatch switch lost its default arm in the except work, so an unrecognized value on a zero-match scope fell through to a silent no-op reported as unchanged/proven-tree. Restored as an exit-3 InvalidError naming the value; the sibling OnExceptZeroMatch switch had the milder form of the same hole (typos swallowed into require behavior at the wrong exit class) and gets the same treatment. - co-own.sh compared owners byte-exactly while the tool folds @handle case everywhere (R-38), so a case-variant spelling landed the repo in the wrong lane. Every owner comparison in the script's jq and awk now folds @-prefixed owners and leaves email owners byte-exact, mirroring ops.FoldOwner. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lg69i7e9Z96dPzrW2Zfq3K
- refuseSymlinkedTarget now Lstats every component of the write path below the repo root, not just the final one: git records a symlinked directory as a link blob, so .github -> elsewhere made .github/CODEOWNERS a path GitHub's tree never contains while sync/apply/lint wrote through it and reported applied. The refusal names the offending component and its target; symlinks off the write path stay irrelevant. - The R-25 ceiling refusal no longer names already-satisfied ops as 'the op(s) behind the number' — only ops that actually change paths. - Every record-less sync exit-3 now emits the documented 'no record was written' stderr note when --out/--summary-out was asked for, through one choke point instead of two special cases. - The stale-comment warning checks the leading token boundary too, so a handle embedded in a longer token (someone@old-team) no longer draws a false 'a comment still names' warning on rename. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lg69i7e9Z96dPzrW2Zfq3K
Review findings against the new offline tree-only mode: - The mode now engages only when NEITHER a token nor --github-repo was given. A run that supplied either one asked for the credentialed lint and refuses at exit 5 naming exactly the missing half — never a silent degradation — and a malformed --github-repo is exit 3 whether or not a token is present. The disclosure sentence is now true by construction. - The exit-5 refusal's escape hatch is policy-aware: under --policy it names the remove_stale_paths field in the lint block instead of the exit-3-banned flag (the R-36b double-refusal trap). - Offline runs report invalid lines instead of exiting 0 over them: a syntactically broken line is a tree fact GitHub is already skipping, so it is needs-human at exit 4 with a reason naming the credentialed run that can repair it. No repair happens offline. - Pre-existing one-token bug: remove-dead-owner actions read the reason map with the unfolded spelling, so a mixed-case dead owner got Reason:. Docs conditioned to match (LINTING.md flag/exit tables and error walk- through, REFERENCE.md required-flags and exit-5 rows, owner_checks_skipped in the JSON contract); BEHAVIOR.md regenerated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lg69i7e9Z96dPzrW2Zfq3K
jordonpeterson
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-release deep dive in three phases: README simplification with verified accuracy, a blind-UAT acceptance loop on the docs, and an adversarial bug hunt whose eleven confirmed findings were first encoded as failing e2e tests and are now all fixed — each test flipped into a regression guard. CI should be fully green.
Phase 1 — README (commits 1, 4, 5, 8)
docs/GUIDE.md(<200 lines, per the repo's own rule). ~60% of the old README duplicated REFERENCE/LINTING/FLEET verbatim.docs/CONCEPTS.md,SECURITY.md,CHANGELOG.md; documented the R-33 list form and theexceptclause; fixed a dead anchor; addedlint --policyto the synopsis.Phase 2 — Blind-UAT acceptance loop (commits 4–5)
A sandbox with only docs + binary + fixture repos; fresh "blind" subagents complete increasingly nasty CODEOWNERS change requests from the docs alone, graded by script against resolved ownership. Round 1: 9/9 with logged friction → docs iteration (runnable escape hatches in "When it refuses", snapshot/verify hygiene,
[]vsnull, a REFERENCEsnapshot/verifysection). Round 2, harder 14-task suite, fresh agent: 14/14. Acceptance criterion met.Phase 3 — Eleven bugs: encoded as failing tests, then fixed (commits 2–3, 6–13)
Every bug was reproduced, minimized, and pinned by a failing e2e test before any fix; fixes were built by parallel subagents with disjoint file ownership, then the combined diff went through a code-review pass and an adversarial verification agent whose findings were fixed in turn.
sync.go/cli.goplan/applyskipped the repo-root guard — wrote a CODEOWNERS GitHub never reads, reported successco-own.shverifyproof ran broader than typedpattern.go\#patterns written and "proven" although GitHub honors no such escape (leading position rejected; oracle-verified, fuzz 0 mismatches)cli.gocli.goaudit --format jsonprinted "audit clean" after the JSON, breakingjqon healthy reposcli.golint--remove-stale-paths(engages only when neither credential is given; owner checks skipped, disclosed, never degraded silently; invalid lines still reported at exit 4)sync.go--end-of-optionsechoed into the S-7 refusal; detached-HEAD label unreachablesync.go--filespellings drew a false "governs nothing" warning into records and PR bodiescli.goauditsilently accepted unknown--format; now exit 3 like every other verbplan.goset_ownersauthored a shadowed duplicate without disclosing it; the R-7 warning now fires in the authoring runReview-pass findings also fixed (commits 11–13): restored
default:refusal arms in both zero-match switches (unknown values had become silent no-ops); owner comparisons inco-own.shnow fold@handlecase per R-38 (emails stay byte-exact); the R-25 ceiling refusal names only ops that changed paths; every record-lesssyncexit-3 emits the documented "no record was written" note; the stale-comment rename warning checks the leading token boundary (no more false hits onsomeone@old-team); policy-aware offline-lint refusal wording; a mixed-case dead owner's emptyReason; five stale doc lines updated to the new lint contract.Deliberately deferred (follow-up material)
ops.StaticConflictis O(n²) with the expensive containment check first (a 1MB policy under the documented cap extrapolates to hours — swapcommutesahead ofscopeCoverage);commuteOnEveryOwnerSetenumerates 2^n owner subsets on large R-33 lists in the declare path.verifyrenders tree-membership changes as(unowned) → (unowned); UTF-8 BOM makes the first rule silently dead; audit of user-owned (non-org) repos can never conclude; A-7 prints Go-style line lists; refusal text prints twice (error + summary echo); the difftest generator's alphabet contains no#, so the\#change is covered by unit tests and a hand harness, not the fuzz.docs/PROPOSAL-*.mdout of user-facing docs.Test suite: full
go test ./...green, differential fuzz 0 mismatches (multiple seeds),make docsclean, blind-UAT harness available in session scratchpad for re-running against future changes.🤖 Generated with Claude Code
https://claude.ai/code/session_01Lg69i7e9Z96dPzrW2Zfq3K