Skip to content

ci(slow): find the Discord seam's engine line in the unit's log, not its answer (#1112) - #1116

Merged
jayden1711 merged 1 commit into
mainfrom
cw-1112-discord-seam
Oct 26, 2026
Merged

jayden1711 merged 1 commit into
mainfrom
cw-1112-discord-seam

Conversation

@jayden1711

@jayden1711 jayden1711 commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Fixes #1112

What broke

The slow job add-project (real systemd) failed at step 19 ("credentials.sh intake-discord … the bot's channels as the coordinator; the Discord step through the hub"), in the Discord seam's case 1 (the engine's own listing, no stand-in, for a token file the coordinator can't read).

Root cause

The job's expectation is stale, and #1096's behaviour is right (inferred, from #1064's intent and the #1096 test above). The answer must not quote the child's line, because it can name a path or quote Discord. Case 1 used that line in the answer as its proof that the engine's own CLI ran, not a decoy. The proof is still available, now in the dashboard's log.

What changed

  • .github/workflows/ci-slow.yml, seam case 1:
    • It checks the exact fixed answer (. == {error: "the bot's channel listing failed; the detail is in the dashboard's log on the machine"}).
    • It reads the seam unit's journal (journalctl -o cat -u ci-discord-seam-engine.service, with a short wait for journald) and grep -qFs the engine CLI's own line: GET /api/first-run/discord/channels: instance ci: can't read its bot token file $seam/unreadable-token (EACCES). That still proves the engine's own CLI ran. The decoy-marks check is unchanged.
    • The new log file is added to the step's "token never echoed" scan.
  • test/discord-seam-ci.test.ts (new, ordinary suite): runs scripts/ci/discord-seam-dashboard.mjs engine as the current user, without systemd. It uses an instance whose token file is mode 000, plus the same decoys (a CLI and a node first on PATH, a NODE_OPTIONS preload, the *_CLI/*_ENGINE variables). It asserts the 502 answer, the logged line and that no decoy ran. Then it runs the workflow's own case-1 check lines ([ "$code" = … ], every jq -e and grep -q) against the answer and the dashboard's output. A stale expectation in the workflow now fails npm test, not only the real-systemd job.
  • changelog.d/1112-discord-seam-log-check.md (Guardrail changes).

Nothing #1096 added is weakened: no src/ change, and test/discord-setup.test.ts is untouched (measured, git show --stat HEAD: three files, none under src/).

Fail-before / pass-after (measured)

Build: nice -n 15 ionice -c3 npm run build. Test: node --import ./dist/test/isolate.js --test --test-timeout=180000 dist/test/discord-seam-ci.test.js.

  • Before (git show origin/main:.github/workflows/ci-slow.yml > .github/workflows/ci-slow.yml, same test): fail 1, with the message the workflow's check failed: jq -e '.error | test("^instance ci: can.t read its bot token file /var/tmp/ci-discord-seam/unreadable-token \\(EACCES\\)")' "$F/seam-engine.json" and false. That is CI's failure.
  • After (head of this PR): pass 1, fail 0.

Other checks (measured)

  • npm run typecheck: exit 0.
  • npm run check:denylist: denylist: 1684 files clean.
  • Targeted: discord-seam-ci 1/1, discord-setup 13/13, changelog 27/27, testrun-tmpdir 7/7, ci-batch 17/17.
  • Full suite, once, under the shared lock (flock /var/tmp/wl-coord/fulltest.lock nice -n 15 ionice -c3 npm test): 5952 tests, 5838 pass, 19 fail, 69 skipped. The failures are in machine-sudoers-closed (13), machine-sudoers-removal, machine-sudoers-narrow, coordinator, add-project-api-key, desktop-removed and browser-run.
    • None of those files reads a file this PR changes (measured, rg for ci-slow|changelog.d|discord-seam in them: none).
    • Rerun with origin/main's files in place, they fail the same way, except browser-run, which passed (12/0) and so looks flaky (measured).
    • The desktop-removed failure is yaml resolves to …/payload/worklane/node_modules/yaml/…, from the worktree's symlinked node_modules (inferred: environment, not this change).
    • CI is the authoritative run.

Not done here

  • The journal read on a real host isn't exercised by the ordinary test (inferred: the test uses the process's own stdout and stderr, which is what systemd writes to the unit's journal). The slow-job run below covers it.

Slow job on this branch (measured)

gh workflow run ci-slow.yml --ref cw-1112-discord-seam -f job=add-project-real-systemd started run 40320282250 on f0ae293. It ended success: add-project (real systemd) passed, including step 19. Its log (gh api repos/jayden1711/worklane/actions/jobs/121563232114/logs) shows these for case 1:

  • the answer {"error":"the bot's channel listing failed; the detail is in the dashboard's log on the machine"};
  • in the unit's journal, GET /api/first-run/discord/channels: instance ci: can't read its bot token file /var/tmp/ci-discord-seam/unreadable-token (EACCES); run this as the instance's user;
  • an empty marks:, so no decoy ran.

Cases 2 and 3 passed unchanged.

Critical set

Yes. It changes .github/workflows/ci-slow.yml ("CI workflows" in docs/process.md's critical set; src/review.ts lists .github/workflows/**), so it needs two independent reviews.

…its answer (#1112)

Since #1064 (PR #1096) a failed channel listing answers only that the
listing failed and logs the engine CLI's own line. The add-project job's
seam case 1 still expected that line in the answer, so it went red.

Check the exact fixed answer, and find the engine CLI's line for the
unreadable token file in the seam unit's journal, which still proves the
engine's own CLI ran. test/discord-seam-ci.test.ts runs the same seam
script and decoys in the ordinary suite and runs the workflow's own
checks of that case against what came back.

Fixes #1112

Signed-off-by: Jayden Vasquez <jaydenvasquez1711@gmail.com>
@jayden1711

Copy link
Copy Markdown
Owner Author

Independent final check (R116a), posted by the coordinator.

VERDICT: PASS @f0ae293b448b64a6358d17a2608ff2b0e39f493f (final check)

Reviewer R116a, final check of PR #1116 (fixes #1112). Critical set: yes (.github/workflows/ci-slow.yml).

Findings

  • Head (measured, gh pr view 1116 --json headRefOid): f0ae293. Unchanged while I worked.
  • Diff matches the body (measured, git diff origin/main...HEAD): three files (ci-slow.yml +7/-3, the changelog fragment and the new test). No src/ change.
  • The proof still holds (measured, rg): the EACCES line is only in src/intake/discord-channels.ts:183. Only src/cli.ts imports that module at runtime (discord-setup.ts imports only types from it). So the line in the unit's journal still shows the engine's own CLI ran. A decoy prints {} and would not produce it. The answer check is now an exact . == {error: …}, which is stricter than the old test(^…). The marks (no-decoy) check at line 1513 is unchanged.
  • The test runs the workflow's own lines (measured): engineCase() reads ci-slow.yml at runtime and runs every [ "$code" =, jq -e and grep -q line of case 1 in bash. These mutants of the head's yml all failed the test 1/1:
    • main's yml (the stale .error jq);
    • the log grep dropped;
    • a wrong path in the grep;
    • wrong answer text;
    • code = 200.
  • Fail-before and pass-after (measured): node --import ./dist/test/isolate.js --test dist/test/discord-seam-ci.test.js. It fails 1/1 with origin/main:.github/workflows/ci-slow.yml (the same jq failure as CI) and passes 1/1 on the head.
  • Slow run 40320282250 (measured, gh run view … --json headSha,conclusion,jobs): headSha f0ae293, workflow_dispatch, success. Step 19 of add-project (real systemd) succeeded. Its job log (gh api …/jobs/121563232114/logs) shows:
    • the fixed answer;
    • the journal line …unreadable-token (EACCES); run this as the instance's user;
    • an empty marks:.
  • typecheck (measured, npm run typecheck): exit 0. Denylist (measured, npm run check:denylist): 1684 files clean.
  • Required CI (measured, gh pr checks 1116 --required): every check passes except test (ubuntu-latest, 22). That check is red because test shard 2/4 (ubuntu-latest, 22) had one failure, in dashboard-switch-speed.test.js:328. The message is "busy's dashboard isn't answering … card count: 0" (gh api …/jobs/121562946525/logs). That is the known flake in open issue dashboard-switch-speed: say which check failed; keep a shown Board when one fresh read fails (rare flake) #639. The same test passed on Node 24. The new test passed in that shard (ok 417). Not caused by this change (inferred: no src change, and the failing test reads none of the changed files). A rerun of that shard is needed before merge.
  • Changelog fragment: present and accurate. "Since dashboard: the remaining raw git/gh stderr and paths in answers (follow-up to #882) #1064" is right: fix(dashboard): first-run answers log git's, gh's and a child's text, never show it (#1064) #1096 is the PR that fixed dashboard: the remaining raw git/gh stderr and paths in answers (follow-up to #882) #1064.

Weakening scan flags

  1. The .error | test(EACCES line) jq was removed. Not a weakening: an exact-answer check plus a grep for the same line in the journal replace it.
  2. The journal read loop …&& break gives up quietly after 10 tries. Not a weakening: the grep -qF after it is the gate and fails if the line is missing.
  3. The token-echo scan list gained seam-engine.log. Strengthening.
  4. The new test skips on win32 and as root. Not a weakening: the test is new, and a 000 file can't be tested as root.
  5. The test's line filter only runs [ "$code" =, jq -e and grep -q lines. Not a weakening. There is a follow-up below.

What I ran

  • A detached worktree at
  • TMPDIR=<tmp>, with nice and ionice.
  • The build, the targeted test with five yml variants, typecheck and denylist.
  • Read-only gh calls only. No systemd touched.

Follow-ups

@jayden1711

Copy link
Copy Markdown
Owner Author

Independent adversarial review (R116b), posted by the coordinator.

VERDICT: PASS @f0ae293b448b64a6358d17a2608ff2b0e39f493f (adversarial)

Head confirmed with gh pr view. Own detached worktree, TMPDIR from mktemp under /var/tmp, nice/ionice, targeted tests only. No systemd units touched.

Findings

  • Fail-before / pass-after (measured, node --import ./dist/test/isolate.js --test dist/test/discord-seam-ci.test.js). fail 0 on the head; fail 1 with origin/main's ci-slow.yml in place. Typecheck exit 0; denylist reports 1684 files clean.
  • The checks come from the workflow file (measured). engineCase() reads ci-slow.yml at runtime. One workflow mutation at a time:
    • Caught: main's old jq line, wrong grep text, grep -v, jq -e changed to jq, code = 200, the jq line deleted, the grep line deleted.
    • Not caught: see follow-ups.
  • Leaked raw error (measured). refusalText patched to append the cause makes the test fail. The workflow's jq line exits 1 for an answer that quotes the cause or adds a cause key, and 0 only for the exact fixed answer.
  • Decoys (measured). Patching the listing to run the WORKLANE_DISCORD_CLI decoy fails the test on the marks check.
    • A forger that leaves no mark and prints the engine's exact line passes the answer and log checks.
    • That is no weaker than main (inferred): main matched the same child line in the answer, so the same forger passed it. The marks check, unchanged, is what catches decoys, and every CI decoy writes a mark.
  • Journal scope (inferred). The ci-discord-seam-engine unit appears only in this step (measured, rg). It is started once on a fresh hosted runner, so there are no stale entries. The slow job passed on this head (run 40320282250, gh pr checks).
  • CI (measured, gh api job log). Shard 2/4 on Node 22 is red from dashboard-switch-speed (timing, ECONNRESET), not this change. The new test passed there (ok 417).

Weakening scan flags

The diff removes three lines from ci-slow.yml:

  1. Case-1 comment: reworded. Not a weakening.
  2. jq -e '.error | test("^instance ci: …EACCES")': replaced by an exact-answer jq plus a grep -qF for the full line in the journal. Equal or stronger: the answer is pinned exactly and the engine line is still required, now in the log. Not a weakening.
  3. "Never echoed" scan: replaced by a superset that also scans seam-engine.log. A strengthening.

The marks check is untouched and there are no src/ changes.

Follow-ups (not FAILs)

  • Surviving mutants: the test only checks each line's shape, so these pass: the jq filter weakened to '.error', || true on the grep, an extra file added to the grep. Fix: pin the full lines, or require each check to fail on a wrong answer and log.
  • Journal read untested: the wrong unit or a dropped -u survives the ordinary test (disclosed in the PR; the slow job covers it). _SYSTEMD_INVOCATION_ID scoping would rule out stale entries.
  • Forger gap: a decoy that leaves no mark and prints the engine's exact line is indistinguishable by output, as on main.

@jayden1711

Copy link
Copy Markdown
Owner Author

Weakening scan decision for head f0ae293b448b64a6358d17a2608ff2b0e39f493f, recorded by the coordinator under the owner's delegation.

The owner's instruction (verbatim): "Stp asking me quesgtions and just take recommended option each time". This PR is in the critical set (it changes a CI workflow). Both independent reviews of this exact head passed: the final check (R116a) and the adversarial review (R116b). Both classified each of the scan's flags as a strengthening or not a weakening; neither found a real weakening.

The flags, as classified (measured, R116a and R116b):

  1. ci-slow.yml case 1, the removed jq -e '.error | test("^instance ci: …EACCES")': NOT A WEAKENING. It is replaced by an exact match on the fixed answer fix(dashboard): first-run answers log git's, gh's and a child's text, never show it (#1064) #1096 gives, plus a grep -qF for the same engine-CLI line in the seam unit's journal, so the engine's own CLI is still proven to have run.
  2. The journal read loop gives up after 10 tries: NOT A WEAKENING. The grep -qF after it is the gate and fails when the line is missing.
  3. The "token never echoed" scan now also covers seam-engine.log: STRENGTHENING.
  4. The new test skips on win32 and as root: LIMIT. The test is new, and an unreadable (mode 000) file can't be tested as root.
  5. The case-1 comment was reworded: NOT A WEAKENING.

The decoy-marks check is unchanged and there is no src/ change. Follow-ups from R116b (shape-only line checks, journal read not exercised by the ordinary test, filtering on the unit's invocation ID) will be filed as an issue.

Decision: proceed (recommended option). This covers only this head; any new push needs a fresh scan and review.

@jayden1711
jayden1711 merged commit 246e3f6 into main Oct 26, 2026
62 of 66 checks passed
@jayden1711
jayden1711 deleted the cw-1112-discord-seam branch October 26, 2026 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI batch red: "add-project (real systemd)" broke at c41f37c (#1096)

1 participant