Redaction: one shared list of token shapes, prefixed dashboard keys, no own redactors (#889) [PROTECTED: owner merges]

changelog.d/921-one-shared-redactor.md

@@ -0,0 +1,12 @@
+### Upgrading (from v0.5.2)
+
+- **Dashboard keys now start with `wld_` (#889).** A new key is `wld_` and 32 random characters, a shape the redactor finds wherever it appears. A key from before keeps working until you rotate it. `doctor` warns on each older key it can read (the local dashboard's, each instance's and the hub's) and says how to rotate it: remove the key file and restart what serves that dashboard. On a server with the hub, then run `scripts/setup/dashboards.sh` as root so the hub can read the new key.
+
+### Guardrail changes: what each one does
+
+- **One shared list of token shapes (#889).** The engine's redactor (`src/events/redact.ts`) now knows a Discord bot token, a Tailscale key, a dashboard key and an HTTP Basic credential. A shape with a distinctive prefix (`ghp_`, `github_pat_`, `sk-ant-`, `xoxb-`, `lin_api_`, `AKIA`, `AIza`, `eyJ`, `wld_`, ...) is found wherever its prefix is, whatever is glued before it. The other shapes (a Discord bot token, a Bearer or Basic credential in any case, a Stripe key, a plain `sk-` key) are found after a name glued with `_`, an escape sequence (ESC as a byte, as `\u001b`, `\x1b`, `\033`, `\e` or `^[`, and the 8-bit CSI) or a JSON escape (`\n`). Before, Discord and Tailscale were each redacted only by one module of their own, so a Discord bot token passed through to the event log, issue comments, chat notices and the dashboard's service log.
+- **A private key fails closed.** A key's BEGIN line with no END after it is redacted to the end of the text, and an END line with no BEGIN before it from the start, so a key cut short, still being written, or begun above a log window is not shown. This covers PEM keys (upper-case BEGIN and END lines with five dashes) and PGP's private key block. Not covered: a key whose BEGIN and END are both outside a window (only its middle lines in view), and other formats (an ssh.com SSH2 key, a lower-case PEM line, a PuTTY `.ppk` file).
+- **The rules no longer run one after another.** Every rule reads the text as it was, and each run of what they find is replaced once, so one rule's match can't take in the text another rule starts from. Before, a dashboard key, a JWT, a Slack or Tailscale token or a bearer value written right before a URL with a password, a bearer token or a Linear key could hide it from the next rule (`wld_<key>-redis://app:<password>@…`). A JWT is found in any dotted part of a run (`sess_0123456789.<jwt>`), an unsigned one (`alg: none`) too. An HTTP Basic value is found whatever is glued after it, and a short one too (base64 of a short `user:password`). A password in a URL is found whatever the scheme's length. Where one rule's find covers the others, it shows that rule's marker (a bearer token that is an `sk-` key now shows as `Bearer [REDACTED]`); where finds only overlap, the run shows `[REDACTED]`. A test holds the default mode to the previous release's rules on composed input.
+- **No module keeps its own token rules.** The GitHub sign-in relay, the bench and the root updater now redact through the shared redactor. Its broad mode runs every rule on the text as it was and masks what any of them finds. That covers everything those modules masked before: a token cut short, a clone URL's user info, a value after a secret's name, and a short bearer token. A test compares each with its rules on the previous release, on generated input. The updater is one standalone file, so it carries a generated copy of the redactor, and a test holds that copy to the module.
+- **The dashboard's service log** redacts its journal entries and log-file lines as one text, with 100 more above the ones shown, and redacts a long line before cutting it to 2,000 characters. A PR section, a change's diff and a chat attachment are redacted before they are cut, too. Each of these reads only what it keeps plus a margin, so a huge input costs no more than its window. Each window ends at its last whitespace, so a token cut by the window's end is never shown. A text the redactor can't scan (a token prefix followed by a run of millions of characters) becomes `[REDACTED: too large to scan]`, and a chat attachment like that is refused as "text too large to scan" until it is attached anyway. The URL-password rule no longer slows down on a long dotted line.
+- A Tailscale key in a Tailscale step line now shows as `tskey-[REDACTED]` (was `[a key]`); the login link is still taken out first, so a token glued before it can't hide it. A test fails on any module that redacts a token shape with its own pattern.

scripts/bench/cli.ts

@@ -40,6 +40,7 @@
 import { realpathSync } from 'node:fs';
 import { userInfo } from 'node:os';
 import { fileURLToPath } from 'node:url';
+import { redactString } from '../../src/events/redact.js';
 import { currentUser } from '../../src/os/index.js';
 import { benchLane, runtimeDir } from './agent.js';
 import { LOGIN_SWITCH_VAR, requireBenchUser, requireClaudePin, requireSubscription } from './auth.js';
@@ -88,10 +89,11 @@ export interface Check {
 
 /** A variable whose value may be a credential (container.ts SECRET_NAME without the daemon's names, which are paths). */
 const SECRET_VAR = /KEY|TOKEN|SECRET|PASSW|CREDENTIAL|AUTH|COOKIE|SESSION/i;
-/** Token shapes: Anthropic keys and OAuth tokens, GitHub tokens, Slack tokens, AWS key ids, JWTs. */
-const TOKEN_SHAPE = /sk-ant-[\w-]+|\bgh[opsur]_\w{16,}|\bgithub_pat_\w{16,}|\bxox[abprs]-[\w-]+|\bAKIA[0-9A-Z]{16}\b|\beyJ[\w-]{8,}\.[\w-]{8,}\.[\w-]+/g;
-/** An email address: its domain ends in a letters-only label, so a package spec such as `name@0.0.79` is not one. */
-const EMAIL = /[\w.+-]+@[\w-]+(?:\.[\w-]+)*\.[A-Za-z]{2,}\b/g;
+/**
+ * An email address: its domain ends in a letters-only label, so a package spec such as `name@0.0.79` is not one. Read
+ * after the token masks, so a mask may stand in a label (`x@wld_[redacted].com`): the address is still found.
+ */
+const EMAIL = /(?:[\w.+-]|\[redacted\])+@(?:[\w-]|\[redacted\])+(?:\.(?:[\w-]|\[redacted\])+)*\.[A-Za-z]{2,}\b/g;
 /** What scrub() keeps of one detail, and of one printed line. */
 export const DETAIL_MAX = 400;
 const LINE_MAX = 1000;
@@ -105,7 +107,8 @@ const PATH_VALUE = /^(\/|unix:)/;
  * or switch the login (PIECE_MIN characters or more), and every piece of such a value of PIECE_MIN or more that starts
  * or ends it (#851 round 2: a message cut short upstream, as the driver's checks cut a child's stderr, ends in a
  * prefix of the value; this redacts it whatever cut it); every token-shaped string. Email addresses become [email];
- * control and format characters (an ANSI escape's ESC, a bell, a bidi override) a space. Cut to at most `max`
+ * every control character (an ANSI escape's ESC, a bell, a line break) a space; a format character (a zero-width
+ * space, a bidi override) is dropped, before the token shapes are looked for. Cut to at most `max`
  * characters only after all of that.
  *
  * INVARIANT (#884 round 2): first of all, every word shaped like a task is withheld (withholdTasks), on the text as it
@@ -126,7 +129,11 @@ export function scrub(text: string, env: NodeJS.ProcessEnv, max = DETAIL_MAX): s
       for (const piece of [v.slice(0, n), v.slice(v.length - n)]) if (s.includes(piece)) s = s.split(piece).join('[redacted]');
     }
   }
-  s = s.replace(TOKEN_SHAPE, '[redacted]').replace(EMAIL, '[email]').replace(/[\p{Cc}\p{Cf}]/gu, ' ');
+  // A format character (a zero-width space, a bidi override) is dropped before the token shapes are looked for, so it
+  // can't split one (#889 round 3). Then the shared redactor, its broad rules too (a token cut short, a value after a
+  // secret's name); it finds a token right after an ANSI escape itself (its glue points). Then emails, as before, and
+  // every control character becomes a space.
+  s = redactString(s.replace(/\p{Cf}/gu, ''), { broad: true, mask: '[redacted]' }).replace(EMAIL, '[email]').replace(/\p{Cc}/gu, ' ');
   return s.length > max ? `${s.slice(0, max - 3)}...` : s;
 }
 

scripts/gen-redact-vendor.mjs

@@ -0,0 +1,24 @@
+#!/usr/bin/env node
+// Writes the root updater's generated copy of the shared redactor (scripts/lib/redact-vendor.mjs says why).
+// Usage: node scripts/gen-redact-vendor.mjs [--write]   (without --write: exit 1 when the copy is out of date)
+import { readdirSync, readFileSync, writeFileSync } from 'node:fs';
+import { dirname, join } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { blockIn, vendoredBlock, withBlock } from './lib/redact-vendor.mjs';
+
+const root = join(dirname(fileURLToPath(import.meta.url)), '..');
+const machine = join(root, 'scripts', 'machine');
+const updaters = readdirSync(machine).filter((f) => f.endsWith('-update.cjs'));
+if (updaters.length !== 1) throw new Error(`expected one scripts/machine/*-update.cjs, found ${updaters.length}`);
+const file = join(machine, updaters[0]);
+const text = readFileSync(file, 'utf8');
+const block = vendoredBlock(readFileSync(join(root, 'src', 'events', 'redact.ts'), 'utf8'));
+if (blockIn(text) === block) {
+  console.log(`${updaters[0]}: the redactor copy is current`);
+} else if (process.argv.includes('--write')) {
+  writeFileSync(file, withBlock(text, block));
+  console.log(`${updaters[0]}: the redactor copy written`);
+} else {
+  console.error(`${updaters[0]}: the redactor copy is out of date: run node scripts/gen-redact-vendor.mjs --write`);
+  process.exit(1);
+}

scripts/lib/redact-vendor.mjs

@@ -0,0 +1,56 @@
+// The root updater (scripts/machine/<cli>-update.cjs) is one standalone CommonJS file: it is installed alone, as root,
+// and loads no engine module. So it carries the shared redactor (src/events/redact.ts) as a generated copy, between
+// the two marker lines below: that module transpiled to CommonJS, unchanged. scripts/gen-redact-vendor.mjs writes it;
+// test/redact-vendored.test.ts fails when the copy is not exactly what this makes of the module (#889).
+import ts from 'typescript';
+
+export const BEGIN = '// BEGIN generated from src/events/redact.ts: do not edit (node scripts/gen-redact-vendor.mjs --write)';
+export const END = '// END generated from src/events/redact.ts';
+
+const MARK = '// check-table: ';
+/** The table marks in `source` (#886): each mark line with the name of the table declared on the next line. */
+function marksIn(source) {
+  const lines = source.split('\n');
+  const out = [];
+  for (let k = 0; k + 1 < lines.length; k++) {
+    const t = lines[k].trim();
+    const name = /^(?:export\s+)?(?:const|let|var)\s+([A-Za-z_]\w*)/.exec(lines[k + 1].trimStart())?.[1];
+    if (t.startsWith(MARK) && name) out.push([t, name]);
+  }
+  return out;
+}
+
+/**
+ * The block for `source` (src/events/redact.ts's text): the module, transpiled, as the object `sharedRedact`. The
+ * transpile drops comments, so each table mark in the module is put back before the same table in the copy: the copy's
+ * tables are held as the module's are (#889 round 7).
+ */
+export function vendoredBlock(source) {
+  const out = ts.transpileModule(source, {
+    compilerOptions: { module: ts.ModuleKind.CommonJS, target: ts.ScriptTarget.ES2022, removeComments: true },
+    fileName: 'redact.ts',
+  });
+  const lines = out.outputText.trimEnd().split('\n');
+  for (const [mark, name] of marksIn(source)) {
+    const at = lines.findIndex((l) => new RegExp(`^(?:export\\s+)?(?:const|let|var)\\s+${name}(?![\\w$])`).test(l));
+    if (at === -1) throw new Error(`the copy has no table ${name} to mark`);
+    lines.splice(at, 0, mark);
+  }
+  // Lines are kept as transpiled (no indent added): a template literal over two lines must not change.
+  return [BEGIN, 'const sharedRedact = (() => {', 'const exports = {};', lines.join('\n'), 'return exports;', '})();', END].join('\n');
+}
+
+/** The block in `text` (the updater's source), markers included; null when its markers are missing or out of order. */
+export function blockIn(text) {
+  const a = text.indexOf(BEGIN);
+  const b = text.indexOf(END);
+  if (a === -1 || b === -1 || b < a || text.indexOf(BEGIN, a + 1) !== -1) return null;
+  return text.slice(a, b + END.length);
+}
+
+/** `text` with its block replaced by `block`. Throws when the markers are missing. */
+export function withBlock(text, block) {
+  const old = blockIn(text);
+  if (old === null) throw new Error('the updater has no generated redactor block (its BEGIN and END marker lines)');
+  return text.replace(old, () => block);
+}