From 8fcc291a31023edd7a170e38445afbd1952b119c Mon Sep 17 00:00:00 2001 From: Brad Huffman Date: Wed, 22 Jul 2026 11:34:20 -0500 Subject: [PATCH 1/5] feat: recursive documentation for monorepos (#162) Adds a recursive documentation strategy so large monorepos can be documented as a tree of wikis: each subproject gets its own nested `/openwiki/` sub-wiki documenting its subtree in depth, while the repository-root wiki links down to them and covers only cross-cutting concerns. Closes the request in langchain-ai/openwiki#162. How it works: - Subprojects come from an `openwiki/workspaces.json` manifest, or from `--recursive` auto-detection of common workspace layouts (pnpm/npm/yarn, Cargo, Go, uv, Gradle, Maven, .NET solutions, Bazel), which writes a manifest for review then proceeds. Manifest presence auto-enables recursion; `--recursive=false` forces a single run. - An orchestrator runs the existing per-repo core once per subproject (backend rooted at the subproject so the docs-only write guard, snapshot, metadata, plan cleanup, and index-sync all scope for free), then writes a generated `openwiki/workspaces.md` aggregation index, then the root run last so its index-sync links the aggregation in. One model is resolved and reused across all runs; runs are sequential. - Git evidence and the incremental no-op check are scoped per subproject (subproject-relative diff), so `--update` regenerates only subprojects whose own subtree changed and skips the rest without a model call. Root git evidence excludes nested `**/openwiki`. - .NET solution detection coarsens per-project paths to product-area roots (`area/src/X`, `area/tests/X` -> `area`) to avoid one wiki per .csproj. - Backward compatible: with no manifest and no flag, behavior is unchanged. Documents the current limitation that updates do not cascade across subprojects (a shared-dependency change refreshes only that sub-wiki and the root, not its dependents) in README.md and the orchestrator JSDoc. Co-Authored-By: Claude --- README.md | 48 + src/agent/index.ts | 208 +++-- src/agent/prompt.ts | 59 +- src/agent/types.ts | 16 + src/agent/utils.ts | 92 +- src/cli.tsx | 132 ++- src/code-mode.ts | 36 +- src/commands.ts | 40 + src/monorepo/orchestrator.ts | 291 +++++++ src/monorepo/workspaces.ts | 1256 +++++++++++++++++++++++++++ src/okf/index-sync.ts | 2 + test/code-mode.test.ts | 28 + test/commands.test.ts | 50 ++ test/docs-only-backend.test.ts | 27 + test/orchestrator.test.ts | 288 ++++++ test/prompt.test.ts | 62 +- test/recursion-activation.test.ts | 124 +++ test/workspace-git-scope.test.ts | 292 +++++++ test/workspaces-aggregation.test.ts | 69 ++ test/workspaces-state.test.ts | 106 +++ test/workspaces.test.ts | 1041 ++++++++++++++++++++++ 21 files changed, 4152 insertions(+), 115 deletions(-) create mode 100644 src/monorepo/orchestrator.ts create mode 100644 src/monorepo/workspaces.ts create mode 100644 test/orchestrator.test.ts create mode 100644 test/recursion-activation.test.ts create mode 100644 test/workspace-git-scope.test.ts create mode 100644 test/workspaces-aggregation.test.ts create mode 100644 test/workspaces-state.test.ts create mode 100644 test/workspaces.test.ts diff --git a/README.md b/README.md index 1ccc96d0..810b5813 100644 --- a/README.md +++ b/README.md @@ -179,6 +179,54 @@ repository wiki: OpenWiki reads it for scope and priorities, but it is not generated documentation and is not rewritten during normal init, update, or chat runs unless you explicitly ask to change the brief. +## Monorepos (recursive documentation) + +Large monorepos are hard to capture in a single wiki. OpenWiki can instead +document each subproject in its own nested `openwiki/` sub-wiki, while the +repository-root wiki links down to them and covers only cross-cutting concerns: + +```sh +openwiki code --update --recursive +``` + +Subprojects are read from an `openwiki/workspaces.json` manifest at the +repository root: + +```json +{ + "version": 1, + "workspaces": [ + { "path": "packages/api", "name": "API", "goal": "Optional per-subproject brief." }, + { "path": "packages/web" } + ], + "root": { "goal": "Optional brief for the aggregating root wiki." } +} +``` + +- **Activation.** If `openwiki/workspaces.json` is present, recursive mode is + enabled automatically. Passing `--recursive` with no manifest triggers + auto-detection of common workspace layouts (pnpm/npm/yarn workspaces, Cargo, + Go, uv, Gradle, Maven, .NET solutions, Bazel), writes a `workspaces.json` for + you to review, and proceeds. Pass `--recursive=false` to force a single-repo + run even when a manifest exists. +- **Layout.** Each subproject's docs live at `/openwiki/`. The root + wiki gets a generated `openwiki/workspaces.md` index linking to every + sub-wiki; do not hand-edit it. +- **Incremental updates.** Each subproject is evaluated independently: on + `--update`, a subproject's sub-wiki regenerates only when files *within that + subproject's own subtree* have changed since it was last documented (tracked in + its own `openwiki/.last-update.json`). Unchanged subprojects are skipped + without a model call, so scheduled refreshes stay cheap. The root wiki + regenerates on every run. +- **Known limitation — no dependency cascade.** Updates do not propagate across + subprojects. If a shared subproject (for example a common kernel or contracts + package) changes, only *that* sub-wiki and the root wiki are refreshed — the + sibling subprojects that depend on it are **not** automatically regenerated, + even though their documented context may reference the changed code. Until + dependency-aware invalidation exists, force a full refresh after a significant + shared-code change by removing the relevant `openwiki/.last-update.json` files + (or running each affected subproject explicitly). + On the first interactive run, OpenWiki will have you configure your inference provider, API key, and LLM. You will also be able to set a LangSmith API key to trace your OpenWiki runs to a LangSmith tracing project named "openwiki" (optional). These configuration options and secrets will be saved to `~/.openwiki/.env` on your local machine. diff --git a/src/agent/index.ts b/src/agent/index.ts index d96a00c9..170712b8 100644 --- a/src/agent/index.ts +++ b/src/agent/index.ts @@ -95,32 +95,91 @@ import { persistRunMetadataIfChanged, removeTemporaryPlanFile, shouldCheckUpdateNoop, + type GitScope, } from "./utils.js"; import { classifyError, recordRunSafe } from "../telemetry/index.js"; -export async function runOpenWikiAgent( - command: OpenWikiCommand, - cwd = openWikiLocalWikiDir, - options: OpenWikiRunOptions = {}, -): Promise { - const runtimeCwd = options.outputMode ? cwd : openWikiLocalWikiDir; +/** + * A model resolved once per process. The orchestrator resolves this a single + * time and reuses it across every subproject + root run so all runs share one + * provider/model/credential validation (and a single ChatGPT token refresh + * happens separately, before the loop). + */ +export type ResolvedRunModel = { + provider: OpenWikiProvider; + modelId: string; + providerRetryAttempts: number; +}; - emitDebug(options, `command=${command}`); - emitDebug(options, `cwd=${runtimeCwd}`); - emitDebug( - options, - `userMessage=${options.userMessage ? "provided" : "not-provided"}`, - ); - emitDebug(options, `userMessage.followup=${options.isFollowup === true}`); - emitDebug(options, `env.beforeLoad ${formatEnvironmentDebug()}`); +/** + * Validates provider credentials/base URL/secret/region and resolves the model + * ID + retry attempts. Extracted so a single resolution can be shared by both + * the single-run path and the recursive orchestrator. Does NOT perform the + * ChatGPT token refresh (that is async + env-mutating and must run exactly once + * before any run); callers that use ChatGPT must call + * {@link refreshChatGptTokensIfNeeded} once before building models. + */ +export function resolveRunModel(options: OpenWikiRunOptions): ResolvedRunModel { + const provider = resolveConfiguredProvider(); + const providerBaseUrl = resolveProviderBaseUrl(provider); + emitDebug(options, `provider=${provider}`); + if (providerBaseUrl) { + emitDebug(options, `provider.baseUrl=${JSON.stringify(providerBaseUrl)}`); + } + ensureProviderCredentials(provider); + emitDebug(options, `credentials=${provider} present`); + ensureProviderBaseUrl(provider); + ensureProviderSecretKey(provider); + ensureProviderRegion(provider); - await loadOpenWikiEnv(); - await syncBundledSkills(); - emitDebug(options, "env=loaded ~/.openwiki/.env"); - emitDebug(options, `env.afterLoad ${formatEnvironmentDebug()}`); + const modelId = resolveModelId(options, provider); + emitDebug(options, `model=${modelId}`); + const providerRetryAttempts = resolveProviderRetryAttempts(); + emitDebug(options, `provider.retryAttempts=${providerRetryAttempts}`); + + return { provider, modelId, providerRetryAttempts }; +} + +/** + * Refreshes ChatGPT OAuth tokens once when the configured provider needs them. + * Must be called exactly once before any run in a process (it mutates env). + */ +export async function refreshChatGptTokensIfNeeded( + provider: OpenWikiProvider, + options: OpenWikiRunOptions, +): Promise { + if (provider !== "openai-chatgpt") { + return; + } + // Refresh before the model is built, so `createModel` stays synchronous. + await ensureFreshChatGptTokens(); + emitDebug(options, "chatgpt.token=fresh"); +} + +/** + * Runs one OpenWiki agent invocation with all per-run concerns that live + * OUTSIDE the core: the update no-op short-circuit, telemetry recording, the + * OpenRouter debug-fetch install/restore, and error debug-info attachment. + * + * Both {@link runOpenWikiAgent} and the recursive orchestrator drive their runs + * through this wrapper so a subproject run behaves identically to a top-level + * run. The caller is responsible for the once-per-process work + * ({@link loadOpenWikiEnv}, {@link syncBundledSkills}, {@link resolveRunModel}, + * {@link refreshChatGptTokensIfNeeded}). + * + * `scope` narrows the git evidence + no-op check to a subtree (see + * {@link GitScope}); omit it for byte-identical whole-repo behavior. + */ +export async function runOne( + command: OpenWikiCommand, + cwd: string, + options: OpenWikiRunOptions, + model: ResolvedRunModel, + scope?: GitScope, +): Promise { if (command === "update" && shouldCheckUpdateNoop(options)) { - const noopStatus = await getUpdateNoopStatus(cwd); + const noopStatus = await getUpdateNoopStatus(cwd, scope); if (noopStatus.shouldSkip) { const message = @@ -129,7 +188,7 @@ export async function runOpenWikiAgent( options.onEvent?.({ type: "text", text: message }); await recordRunSafe(command, options, { - provider: resolveConfiguredProvider(), + provider: model.provider, outcome: "noop", }); @@ -147,47 +206,19 @@ export async function runOpenWikiAgent( const debugFetchCapture = installOpenRouterDebugFetch(options); - // Resolved inside the try so a failure during resolution (missing key, - // invalid model, missing base URL) is still recorded. They may be undefined - // in the catch if resolution threw before assigning them. - let provider: OpenWikiProvider | undefined; - let modelId: string | undefined; - try { - provider = resolveConfiguredProvider(); - const providerBaseUrl = resolveProviderBaseUrl(provider); - emitDebug(options, `provider=${provider}`); - if (providerBaseUrl) { - emitDebug(options, `provider.baseUrl=${JSON.stringify(providerBaseUrl)}`); - } - ensureProviderCredentials(provider); - emitDebug(options, `credentials=${provider} present`); - ensureProviderBaseUrl(provider); - ensureProviderSecretKey(provider); - ensureProviderRegion(provider); - - if (provider === "openai-chatgpt") { - // Refresh before the model is built, so `createModel` stays synchronous. - await ensureFreshChatGptTokens(); - emitDebug(options, "chatgpt.token=fresh"); - } - - modelId = resolveModelId(options, provider); - emitDebug(options, `model=${modelId}`); - const providerRetryAttempts = resolveProviderRetryAttempts(); - emitDebug(options, `provider.retryAttempts=${providerRetryAttempts}`); - const result = await runOpenWikiAgentCore( command, - runtimeCwd, + cwd, options, - provider, - modelId, - providerRetryAttempts, + model.provider, + model.modelId, + model.providerRetryAttempts, + scope, ); await recordRunSafe(command, options, { - provider, + provider: model.provider, outcome: "success", }); @@ -196,7 +227,7 @@ export async function runOpenWikiAgent( attachOpenRouterDebugInfo(error, debugFetchCapture.getLastFailure()); await recordRunSafe(command, options, { - provider, + provider: model.provider, outcome: "failure", errorClass: classifyError(error), }); @@ -207,16 +238,76 @@ export async function runOpenWikiAgent( } } -async function runOpenWikiAgentCore( +export async function runOpenWikiAgent( + command: OpenWikiCommand, + cwd = openWikiLocalWikiDir, + options: OpenWikiRunOptions = {}, +): Promise { + const runtimeCwd = options.outputMode ? cwd : openWikiLocalWikiDir; + + emitDebug(options, `command=${command}`); + emitDebug(options, `cwd=${runtimeCwd}`); + emitDebug( + options, + `userMessage=${options.userMessage ? "provided" : "not-provided"}`, + ); + emitDebug(options, `userMessage.followup=${options.isFollowup === true}`); + emitDebug(options, `env.beforeLoad ${formatEnvironmentDebug()}`); + + await loadOpenWikiEnv(); + await syncBundledSkills(); + emitDebug(options, "env=loaded ~/.openwiki/.env"); + emitDebug(options, `env.afterLoad ${formatEnvironmentDebug()}`); + + // The no-op short-circuit lives in runOne, but it must key off runtimeCwd + // (which equals cwd for repository runs). runOne handles this via its cwd arg. + + // runOne records its own success/failure telemetry. Anything that throws + // BEFORE runOne is entered (credential/model resolution, ChatGPT token + // refresh) must still be recorded here exactly once, matching the pre-refactor + // behavior where every failure produced a telemetry record. + let provider: OpenWikiProvider | undefined; + let enteredRunOne = false; + + try { + const model = resolveRunModel(options); + provider = model.provider; + + await refreshChatGptTokensIfNeeded(model.provider, options); + + enteredRunOne = true; + return await runOne(command, runtimeCwd, options, model); + } catch (error) { + if (!enteredRunOne) { + await recordRunSafe(command, options, { + provider, + outcome: "failure", + errorClass: classifyError(error), + }); + } + + throw error; + } +} + +export async function runOpenWikiAgentCore( command: OpenWikiCommand, cwd: string, options: OpenWikiRunOptions, provider: OpenWikiProvider, modelId: string, providerRetryAttempts: number, + scope?: GitScope, ): Promise { const outputMode = options.outputMode ?? "local-wiki"; - const context = await createRunContext(command, cwd, outputMode); + const recursionRole = options.recursionRole; + const context = await createRunContext( + command, + cwd, + outputMode, + scope, + options.wikiGoalOverride, + ); emitDebug(options, "context=created"); const openWikiSnapshotBefore = command === "chat" @@ -263,7 +354,7 @@ async function runOpenWikiAgentCore( permissions: [ { operations: ["write"], paths: ["/skills/**"], mode: "deny" }, ], - systemPrompt: createSystemPrompt(command, outputMode), + systemPrompt: createSystemPrompt(command, outputMode, recursionRole), }); emitDebug(options, "agent=created"); @@ -406,6 +497,7 @@ ${createUserPrompt( context, options.userMessage ?? null, options.outputMode ?? "local-wiki", + options.recursionRole, )} ${formatRuntimeRootLabel(options.outputMode ?? "local-wiki")}: diff --git a/src/agent/prompt.ts b/src/agent/prompt.ts index 560ded1d..50d8135e 100644 --- a/src/agent/prompt.ts +++ b/src/agent/prompt.ts @@ -1,10 +1,38 @@ import { OpenWikiCommand, OpenWikiOutputMode, + RecursionRole, RunContext, UpdateMetadata, } from "./types.js"; +/** + * Extra guidance injected into the system prompt for one recursion role in a + * monorepo pass. Empty for ordinary single-repo runs. + */ +function recursionRoleSystemInstructions( + recursionRole: RecursionRole | undefined, +): string { + if (recursionRole === "subproject") { + return ` +Monorepo subproject scope: +- This run is scoped to ONE subproject of a larger monorepo. The virtual root / is THIS subproject's directory, and /openwiki is this subproject's own sub-wiki. +- Document only this subproject's subtree. Do not document, read into, or write to sibling subprojects or the repository root; the git evidence for this run is already scoped to this subtree. +- Assume the monorepo root wiki links DOWN to this sub-wiki. Write a self-contained sub-wiki for this subproject; you do not need to re-explain repository-wide concerns that belong in the root wiki.`.trim(); + } + + if (recursionRole === "root") { + return ` +Monorepo root scope: +- This is the monorepo ROOT run. Each subproject listed in openwiki/workspaces.json has its OWN detailed sub-wiki under /openwiki/. +- Do NOT deep-document the subprojects' internals here. Instead, link DOWN to each subproject's sub-wiki entrypoint (its openwiki/quickstart.md) and describe repository-wide concerns: the overall architecture, how subprojects fit together, shared tooling, and cross-cutting workflows. +- The file openwiki/workspaces.md is generated deterministically to aggregate links to the sub-wikis. Treat it as generated: link to it from openwiki/quickstart.md, but do not hand-maintain its per-subproject list. +- The git evidence for this run deliberately excludes the freshly generated nested openwiki/ sub-wikis so you are not tempted to re-document them.`.trim(); + } + + return ""; +} + function formatLastUpdate(lastUpdate: UpdateMetadata | null): string { if (lastUpdate === null) { return "No previous OpenWiki update metadata was found."; @@ -16,8 +44,13 @@ function formatLastUpdate(lastUpdate: UpdateMetadata | null): string { export function createSystemPrompt( command: OpenWikiCommand, outputMode: OpenWikiOutputMode = "local-wiki", + recursionRole?: RecursionRole, ): string { const output = getOutputPromptConfig(outputMode); + const recursionInstructions = recursionRoleSystemInstructions(recursionRole); + const recursionSection = recursionInstructions + ? `\n\n${recursionInstructions}` + : ""; return ` You are OpenWiki, an expert technical writer, software architect, and product analyst. @@ -196,7 +229,7 @@ Coverage self-check: - If an area is backlogged, include its area name, source anchor, and a one-line reason it was deferred. Mode-specific behavior: -${createModeInstructions(command, outputMode)} +${createModeInstructions(command, outputMode)}${recursionSection} `.trim(); } @@ -260,8 +293,10 @@ export function createUserPrompt( context: RunContext, userMessage: string | null = null, outputMode: OpenWikiOutputMode = "local-wiki", + recursionRole?: RecursionRole, ): string { const output = getOutputPromptConfig(outputMode); + const recursionReminder = recursionRoleUserReminder(recursionRole); if (command === "chat") { return userMessage?.trim() || "Start an OpenWiki chat."; @@ -280,7 +315,7 @@ Wiki brief: ${formatWikiGoal(context.wikiGoal)} Git context: -${context.gitSummary} +${context.gitSummary}${recursionReminder} `.trim(), userMessage, ); @@ -299,7 +334,7 @@ Wiki brief: ${formatWikiGoal(context.wikiGoal)} Git change summary: -${context.gitSummary} +${context.gitSummary}${recursionReminder} `.trim(), userMessage, ); @@ -309,6 +344,24 @@ function formatWikiGoal(wikiGoal: string | undefined): string { return wikiGoal?.trim() || "(not provided)"; } +/** + * A concise reminder appended to the init/update user prompt so the recursion + * role is reinforced next to the concrete task, not only in the system prompt. + */ +function recursionRoleUserReminder( + recursionRole: RecursionRole | undefined, +): string { + if (recursionRole === "subproject") { + return "\n\nRecursive monorepo run: you are documenting a single subproject. The virtual root / is this subproject. Document only this subtree and do not touch sibling subprojects or the repository root."; + } + + if (recursionRole === "root") { + return "\n\nRecursive monorepo run: this is the monorepo root. Link down to each subproject's own openwiki/quickstart.md (aggregated in the generated openwiki/workspaces.md) instead of deep-documenting subprojects."; + } + + return ""; +} + type OutputPromptConfig = { canonicalLocationInstruction: string; docsLocation: string; diff --git a/src/agent/types.ts b/src/agent/types.ts index a033057a..474c9528 100644 --- a/src/agent/types.ts +++ b/src/agent/types.ts @@ -1,6 +1,14 @@ export type OpenWikiCommand = "chat" | "init" | "update"; export type OpenWikiOutputMode = "local-wiki" | "repository"; +/** + * The role a run plays inside a recursive monorepo documentation pass. Absent + * for ordinary single-repo runs. `subproject` runs are rooted at a subproject + * and document only that subtree; the `root` run documents the monorepo root and + * links down to the per-subproject sub-wikis instead of deep-documenting them. + */ +export type RecursionRole = "subproject" | "root"; + export type OpenWikiRunResult = { command: OpenWikiCommand; model: string; @@ -37,9 +45,17 @@ export type OpenWikiRunOptions = { modelId?: string | null; onEvent?: (event: OpenWikiRunEvent) => void; outputMode?: OpenWikiOutputMode; + recursionRole?: RecursionRole; threadId?: string; userMessage?: string | null; telemetryFile?: string; + /** + * When set, overrides the wiki brief that would otherwise be read from the + * run root's openwiki/INSTRUCTIONS.md. Used by the recursive orchestrator to + * inject a manifest-supplied per-subproject or root goal (manifest goal takes + * precedence over a subproject's INSTRUCTIONS.md). + */ + wikiGoalOverride?: string; }; export type UpdateMetadata = { diff --git a/src/agent/utils.ts b/src/agent/utils.ts index c6fe42c1..abcc3d15 100644 --- a/src/agent/utils.ts +++ b/src/agent/utils.ts @@ -24,9 +24,57 @@ import type { Dirent } from "node:fs"; const execFileAsync = promisify(execFile); const LOCAL_WIKI_METADATA_PATH = ".last-update.json"; const TEMPORARY_PLAN_FILE = "_plan.md"; +/** + * Per-subproject gitHead tracking written by the recursive orchestrator at the + * root wiki. It records generator state, not documentation, so it is excluded + * from the content snapshot (so it never triggers a spurious metadata write) and + * from generated indexes. + */ +export const WORKSPACES_STATE_FILE = ".workspaces-state.json"; export type OpenWikiContentSnapshot = string; +/** + * Restricts the git evidence/no-op commands to a subtree of the repository. + * + * - `subproject`: cwd is the subproject directory and commands use a `-- .` + * pathspec plus `--relative` (on log/diff) so output paths are subproject- + * relative (`openwiki/x.md`, `src/code.ts`). The OpenWiki-prefix check then + * stays keyed off `OPEN_WIKI_DIR` unchanged. + * - `root-excluding-nested`: cwd is the repository root and commands exclude any + * nested `openwiki/` directories (so freshly generated sub-wikis are not + * presented as source churn), while the root's own `openwiki/` is preserved. + * + * When `scope` is undefined the git commands are byte-identical to the + * non-recursive behavior (no pathspec, no `--relative`). + */ +export type GitScope = + { mode: "subproject" } | { mode: "root-excluding-nested" }; + +/** + * Builds the `--relative` flag and trailing pathspec args for a scoped git + * command. `useRelative` is only honored for the subproject scope (status stays + * cwd-relative and does not take `--relative`). + */ +function gitScopeArgs( + scope: GitScope | undefined, + useRelative: boolean, +): { relative: string[]; pathspec: string[] } { + if (!scope) { + return { relative: [], pathspec: [] }; + } + + if (scope.mode === "subproject") { + return { + relative: useRelative ? ["--relative"] : [], + pathspec: ["--", "."], + }; + } + + // Root scope: keep the root wiki but drop any nested openwiki subtrees. + return { relative: [], pathspec: ["--", ".", ":(exclude)**/openwiki/**"] }; +} + export type UpdateNoopStatus = | { shouldSkip: true; @@ -45,9 +93,12 @@ export async function createRunContext( command: OpenWikiCommand, cwd: string, outputMode: OpenWikiOutputMode = "repository", + scope?: GitScope, + wikiGoalOverride?: string, ): Promise { const lastUpdate = await readLastUpdate(cwd, outputMode); - const wikiGoal = await readRunWikiGoal(cwd, outputMode); + const wikiGoal = + wikiGoalOverride?.trim() || (await readRunWikiGoal(cwd, outputMode)); if (command === "chat") { return { @@ -68,7 +119,7 @@ export async function createRunContext( return { lastUpdate, - gitSummary: await createGitSummary(command, cwd, lastUpdate), + gitSummary: await createGitSummary(command, cwd, lastUpdate, scope), wikiGoal, }; } @@ -86,6 +137,7 @@ async function readRunWikiGoal( export async function getUpdateNoopStatus( cwd: string, + scope?: GitScope, ): Promise { const lastUpdate = await readLastUpdate(cwd, "repository"); @@ -99,10 +151,12 @@ export async function getUpdateNoopStatus( return { shouldSkip: false, reason: "missing current git head" }; } + const { pathspec } = gitScopeArgs(scope, false); const status = await runGit(cwd, [ "status", "--short", "--untracked-files=all", + ...pathspec, ]); const meaningfulStatus = status .split("\n") @@ -118,6 +172,7 @@ export async function getUpdateNoopStatus( const committedPaths = await getChangedPathsSinceLastUpdate( cwd, lastUpdate.gitHead, + scope, ); if ( @@ -346,7 +401,8 @@ function isIgnoredSnapshotPath(relativePath: string): boolean { return ( relativePath === path.basename(UPDATE_METADATA_PATH) || relativePath === LOCAL_WIKI_METADATA_PATH || - relativePath === TEMPORARY_PLAN_FILE + relativePath === TEMPORARY_PLAN_FILE || + relativePath === WORKSPACES_STATE_FILE ); } @@ -372,9 +428,13 @@ async function createGitSummary( command: OpenWikiCommand, cwd: string, lastUpdate: UpdateMetadata | null, + scope?: GitScope, ): Promise { const sections: string[] = []; - const status = await runGit(cwd, ["status", "--short"]); + // status stays cwd-relative and does not take `--relative`; log/diff take it. + const { pathspec } = gitScopeArgs(scope, false); + const { relative, pathspec: relativePathspec } = gitScopeArgs(scope, true); + const status = await runGit(cwd, ["status", "--short", ...pathspec]); const head = await getGitHead(cwd); sections.push(formatGitSection("git status --short", status)); @@ -386,6 +446,8 @@ async function createGitSummary( `${lastUpdate.gitHead}..HEAD`, "--name-status", "--oneline", + ...relative, + ...relativePathspec, ]); sections.push( @@ -401,6 +463,8 @@ async function createGitSummary( lastUpdate.updatedAt, "--name-status", "--oneline", + ...relative, + ...relativePathspec, ]); sections.push( @@ -415,6 +479,8 @@ async function createGitSummary( "--max-count=20", "--name-status", "--oneline", + ...relative, + ...relativePathspec, ]); if (command === "update") { @@ -429,7 +495,13 @@ async function createGitSummary( ); } - const diff = await runGit(cwd, ["diff", "--name-status", "HEAD"]); + const diff = await runGit(cwd, [ + "diff", + "--name-status", + ...relative, + "HEAD", + ...relativePathspec, + ]); sections.push(formatGitSection("git diff --name-status HEAD", diff)); return sections.join("\n\n"); @@ -487,8 +559,16 @@ function isUpdateMetadataStatusLine(line: string): boolean { async function getChangedPathsSinceLastUpdate( cwd: string, gitHead: string, + scope?: GitScope, ): Promise { - const diff = await runGit(cwd, ["diff", "--name-only", `${gitHead}..HEAD`]); + const { relative, pathspec } = gitScopeArgs(scope, true); + const diff = await runGit(cwd, [ + "diff", + "--name-only", + ...relative, + `${gitHead}..HEAD`, + ...pathspec, + ]); return diff .split("\n") diff --git a/src/cli.tsx b/src/cli.tsx index e47a6553..d2e91758 100644 --- a/src/cli.tsx +++ b/src/cli.tsx @@ -34,6 +34,11 @@ import { type CredentialDiagnostic, } from "./env.js"; import { createOpenWikiThreadId, runOpenWikiAgent } from "./agent/index.js"; +import { runRecursiveOpenWiki } from "./monorepo/orchestrator.js"; +import { + resolveRecursionActivation, + type RecursionActivation, +} from "./monorepo/workspaces.js"; import { formatChatGptAccountFromEnv } from "./agent/openai-chatgpt-oauth.js"; import { getErrorMessage, @@ -577,42 +582,66 @@ function App({ command }: AppProps) { }); } - const setupPromise = - runMode === "code" - ? ensureCodeModeRepoSetup(runtimeCwd) - : Promise.resolve(); - - setupPromise - .then(() => - runOpenWikiAgent(resolvedCommand, runtimeCwd, { - debug: isDebugMode(), - isFollowup: activeMessageIsFollowup, - modelId: sessionModelId, - outputMode: runtimeOutputMode, - threadId: sessionThreadId.current, - userMessage: activeUserMessage, - telemetryFile: command.telemetryFile ?? undefined, - onEvent: (event) => { - if (!mountedRef.current || activeRunId.current !== runId) { - return; - } + const runOptions = { + debug: isDebugMode(), + isFollowup: activeMessageIsFollowup, + modelId: sessionModelId, + outputMode: runtimeOutputMode, + threadId: sessionThreadId.current, + userMessage: activeUserMessage, + telemetryFile: command.telemetryFile ?? undefined, + onEvent: (event: OpenWikiRunEvent) => { + if (!mountedRef.current || activeRunId.current !== runId) { + return; + } - activeRunLog.current = appendRunLogEvent( - activeRunLog.current, - event, - nextLogId, - ); - setRunState((currentState) => - currentState.status === "running" - ? { - ...currentState, - log: activeRunLog.current, - } - : currentState, - ); - }, - }), - ) + activeRunLog.current = appendRunLogEvent( + activeRunLog.current, + event, + nextLogId, + ); + setRunState((currentState) => + currentState.status === "running" + ? { + ...currentState, + log: activeRunLog.current, + } + : currentState, + ); + }, + }; + + // Recursive monorepo docs only applies to code-mode init/update runs. The + // orchestrator runs subprojects then the root sequentially, emitting a + // per-subproject boundary text event so the Ink log shows coherent progress + // across the N runs. Chat and personal runs always take the single path. + const recursiveRunCommand = resolvedCommand; + const activationPromise: Promise = + runMode === "code" && recursiveRunCommand !== "chat" + ? resolveRecursionActivation(runtimeCwd, command.recursive) + : Promise.resolve({ + kind: "plain", + reason: "not a code init/update run", + }); + + activationPromise + .then(async (activation) => { + if (activation.kind === "recurse") { + const recursiveResult = await runRecursiveOpenWiki( + recursiveRunCommand, + runtimeCwd, + runOptions, + activation.manifest, + ); + return recursiveResult.rootResult; + } + + if (runMode === "code") { + await ensureCodeModeRepoSetup(runtimeCwd); + } + + return runOpenWikiAgent(recursiveRunCommand, runtimeCwd, runOptions); + }) .then((result) => { if (!mountedRef.current || activeRunId.current !== runId) { return; @@ -4086,11 +4115,7 @@ async function runPrintCommand( const runtimeCwd = getRunModeCwd(command.mode); const runtimeOutputMode = getRunModeOutputMode(command.mode); - if (command.mode === "code") { - await ensureCodeModeRepoSetup(runtimeCwd); - } - - await runOpenWikiAgent(command.command, runtimeCwd, { + const runOptions = { debug: isDebugMode(), isFollowup: command.command === "chat", modelId: command.modelId, @@ -4098,12 +4123,35 @@ async function runPrintCommand( threadId: createOpenWikiThreadId(runtimeCwd), userMessage: command.userMessage, telemetryFile: command.telemetryFile ?? undefined, - onEvent: (event) => { + onEvent: (event: OpenWikiRunEvent) => { if (event.type === "text" && event.source !== "subgraph") { output.push(event.text); } }, - }); + }; + + // Recursive monorepo docs: the orchestrator handles ensureCodeModeRepoSetup + // once and delimits each subproject in the streamed output. It only applies + // to code (repository) mode and to init/update runs. + const activation = + command.mode === "code" && command.command !== "chat" + ? await resolveRecursionActivation(runtimeCwd, command.recursive) + : ({ kind: "plain", reason: "not a code init/update run" } as const); + + if (activation.kind === "recurse") { + await runRecursiveOpenWiki( + command.command, + runtimeCwd, + runOptions, + activation.manifest, + ); + } else { + if (command.mode === "code") { + await ensureCodeModeRepoSetup(runtimeCwd); + } + + await runOpenWikiAgent(command.command, runtimeCwd, runOptions); + } const text = output.join("").trim(); diff --git a/src/code-mode.ts b/src/code-mode.ts index f3447c5e..e3ff8be6 100644 --- a/src/code-mode.ts +++ b/src/code-mode.ts @@ -10,17 +10,32 @@ const DEFAULT_CODE_MODE_CRON = "0 8 * * *"; // Each is created when missing and refreshed in place when already present. const CODE_MODE_AGENT_FILES = ["AGENTS.md", "CLAUDE.md"]; +/** + * Options controlling generated code-mode scaffolding. + */ +export type CodeModeRepoSetupOptions = { + cronExpression?: string; + /** + * When true, the generated GitHub Actions workflow runs + * `openwiki code --update --recursive --print` so scheduled refreshes keep the + * monorepo's per-subproject sub-wikis up to date. + */ + recursive?: boolean; +}; + export async function ensureCodeModeRepoSetup( cwd: string, - cronExpression = DEFAULT_CODE_MODE_CRON, + options: CodeModeRepoSetupOptions = {}, ): Promise { - await writeCodeModeWorkflow(cwd, cronExpression); + const cronExpression = options.cronExpression ?? DEFAULT_CODE_MODE_CRON; + await writeCodeModeWorkflow(cwd, cronExpression, options.recursive === true); await writeCodeModeAgentSnippets(cwd); } async function writeCodeModeWorkflow( cwd: string, cronExpression: string, + recursive: boolean, ): Promise { const workflowPath = path.join( cwd, @@ -29,7 +44,11 @@ async function writeCodeModeWorkflow( "openwiki-update.yml", ); await mkdir(path.dirname(workflowPath), { recursive: true }); - await writeFile(workflowPath, createCodeModeWorkflow(cronExpression), "utf8"); + await writeFile( + workflowPath, + createCodeModeWorkflow(cronExpression, recursive), + "utf8", + ); } async function writeCodeModeAgentSnippets(cwd: string): Promise { @@ -66,7 +85,13 @@ async function writeCodeModeAgentSnippet( await writeFile(agentsPath, nextContent, "utf8"); } -function createCodeModeWorkflow(cronExpression: string): string { +function createCodeModeWorkflow( + cronExpression: string, + recursive: boolean, +): string { + const updateCommand = recursive + ? "openwiki code --update --recursive --print" + : "openwiki code --update --print"; return `name: OpenWiki Update on: @@ -94,7 +119,7 @@ jobs: run: npm install --global openwiki - name: Run OpenWiki - run: openwiki code --update --print + run: ${updateCommand} env: OPENWIKI_PROVIDER: openrouter OPENROUTER_API_KEY: \${{ secrets.OPENROUTER_API_KEY }} @@ -108,6 +133,7 @@ jobs: with: add-paths: | openwiki + **/openwiki AGENTS.md CLAUDE.md .github/workflows/openwiki-update.yml diff --git a/src/commands.ts b/src/commands.ts index eaf84c4e..9798a798 100644 --- a/src/commands.ts +++ b/src/commands.ts @@ -62,6 +62,13 @@ export type CliCommand = modeSource: OpenWikiRunModeSource; modelId: string | null; print: boolean; + /** + * Recursive monorepo docs activation. `undefined` = default (auto-enable + * only when openwiki/workspaces.json exists); `true` = force on (auto- + * detect workspaces and write a manifest when none exists); `false` = + * force off even when a manifest exists. + */ + recursive: boolean | undefined; shouldStart: boolean; userMessage: string | null; telemetryFile: string | null; @@ -339,6 +346,7 @@ function parseRunCommand( let modeSource = initialModeSource; let modelId: string | null = null; let print = false; + let recursive: boolean | undefined; let command: OpenWikiCommand = "chat"; let telemetryFile: string | null = null; @@ -369,6 +377,21 @@ function parseRunCommand( continue; } + if (arg === "--recursive") { + recursive = true; + continue; + } + + if (arg === "--no-recursive" || arg === "--recursive=false") { + recursive = false; + continue; + } + + if (arg === "--recursive=true") { + recursive = true; + continue; + } + if (arg === "--debug") { // isDebugMode() reads OPENWIKI_DEBUG; setting it at parse time is the // least-invasive way to opt into full credential/error diagnostics. @@ -556,6 +579,17 @@ function parseRunCommand( }; } + // Recursion is a monorepo (code-mode) concept. The personal brain has no + // repository subprojects, so reject the flag there rather than silently + // ignoring it. + if (recursive !== undefined && mode === "personal") { + return { + kind: "error", + exitCode: 1, + message: "--recursive is only supported in code mode, not personal mode.", + }; + } + return { kind: "run", exitCode: 0, @@ -565,6 +599,7 @@ function parseRunCommand( modeSource, modelId, print, + recursive, shouldStart, userMessage, telemetryFile, @@ -734,6 +769,11 @@ export const helpContent: HelpContent = { label: "-p, --print", description: "Run once and print the final assistant output.", }, + { + label: "--recursive[=false]", + description: + "Recursive monorepo docs: give each subproject its own openwiki/ sub-wiki and link down from the root. Auto-enabled when openwiki/workspaces.json exists; --recursive auto-detects and writes one; --recursive=false forces off (code mode only).", + }, { label: "--debug", description: diff --git a/src/monorepo/orchestrator.ts b/src/monorepo/orchestrator.ts new file mode 100644 index 00000000..9303b4c9 --- /dev/null +++ b/src/monorepo/orchestrator.ts @@ -0,0 +1,291 @@ +import { execFile } from "node:child_process"; +import { mkdir, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { promisify } from "node:util"; +import { + refreshChatGptTokensIfNeeded, + resolveRunModel, + runOne, +} from "../agent/index.js"; +import { loadOpenWikiEnv } from "../env.js"; +import { syncBundledSkills } from "../agent/skills.js"; +import { ensureCodeModeRepoSetup } from "../code-mode.js"; +import { readRepositoryWikiInstructions } from "../onboarding.js"; +import type { + OpenWikiCommand, + OpenWikiRunOptions, + OpenWikiRunResult, +} from "../agent/types.js"; +import { + getWorkspaceSkipReason, + readWorkspacesState, + resolveWorkspaceRuns, + writeWorkspacesState, + type ResolvedWorkspacePlan, + type ResolvedWorkspaceRun, + type WorkspaceManifest, + type WorkspacesState, +} from "./workspaces.js"; + +const execFileAsync = promisify(execFile); + +/** + * Result of a full recursive run: one entry per subproject that actually ran + * (or was skipped as a no-op), the root run result, the paths skipped because + * they had no documentable evidence, and the paths whose run threw. + */ +export interface RecursiveRunResult { + subprojectResults: OpenWikiRunResult[]; + rootResult: OpenWikiRunResult; + skippedWorkspaces: { path: string; reason: string }[]; + failedWorkspaces: { path: string; error: string }[]; +} + +/** + * Runs OpenWiki recursively across a monorepo: one run per subproject (rooted at + * that subproject, scoped git evidence, subproject prompt role), then the + * generated aggregation page, then the root run last (so the root's index-sync + * middleware picks up openwiki/workspaces.md), reusing ONE resolved model and a + * single ChatGPT token refresh across every run. + * + * Runs are SEQUENTIAL by design: they share one bundled-skills directory and one + * resolved model, and the root run must observe the completed sub-wikis. + * + * A subproject run that throws is RESILIENT: the failure is collected in + * `failedWorkspaces` and the pass continues, so one broken subproject does not + * abandon the sub-wikis already generated or block the aggregation + root run + * for the subprojects that succeeded. The root run still executes; the caller + * decides how to surface `failedWorkspaces`. + * + * Update model: each subproject is evaluated INDEPENDENTLY. Its no-op check + * (inside runOne, via getUpdateNoopStatus) diffs only that subproject's own + * subtree against the gitHead in its own openwiki/.last-update.json, so a + * subproject regenerates only when files under its path changed, and the root + * run always executes. There is NO dependency cascade: a change to a shared + * subproject refreshes only that sub-wiki (and the root), not the sibling + * subprojects that depend on it. Dependency-aware invalidation is intentionally + * out of scope here; see the "no dependency cascade" note in README.md. + * + * Falls back to a single plain root run (no recursion role) when the manifest + * resolves to zero workspaces. + */ +export async function runRecursiveOpenWiki( + command: OpenWikiCommand, + repoRoot: string, + options: OpenWikiRunOptions, + manifest: WorkspaceManifest, +): Promise { + const plan = resolveWorkspaceRuns(repoRoot, manifest); + + // Once-per-process setup, mirroring runOpenWikiAgent but hoisted so every run + // in the loop shares it (avoids a skills-dir race and repeated model builds). + await loadOpenWikiEnv(); + await syncBundledSkills(); + // Recursive runs scaffold a workflow that reruns with --recursive so scheduled + // refreshes keep every sub-wiki current. + await ensureCodeModeRepoSetup(repoRoot, { recursive: true }); + + const model = resolveRunModel(options); + await refreshChatGptTokensIfNeeded(model.provider, options); + + if (plan.runs.length === 0) { + // Empty manifest: fall back to a plain single run (NOT the root role). + const rootResult = await runOne(command, repoRoot, options, model); + return { + subprojectResults: [], + rootResult, + skippedWorkspaces: [], + failedWorkspaces: [], + }; + } + + const subprojectResults: OpenWikiRunResult[] = []; + const skippedWorkspaces: { path: string; reason: string }[] = []; + const failedWorkspaces: { path: string; error: string }[] = []; + const state: WorkspacesState = await readWorkspacesState(repoRoot); + + for (const run of plan.runs) { + const skipReason = await getWorkspaceSkipReason(repoRoot, run); + if (skipReason) { + emitBoundary( + options, + `Skipping workspace ${run.relativePath}: ${skipReason}`, + ); + skippedWorkspaces.push({ path: run.relativePath, reason: skipReason }); + continue; + } + + emitBoundary( + options, + `OpenWiki subproject: ${run.name ?? run.relativePath}`, + ); + + try { + const subprojectGoal = await resolveSubprojectGoal(run); + const result = await runOne( + command, + run.absolutePath, + { + ...options, + recursionRole: "subproject", + // Each run gets a distinct thread; never reuse the top-level threadId. + threadId: undefined, + wikiGoalOverride: subprojectGoal, + }, + model, + { mode: "subproject" }, + ); + subprojectResults.push(result); + + // Record the subproject's git HEAD so future runs can reason about which + // subprojects moved. Best-effort: a missing HEAD does not fail the run. + state.workspaces[run.relativePath] = { + gitHead: await readGitHead(run.absolutePath), + updatedAt: new Date().toISOString(), + }; + await writeWorkspacesState(repoRoot, state); + } catch (error) { + // A broken subproject must not abandon the sub-wikis already generated or + // block aggregation + the root run for the ones that succeeded. Collect + // the failure and keep going. + const message = error instanceof Error ? error.message : String(error); + emitBoundary( + options, + `Subproject ${run.relativePath} failed: ${message}`, + ); + failedWorkspaces.push({ path: run.relativePath, error: message }); + } + } + + // Aggregation MUST be written before the root run so the root's index-sync + // (afterAgent middleware) links openwiki/workspaces.md into openwiki/index.md. + // Link only subprojects that actually produced a sub-wiki: skipped (no + // evidence) and failed subprojects have no quickstart to link to. + const excluded = new Set([ + ...skippedWorkspaces.map((entry) => entry.path), + ...failedWorkspaces.map((entry) => entry.path), + ]); + const documentedPlan: ResolvedWorkspacePlan = { + ...plan, + runs: plan.runs.filter((run) => !excluded.has(run.relativePath)), + }; + await writeRootAggregation(repoRoot, documentedPlan); + + emitBoundary(options, "OpenWiki root wiki"); + const rootResult = await runOne( + command, + repoRoot, + { + ...options, + recursionRole: "root", + threadId: undefined, + wikiGoalOverride: plan.rootGoal, + }, + model, + { mode: "root-excluding-nested" }, + ); + + return { subprojectResults, rootResult, skippedWorkspaces, failedWorkspaces }; +} + +/** + * Resolves a subproject's wiki brief with the approved precedence: the + * manifest-supplied goal wins, then the subproject's own + * openwiki/INSTRUCTIONS.md, then none. + */ +async function resolveSubprojectGoal( + run: ResolvedWorkspaceRun, +): Promise { + if (run.goal) { + return run.goal; + } + + return readRepositoryWikiInstructions(run.absolutePath); +} + +/** + * Writes the deterministic aggregation page openwiki/workspaces.md at the repo + * root, linking down to each subproject's sub-wiki entrypoint. Includes valid + * OKF front matter (type: Reference) so migrateWikiToOkf does not rewrite or + * retag it during the root run. + */ +export async function writeRootAggregation( + repoRoot: string, + plan: ResolvedWorkspacePlan, +): Promise { + const openWikiDir = path.join(repoRoot, "openwiki"); + await mkdir(openWikiDir, { recursive: true }); + + const rows = plan.runs + .map((run) => { + const label = run.name ?? run.relativePath; + const href = `../${run.relativePath}/openwiki/quickstart.md`; + const goal = run.goal ? ` — ${escapeTableCell(run.goal)}` : ""; + return `- [${escapeLinkLabel(label)}](${encodeSubwikiHref(href)})${goal}`; + }) + .join("\n"); + + const content = `--- +type: Reference +title: Workspaces +description: Generated index of this monorepo's subproject sub-wikis. Each entry links to that subproject's own OpenWiki quickstart. +--- + +# Workspaces + +This monorepo documents each subproject in its own OpenWiki sub-wiki. This page is generated automatically; do not hand-edit the list below. + +${rows || "No documented subprojects."} +`; + + await writeFile(path.join(openWikiDir, "workspaces.md"), content, "utf8"); +} + +/** + * URL-encodes each path segment of a sub-wiki href while preserving separators. + */ +function encodeSubwikiHref(href: string): string { + return href + .split("/") + .map((segment) => + segment === ".." ? segment : encodeURIComponent(segment), + ) + .join("/"); +} + +function escapeLinkLabel(value: string): string { + return value + .replaceAll("\\", "\\\\") + .replaceAll("[", "\\[") + .replaceAll("]", "\\]"); +} + +function escapeTableCell(value: string): string { + return value.replace(/\s+/gu, " ").trim(); +} + +/** + * Reads the git HEAD for a directory, returning undefined when not in a repo or + * git is unavailable. Best-effort; never throws. + */ +async function readGitHead(cwd: string): Promise { + try { + const { stdout } = await execFileAsync( + "git", + ["--no-pager", "rev-parse", "HEAD"], + { cwd }, + ); + const head = stdout.trim(); + return head.length > 0 ? head : undefined; + } catch { + return undefined; + } +} + +/** + * Emits a subproject/root boundary marker as a text event so the CLI can render + * coherent per-run progress across the sequential recursive pass. + */ +function emitBoundary(options: OpenWikiRunOptions, label: string): void { + options.onEvent?.({ type: "text", text: `\n=== ${label} ===\n` }); +} diff --git a/src/monorepo/workspaces.ts b/src/monorepo/workspaces.ts new file mode 100644 index 00000000..a942681b --- /dev/null +++ b/src/monorepo/workspaces.ts @@ -0,0 +1,1256 @@ +import { + mkdir, + readFile, + readdir, + realpath, + stat, + writeFile, +} from "node:fs/promises"; +import path from "node:path"; +import { parse as parseYaml } from "yaml"; + +/** + * One workspace entry as written in `openwiki/workspaces.json`. `path` is a + * repository-root-relative POSIX path (no globs at runtime; auto-detect expands + * globs before writing the manifest). `goal` and `name` are optional overrides. + */ +export interface WorkspaceEntry { + path: string; + goal?: string; + name?: string; +} + +/** + * The parsed `openwiki/workspaces.json` manifest. `version` is 1 for the current + * explicit-paths schema. + */ +export interface WorkspaceManifest { + version: number; + workspaces: WorkspaceEntry[]; + root?: { goal?: string }; +} + +/** + * A validated, normalized workspace run target. + */ +export interface ResolvedWorkspaceRun { + /** Repository-root-relative POSIX path (normalized, no trailing slash). */ + relativePath: string; + /** Host absolute path to the subproject directory. */ + absolutePath: string; + /** Optional per-subproject wiki brief from the manifest. */ + goal?: string; + /** Optional display name from the manifest. */ + name?: string; +} + +/** + * The fully resolved recursion plan: subproject runs plus the optional root brief. + */ +export interface ResolvedWorkspacePlan { + runs: ResolvedWorkspaceRun[]; + rootGoal?: string; +} + +export const WORKSPACES_MANIFEST_RELATIVE_PATH = "openwiki/workspaces.json"; + +/** + * The outcome of deciding whether a run should recurse across a monorepo. + * + * - `recurse`: run the orchestrator with `manifest`. + * - `plain`: run a single ordinary run (no recursion). + */ +export type RecursionActivation = + | { kind: "recurse"; manifest: WorkspaceManifest; autoDetected: boolean } + | { kind: "plain"; reason: string }; + +/** + * Decides whether to recurse, honoring the product rules: + * - `recursive === false`: force off, even if a manifest exists. + * - a manifest exists: recurse (auto-enabled). + * - `recursive === true` and no manifest: auto-detect workspaces; if any are + * found, WRITE openwiki/workspaces.json (for the user to review) and recurse; + * otherwise fall back to a plain run. + * - otherwise (default, no manifest): plain run. + */ +export async function resolveRecursionActivation( + repoRoot: string, + recursive: boolean | undefined, +): Promise { + if (recursive === false) { + return { + kind: "plain", + reason: "recursion disabled with --recursive=false", + }; + } + + const manifest = await readWorkspaceManifest(repoRoot); + if (manifest) { + return { kind: "recurse", manifest, autoDetected: false }; + } + + if (recursive !== true) { + return { kind: "plain", reason: "no openwiki/workspaces.json manifest" }; + } + + const detected = await detectWorkspaces(repoRoot); + if (detected.length === 0) { + return { + kind: "plain", + reason: + "--recursive requested but no monorepo workspaces were detected (pnpm-workspace.yaml, package.json workspaces, Cargo.toml [workspace], go.work, *.sln/*.slnx, Maven pom.xml modules, settings.gradle[.kts], pyproject.toml [tool.uv.workspace], Bazel MODULE.bazel/WORKSPACE)", + }; + } + + // Never persist a manifest that cannot be resolved: a poisoned + // openwiki/workspaces.json would make every future default run auto-recurse + // into a throw, wedging the repo until the user hand-edits it. Prune to a + // resolvable (leaf-only) set before writing. + const writtenManifest: WorkspaceManifest = { + version: 1, + workspaces: pruneToResolvableWorkspaces(repoRoot, detected), + }; + await writeWorkspaceManifest(repoRoot, writtenManifest); + + return { kind: "recurse", manifest: writtenManifest, autoDetected: true }; +} + +/** + * Returns a workspace set that resolveWorkspaceRuns accepts. If the raw detected + * set already resolves, it is returned unchanged; otherwise overlaps are pruned + * by dropping any entry that is an ancestor of another, keeping the leaves. + */ +function pruneToResolvableWorkspaces( + repoRoot: string, + detected: WorkspaceEntry[], +): WorkspaceEntry[] { + try { + resolveWorkspaceRuns(repoRoot, { version: 1, workspaces: detected }); + return detected; + } catch { + const leaves = new Set( + dropAncestorPaths(detected.map((entry) => entry.path)), + ); + return detected.filter((entry) => leaves.has(entry.path)); + } +} + +/** + * Writes an auto-detected manifest to openwiki/workspaces.json so the user can + * review and edit the detected workspace set. Written with a comment-free JSON + * body and a trailing newline. + */ +export async function writeWorkspaceManifest( + repoRoot: string, + manifest: WorkspaceManifest, +): Promise { + const manifestPath = path.join(repoRoot, WORKSPACES_MANIFEST_RELATIVE_PATH); + await mkdir(path.dirname(manifestPath), { recursive: true }); + await writeFile( + manifestPath, + `${JSON.stringify(manifest, null, 2)}\n`, + "utf8", + ); +} + +/** + * Reads and structurally validates `openwiki/workspaces.json` at the repo root. + * Returns null when the manifest is absent. Throws on malformed JSON or a + * structurally invalid manifest so activation fails loudly rather than silently + * degrading to a plain run. + */ +export async function readWorkspaceManifest( + repoRoot: string, +): Promise { + const manifestPath = path.join(repoRoot, WORKSPACES_MANIFEST_RELATIVE_PATH); + + let raw: string; + try { + raw = await readFile(manifestPath, "utf8"); + } catch (error) { + if (isFileNotFoundError(error)) { + return null; + } + throw error; + } + + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch (error) { + throw new Error( + `openwiki/workspaces.json is not valid JSON: ${ + error instanceof Error ? error.message : String(error) + }`, + { cause: error }, + ); + } + + return normalizeManifest(parsed); +} + +/** + * Normalizes an untrusted parsed manifest into a WorkspaceManifest, rejecting + * structurally invalid input. + */ +export function normalizeManifest(value: unknown): WorkspaceManifest { + if (!isRecord(value)) { + throw new Error("openwiki/workspaces.json must be a JSON object."); + } + + const version = value.version; + if (version !== undefined && version !== 1) { + throw new Error( + `openwiki/workspaces.json version ${JSON.stringify( + version, + )} is unsupported; only version 1 is understood.`, + ); + } + + if (!Array.isArray(value.workspaces)) { + throw new Error( + "openwiki/workspaces.json must contain a `workspaces` array.", + ); + } + + const workspaces: WorkspaceEntry[] = value.workspaces.map((entry, index) => { + if (!isRecord(entry) || typeof entry.path !== "string") { + throw new Error( + `openwiki/workspaces.json workspaces[${index}] must be an object with a string \`path\`.`, + ); + } + + return { + path: entry.path, + goal: typeof entry.goal === "string" ? entry.goal : undefined, + name: typeof entry.name === "string" ? entry.name : undefined, + }; + }); + + const rootGoal = + isRecord(value.root) && typeof value.root.goal === "string" + ? value.root.goal + : undefined; + + return { + version: 1, + workspaces, + ...(rootGoal !== undefined ? { root: { goal: rootGoal } } : {}), + }; +} + +/** + * Validates, normalizes, dedupes, and rejects overlapping workspace entries, + * returning an ordered recursion plan (leaves first is the caller's concern; + * this function preserves manifest order after dedupe). + * + * Rejections (throw): + * - absolute paths or paths containing `..` segments (escape attempts) + * - a workspace equal to the repository root + * - a workspace that is an ancestor of another (nested/overlapping) + */ +export function resolveWorkspaceRuns( + repoRoot: string, + manifest: WorkspaceManifest, +): ResolvedWorkspacePlan { + const seen = new Set(); + const runs: ResolvedWorkspaceRun[] = []; + + for (const entry of manifest.workspaces) { + const relativePath = normalizeWorkspacePath(entry.path); + + if (relativePath === "" || relativePath === ".") { + throw new Error( + `Invalid workspace path ${JSON.stringify( + entry.path, + )}: a workspace may not be the repository root (it collides with the root wiki).`, + ); + } + + if (seen.has(relativePath)) { + continue; + } + seen.add(relativePath); + + runs.push({ + relativePath, + absolutePath: path.resolve(repoRoot, relativePath), + goal: entry.goal?.trim() ? entry.goal.trim() : undefined, + name: entry.name?.trim() ? entry.name.trim() : undefined, + }); + } + + assertNoOverlappingWorkspaces(runs); + + return { + runs, + rootGoal: manifest.root?.goal?.trim() + ? manifest.root.goal.trim() + : undefined, + }; +} + +/** + * Normalizes a manifest workspace path to a repo-root-relative POSIX path, + * rejecting absolute paths and `..` traversal (mirrors the normalization in + * isOpenWikiDocsPath and adds traversal rejection). + */ +function normalizeWorkspacePath(rawPath: string): string { + const withForwardSlashes = rawPath.trim().replace(/\\/gu, "/"); + + if (withForwardSlashes.startsWith("/")) { + throw new Error( + `Invalid workspace path ${JSON.stringify( + rawPath, + )}: absolute paths are not allowed. Use a path relative to the repository root.`, + ); + } + + const normalized = path.posix.normalize(withForwardSlashes); + const trimmed = normalized.replace(/\/+$/u, "").replace(/^\.\//u, ""); + + if ( + trimmed === ".." || + trimmed.startsWith("../") || + trimmed.split("/").includes("..") + ) { + throw new Error( + `Invalid workspace path ${JSON.stringify( + rawPath, + )}: paths may not escape the repository root with "..".`, + ); + } + + return trimmed; +} + +/** + * Throws when any workspace is an ancestor of another (nested/overlapping). + */ +function assertNoOverlappingWorkspaces(runs: ResolvedWorkspaceRun[]): void { + for (const outer of runs) { + for (const inner of runs) { + if (outer === inner) { + continue; + } + + if (isAncestorPath(outer.relativePath, inner.relativePath)) { + throw new Error( + `Overlapping workspaces: "${outer.relativePath}" is an ancestor of "${inner.relativePath}". No workspace may be an ancestor of another.`, + ); + } + } + } +} + +/** + * True when `ancestor` is a strict path-segment ancestor of `descendant`. + */ +function isAncestorPath(ancestor: string, descendant: string): boolean { + const ancestorSegments = ancestor.split("/"); + const descendantSegments = descendant.split("/"); + + if (ancestorSegments.length >= descendantSegments.length) { + return false; + } + + return ancestorSegments.every( + (segment, index) => segment === descendantSegments[index], + ); +} + +const WORKSPACE_MANIFEST_CANDIDATES = ["package.json", "Cargo.toml", "go.mod"]; + +/** + * Confirms a resolved workspace directory is a real subtree of the repository + * (rejecting symlink escapes) and carries some evidence worth documenting. + * Returns a reason string when the workspace should be skipped, or null when it + * is runnable. + */ +export async function getWorkspaceSkipReason( + repoRoot: string, + run: ResolvedWorkspaceRun, +): Promise { + let realWorkspace: string; + let realRepoRoot: string; + try { + realWorkspace = await realpath(run.absolutePath); + realRepoRoot = await realpath(repoRoot); + } catch (error) { + if (isFileNotFoundError(error)) { + return `directory "${run.relativePath}" does not exist`; + } + throw error; + } + + const relativeFromRoot = path.relative(realRepoRoot, realWorkspace); + if ( + relativeFromRoot === "" || + relativeFromRoot.startsWith("..") || + path.isAbsolute(relativeFromRoot) + ) { + return `directory "${run.relativePath}" resolves outside the repository (symlink escape)`; + } + + const workspaceStat = await stat(realWorkspace).catch(() => null); + if (!workspaceStat?.isDirectory()) { + return `"${run.relativePath}" is not a directory`; + } + + // A workspace is worth documenting if it has a package manifest, an + // INSTRUCTIONS.md brief, an explicit goal, or any non-trivial source content. + if (run.goal) { + return null; + } + + const hasInstructions = await pathExists( + path.join(run.absolutePath, "openwiki", "INSTRUCTIONS.md"), + ); + if (hasInstructions) { + return null; + } + + for (const candidate of WORKSPACE_MANIFEST_CANDIDATES) { + if (await pathExists(path.join(run.absolutePath, candidate))) { + return null; + } + } + + const hasSource = await directoryHasNonWikiEntries(run.absolutePath); + if (!hasSource) { + return `"${run.relativePath}" has no package manifest, no openwiki/INSTRUCTIONS.md, no goal, and no source files`; + } + + return null; +} + +async function directoryHasNonWikiEntries(directory: string): Promise { + let entries; + try { + entries = await readdir(directory, { withFileTypes: true }); + } catch { + return false; + } + + return entries.some( + (entry) => + !entry.name.startsWith(".") && + entry.name !== "openwiki" && + entry.name !== "node_modules", + ); +} + +/** + * Auto-detects workspaces from common monorepo manifests, expanding any globs + * into concrete, existing directories. Order of precedence: pnpm-workspace.yaml, + * package.json `workspaces`, Cargo.toml `[workspace] members`, go.work `use` + * directives, .NET `*.sln`/`*.slnx` project entries, Maven `pom.xml` modules, + * Gradle `settings.gradle[.kts]` includes, Python uv + * `[tool.uv.workspace] members`, then coarse Bazel top-level project roots. + * Returns [] when nothing is detected. + */ +export async function detectWorkspaces( + repoRoot: string, +): Promise { + const globs = await collectWorkspaceGlobs(repoRoot); + if (globs.length === 0) { + return []; + } + + const relativePaths = new Set(); + for (const glob of globs) { + for (const match of await expandWorkspaceGlob(repoRoot, glob)) { + relativePaths.add(match); + } + } + + return [...relativePaths] + .sort((left, right) => left.localeCompare(right)) + .map((relativePath) => ({ path: relativePath })); +} + +/** + * Gathers raw workspace globs/paths from every recognized manifest. + * + * Precedence order: JS/TS (pnpm-workspace.yaml, package.json), Rust + * (Cargo.toml), Go (go.work), .NET (*.sln / *.slnx), Java/JVM (Maven pom.xml + * modules, Gradle settings.gradle[.kts] includes), Python (uv + * [tool.uv.workspace]), then Bazel (coarse top-level project roots). + */ +async function collectWorkspaceGlobs(repoRoot: string): Promise { + const globs: string[] = []; + + const pnpmGlobs = await readPnpmWorkspaceGlobs(repoRoot); + globs.push(...pnpmGlobs); + + const packageJsonGlobs = await readPackageJsonWorkspaceGlobs(repoRoot); + globs.push(...packageJsonGlobs); + + const cargoGlobs = await readCargoWorkspaceGlobs(repoRoot); + globs.push(...cargoGlobs); + + const goWorkGlobs = await readGoWorkGlobs(repoRoot); + globs.push(...goWorkGlobs); + + const dotnetGlobs = await readDotnetSolutionGlobs(repoRoot); + globs.push(...dotnetGlobs); + + const mavenGlobs = await readMavenModuleGlobs(repoRoot); + globs.push(...mavenGlobs); + + const gradleGlobs = await readGradleIncludeGlobs(repoRoot); + globs.push(...gradleGlobs); + + const uvGlobs = await readUvWorkspaceGlobs(repoRoot); + globs.push(...uvGlobs); + + const bazelGlobs = await readBazelWorkspaceGlobs(repoRoot); + globs.push(...bazelGlobs); + + return dedupeStrings( + globs.map((glob) => glob.trim()).filter((glob) => glob.length > 0), + ); +} + +async function readPnpmWorkspaceGlobs(repoRoot: string): Promise { + const raw = await readFileIfPresent( + path.join(repoRoot, "pnpm-workspace.yaml"), + ); + if (raw === null) { + return []; + } + + try { + const parsed: unknown = parseYaml(raw); + if (isRecord(parsed) && Array.isArray(parsed.packages)) { + return parsed.packages.filter( + (value): value is string => typeof value === "string", + ); + } + } catch { + return []; + } + + return []; +} + +async function readPackageJsonWorkspaceGlobs( + repoRoot: string, +): Promise { + const raw = await readFileIfPresent(path.join(repoRoot, "package.json")); + if (raw === null) { + return []; + } + + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch { + return []; + } + + if (!isRecord(parsed)) { + return []; + } + + const workspaces = parsed.workspaces; + if (Array.isArray(workspaces)) { + return workspaces.filter( + (value): value is string => typeof value === "string", + ); + } + + if (isRecord(workspaces) && Array.isArray(workspaces.packages)) { + return workspaces.packages.filter( + (value): value is string => typeof value === "string", + ); + } + + return []; +} + +async function readCargoWorkspaceGlobs(repoRoot: string): Promise { + const raw = await readFileIfPresent(path.join(repoRoot, "Cargo.toml")); + if (raw === null) { + return []; + } + + // Minimal TOML extraction: find the [workspace] section and its `members` + // array. Avoids a TOML dependency for this narrow detection use. + const workspaceSection = /\[workspace\]([\s\S]*?)(?:\n\[|$)/u.exec(raw); + if (!workspaceSection) { + return []; + } + + const membersMatch = /members\s*=\s*\[([\s\S]*?)\]/u.exec( + workspaceSection[1], + ); + if (!membersMatch) { + return []; + } + + return [...membersMatch[1].matchAll(/["']([^"']+)["']/gu)].map( + (match) => match[1], + ); +} + +async function readGoWorkGlobs(repoRoot: string): Promise { + const raw = await readFileIfPresent(path.join(repoRoot, "go.work")); + if (raw === null) { + return []; + } + + const paths: string[] = []; + const blockMatch = /use\s*\(([\s\S]*?)\)/u.exec(raw); + if (blockMatch) { + for (const line of blockMatch[1].split("\n")) { + const trimmed = line.trim(); + if (trimmed.length > 0 && !trimmed.startsWith("//")) { + paths.push(trimmed); + } + } + } + + for (const match of raw.matchAll(/^\s*use\s+([^\s(][^\s]*)\s*$/gmu)) { + paths.push(match[1]); + } + + return paths.map((value) => value.replace(/^\.\//u, "")); +} + +/** + * The GUID that marks a "solution folder" (a virtual organizational grouping) + * in a classic `.sln` file. Solution-folder entries are not real projects and + * their path field is just a folder name, so they must be skipped. + */ +const SLN_SOLUTION_FOLDER_TYPE_GUID = "2150E333-8FDC-42A3-9474-1A3956D46DE8"; + +const SLN_PROJECT_EXTENSIONS = [".csproj", ".vbproj", ".fsproj"]; + +/** + * Reads .NET solution files at the repo root and returns the directory of every + * referenced project. Both the classic `*.sln` format and the newer XML + * `*.slnx` format (default from `dotnet new sln` on .NET 9+) are handled; all + * solution files in the repo root are scanned (not recursively). + * + * Classic `.sln` lists projects as lines like: + * Project("{TYPE-GUID}") = "Name", "relative\path\Foo.csproj", "{PROJ-GUID}" + * The second quoted field is the project path (backslash-separated). Verified + * `.slnx` schema against Microsoft docs / vs-solutionpersistence: an XML + * `` with `` entries and + * `` elements for organizational folders. In both formats only entries + * whose path ends in a real project extension (.csproj/.vbproj/.fsproj) are + * emitted, which naturally excludes classic solution-folder entries (marked by + * the type GUID 2150E333-... with a bare folder name) and slnx ``s. + */ +async function readDotnetSolutionGlobs(repoRoot: string): Promise { + let entries; + try { + entries = await readdir(repoRoot, { withFileTypes: true }); + } catch { + return []; + } + + const dirs: string[] = []; + for (const entry of entries) { + if (!entry.isFile()) { + continue; + } + const lower = entry.name.toLowerCase(); + if (lower.endsWith(".sln")) { + dirs.push(...(await readClassicSlnProjectDirs(repoRoot, entry.name))); + } else if (lower.endsWith(".slnx")) { + dirs.push(...(await readSlnxProjectDirs(repoRoot, entry.name))); + } + } + + return coarsenDotnetProjectDirs(dirs); +} + +/** + * Coarsens raw .NET project directories up to their product-area roots. + * + * The .NET DDD convention nests each project under an intermediate `src/` or + * `tests/` directory inside its product area, e.g. + * platform/core/admission/src/Core.Admission.Api + * platform/core/admission/src/Core.Admission.Domain + * platform/core/admission/tests/Core.Admission.Domain.Tests + * all belong to the ONE area `platform/core/admission`. Emitting one workspace + * per `.csproj` is far too granular for this feature (a large monorepo yields + * hundreds of projects, hence hundreds of agent runs); the intended unit is the + * product area / bounded context. + * + * Rule: for a project directory whose IMMEDIATE parent segment is exactly `src` + * or `tests`, drop the trailing `/src/` or `/tests/` to + * reach the area root — but ONLY when the resulting area is non-empty and is not + * itself literally `src` or `tests`. Two guards enforce that: + * 1. `segments.length >= 3` (there is something above the `src`/`tests` + * parent), so a top-level `src/Api` is left unchanged instead of collapsing + * to the repository root. + * 2. `area !== "src"` and `area !== "tests"`, so idiomatic layouts where the + * area segment is itself `src`/`tests` (e.g. a top-level `src/tests/App.Tests`, + * or `src/src/Foo`) are NOT collapsed to a bare `src`/`tests` — that would + * make one wiki span the entire source (or test) tree, the exact granularity + * this coarsening exists to prevent. + * Flatter libs whose project directory sits directly under a container (e.g. + * `kernel/Core.Domain.Kernel`, whose parent `kernel` is neither `src` nor + * `tests`) are left unchanged. After coarsening, ancestor paths are dropped so + * the result is a clean, non-overlapping set of area/lib leaves. + * + * Known edge (data-dependent, accepted): coarsen-then-drop-ancestor can drop a + * project that sits directly in an area (`area/Y.csproj` -> `area`) when a + * sibling sits deeper under a non-`src`/`tests` intermediate (`area/group/Z`, + * left granular), because `area` then becomes an ancestor of `area/group/Z` and + * is dropped. This is inherent to coarsening + overlap-pruning and low-frequency + * in practice; auto-detect writes the manifest for human review, so a lost + * project is recoverable by hand rather than silently unrecoverable. + * + * Scoped to .NET on purpose: the pnpm/npm/cargo/go/uv detectors already emit + * package-level units where a `packages/foo/src/bar` may be a legitimate leaf, + * so coarsening is applied only to the .sln/.slnx source, not globally. + */ +function coarsenDotnetProjectDirs(dirs: string[]): string[] { + const coarsened = dirs.map((dir) => { + const segments = dir.split("/"); + // Require at least //: the parent must be + // exactly "src" or "tests", and the trimmed area must be a real area + // segment (non-empty, and not itself "src"/"tests"). + if (segments.length >= 3) { + const parent = segments[segments.length - 2]; + if (parent === "src" || parent === "tests") { + const area = segments.slice(0, -2).join("/"); + if (area !== "" && area !== "src" && area !== "tests") { + return area; + } + } + } + return dir; + }); + + return dropAncestorPaths(coarsened); +} + +/** + * Extracts project directories from a classic `.sln` file, skipping + * solution-folder entries and any non-project path. + */ +async function readClassicSlnProjectDirs( + repoRoot: string, + fileName: string, +): Promise { + const raw = await readFileIfPresent(path.join(repoRoot, fileName)); + if (raw === null) { + return []; + } + + const dirs: string[] = []; + const projectLine = + /Project\("\{([0-9A-Fa-f-]+)\}"\)\s*=\s*"[^"]*",\s*"([^"]+)"/gu; + for (const match of raw.matchAll(projectLine)) { + const typeGuid = match[1].toUpperCase(); + if (typeGuid === SLN_SOLUTION_FOLDER_TYPE_GUID) { + continue; + } + const dir = dotnetProjectPathToDir(match[2]); + if (dir !== null) { + dirs.push(dir); + } + } + + return dirs; +} + +/** + * Extracts project directories from an XML `.slnx` file. Only `` + * elements with a real project-file `Path` are emitted; `` elements + * (organizational only) are ignored by the extension check. + */ +async function readSlnxProjectDirs( + repoRoot: string, + fileName: string, +): Promise { + const raw = await readFileIfPresent(path.join(repoRoot, fileName)); + if (raw === null) { + return []; + } + + const dirs: string[] = []; + const projectAttr = /]*\bPath\s*=\s*["']([^"']+)["']/gu; + for (const match of raw.matchAll(projectAttr)) { + const dir = dotnetProjectPathToDir(match[1]); + if (dir !== null) { + dirs.push(dir); + } + } + + return dirs; +} + +/** + * Normalizes a .NET project path (backslash-separated, may point at a + * .csproj/.vbproj/.fsproj) to its repo-root-relative directory, or null when + * the path is not a recognized project file. + */ +function dotnetProjectPathToDir(rawPath: string): string | null { + const forwardSlashes = rawPath.trim().replace(/\\/gu, "/"); + const lower = forwardSlashes.toLowerCase(); + if (!SLN_PROJECT_EXTENSIONS.some((ext) => lower.endsWith(ext))) { + return null; + } + const dir = path.posix.dirname(forwardSlashes); + return dir === "." ? "" : dir; +} + +/** + * Reads a parent Maven `pom.xml` at the repo root and returns its `` + * directories. Each `` is a relative directory (occasionally a dir + * containing a nested pom). No XML-parser dependency is present, so a narrow + * regex scoped to the `` block is used, mirroring the Cargo approach. + * XML comments inside the block are stripped before extraction so + * `` is ignored. + */ +async function readMavenModuleGlobs(repoRoot: string): Promise { + const raw = await readFileIfPresent(path.join(repoRoot, "pom.xml")); + if (raw === null) { + return []; + } + + const modulesBlock = /]*>([\s\S]*?)<\/modules>/u.exec(raw); + if (!modulesBlock) { + return []; + } + + const withoutComments = modulesBlock[1].replace(//gu, ""); + return [...withoutComments.matchAll(/\s*([^<]+?)\s*<\/module>/gu)].map( + (match) => match[1].replace(/\\/gu, "/"), + ); +} + +/** + * Reads Gradle settings files (`settings.gradle` Groovy and + * `settings.gradle.kts` Kotlin) at the repo root and returns the directories of + * every `include`d project. Gradle project paths use `:` as a separator and map + * to directories: `:foo:bar` -> `foo/bar` (a leading `:` is stripped, remaining + * `:` become `/`). Every quoted token following an `include` keyword is + * extracted, covering `include ':a'`, `include(":a")`, and comma-separated + * multi-project forms `include ':a', ':b'` / `include(":a", ":b")`. + * + * Known limitation (out of scope for v1): `project(':x').projectDir = + * file('...')` can remap a project to a directory that the default `:`->`/` + * mapping would not produce. The common convention is covered; remapped + * projects are not. + */ +async function readGradleIncludeGlobs(repoRoot: string): Promise { + const paths: string[] = []; + for (const fileName of ["settings.gradle", "settings.gradle.kts"]) { + const raw = await readFileIfPresent(path.join(repoRoot, fileName)); + if (raw === null) { + continue; + } + + // Strip block and line comments first so a commented-out include (whole + // line or trailing) never leaks a phantom project. `\binclude\b` matches + // `include` but not `includeBuild` (composite builds are not subprojects). + const noComments = raw + .replace(/\/\*[\s\S]*?\*\//gu, "") + .replace(/\/\/[^\n]*/gu, ""); + + // Match an `include` keyword (Groovy `include ...` or Kotlin + // `include(...)`) and its arguments, spanning continuation lines so that a + // wrapped, comma-separated list is captured whole. The argument run stops + // at a `;` or at a newline that does NOT begin a continuation (a line whose + // first non-space char is a quote, comma, colon, or closing paren). + for (const stmt of noComments.matchAll( + /(?:^|;|\n)\s*include\b\s*\(?((?:[^\n;()]|\n(?=\s*['":,)]))*)/gu, + )) { + for (const token of stmt[1].matchAll(/["']([^"']+)["']/gu)) { + const dir = gradleProjectPathToDir(token[1]); + if (dir.length > 0) { + paths.push(dir); + } + } + } + } + + return paths; +} + +/** + * Maps a Gradle project path (`:foo:bar`) to a repo-root-relative directory + * (`foo/bar`): strip a single leading `:`, then replace remaining `:` with `/`. + */ +function gradleProjectPathToDir(projectPath: string): string { + return projectPath.trim().replace(/^:/u, "").replace(/:/gu, "/"); +} + +/** + * Reads a Python `pyproject.toml` at the repo root and returns the uv workspace + * `members` globs from the `[tool.uv.workspace]` section. Verified against the + * current uv docs: `members` (required) and `exclude` (optional) are both lists + * of globs. `members` entries support `*`, expanded downstream. A narrow regex + * scoped to the section is used (mirroring the Cargo approach) since no TOML + * dependency is present. + * + * Known limitation (deferred): the optional `exclude` list is NOT honored. + * Excluded directories that still match a `members` glob and exist on disk will + * be detected. In practice the downstream skip/overlap guards and manual + * manifest review absorb this; honoring `exclude` cheaply is possible but was + * left out of v1 to keep the extractor a pure member-glob reader. + */ +async function readUvWorkspaceGlobs(repoRoot: string): Promise { + const raw = await readFileIfPresent(path.join(repoRoot, "pyproject.toml")); + if (raw === null) { + return []; + } + + const section = /\[tool\.uv\.workspace\]([\s\S]*?)(?:\n\[|$)/u.exec(raw); + if (!section) { + return []; + } + + const membersMatch = /members\s*=\s*\[([\s\S]*?)\]/u.exec(section[1]); + if (!membersMatch) { + return []; + } + + return [...membersMatch[1].matchAll(/["']([^"']+)["']/gu)].map( + (match) => match[1], + ); +} + +/** + * Detects a Bazel workspace and emits only COARSE top-level project roots. + * + * Bazel deliberately has no notion of "workspace member directories": a Bazel + * *package* is any directory containing a `BUILD`/`BUILD.bazel` file, which in a + * real repo is hundreds of directories at every depth. Enumerating them all + * would produce a massive, deeply-overlapping set that is wrong for this feature + * (which wants "major projects get granular docs", not "every leaf package"). + * + * So detection is intentionally shallow: presence is inferred from a root + * `MODULE.bazel` (Bzlmod) or `WORKSPACE`/`WORKSPACE.bazel` (legacy) file, and + * the emitted set is the immediate child directories of the repo root — and of a + * conventional `src/` directory if present — that themselves contain a + * `BUILD`/`BUILD.bazel`. This yields a handful of top-level project roots rather + * than an explosion of nested packages. If nothing at the top level qualifies, + * [] is returned: for a deep Bazel tree with no BUILD file at the top level, a + * hand-authored openwiki/workspaces.json is the right escape hatch, which is + * strictly better than a wrong, overlapping explosion. + */ +async function readBazelWorkspaceGlobs(repoRoot: string): Promise { + const markers = ["MODULE.bazel", "WORKSPACE.bazel", "WORKSPACE"]; + let hasBazel = false; + for (const marker of markers) { + if (await pathExists(path.join(repoRoot, marker))) { + hasBazel = true; + break; + } + } + if (!hasBazel) { + return []; + } + + const dirs: string[] = []; + for (const base of ["", "src"]) { + for (const child of await listSubdirectories(repoRoot, base)) { + if (await directoryHasBazelBuildFile(repoRoot, child)) { + dirs.push(child); + } + } + } + + return dirs; +} + +/** + * True when a directory contains a Bazel `BUILD` or `BUILD.bazel` file. + */ +async function directoryHasBazelBuildFile( + repoRoot: string, + relativePath: string, +): Promise { + for (const name of ["BUILD.bazel", "BUILD"]) { + if (await pathExists(path.join(repoRoot, relativePath, name))) { + return true; + } + } + return false; +} + +/** + * Expands one workspace glob into concrete, existing, repo-relative directory + * paths. Supports `*` (single segment) and `**` (any depth). A glob with no + * wildcard is treated as a literal directory path. + */ +async function expandWorkspaceGlob( + repoRoot: string, + glob: string, +): Promise { + const normalized = glob + .replace(/\\/gu, "/") + .replace(/^\.\//u, "") + .replace(/\/+$/u, ""); + + if (normalized === "" || normalized.startsWith("..")) { + return []; + } + + const segments = normalized.split("/"); + const matches = (await expandSegments(repoRoot, "", segments)).filter( + (relativePath) => relativePath !== "" && relativePath !== ".", + ); + + // A `**` glob matches at every depth, so naive expansion emits the container + // directory AND all intermediates (e.g. `packages/**` → packages, packages/a, + // packages/a/nested). Tools treat `packages/**` as "the workspace packages", + // not "every directory under packages". Reduce to LEAF package directories so + // the detected set never contains a directory that is an ancestor of another + // match (which resolveWorkspaceRuns would reject as overlapping). + if (segments.includes("**")) { + return filterDoubleStarMatchesToLeafPackages(repoRoot, matches); + } + + return matches; +} + +/** + * Reduces raw `**` matches to leaf workspace packages: prefer directories that + * are actual package roots (contain a package manifest); if none do, fall back + * to dropping any directory that is an ancestor of another match. + */ +async function filterDoubleStarMatchesToLeafPackages( + repoRoot: string, + matches: string[], +): Promise { + const packageDirs: string[] = []; + for (const relativePath of matches) { + if (await directoryHasPackageManifest(repoRoot, relativePath)) { + packageDirs.push(relativePath); + } + } + + if (packageDirs.length > 0) { + return dropAncestorPaths(packageDirs); + } + + return dropAncestorPaths(matches); +} + +/** + * True when a directory contains a recognized package manifest. + */ +async function directoryHasPackageManifest( + repoRoot: string, + relativePath: string, +): Promise { + for (const candidate of WORKSPACE_MANIFEST_CANDIDATES) { + if (await pathExists(path.join(repoRoot, relativePath, candidate))) { + return true; + } + } + return false; +} + +/** + * Removes any path that is a strict ancestor of another path in the set, so + * only leaves survive. Deduplicates as a side effect. + */ +function dropAncestorPaths(paths: string[]): string[] { + const unique = [...new Set(paths)]; + return unique.filter( + (candidate) => + !unique.some( + (other) => other !== candidate && isAncestorPath(candidate, other), + ), + ); +} + +async function expandSegments( + repoRoot: string, + currentRelative: string, + segments: string[], +): Promise { + if (segments.length === 0) { + const absolute = path.join(repoRoot, currentRelative); + const dirStat = await stat(absolute).catch(() => null); + return dirStat?.isDirectory() ? [currentRelative] : []; + } + + const [segment, ...rest] = segments; + + if (segment === "**") { + // Match zero or more directory levels. + const results: string[] = []; + results.push(...(await expandSegments(repoRoot, currentRelative, rest))); + + for (const child of await listSubdirectories(repoRoot, currentRelative)) { + results.push(...(await expandSegments(repoRoot, child, segments))); + } + + return dedupeStrings(results); + } + + if (segment.includes("*")) { + const matcher = globSegmentToRegExp(segment); + const results: string[] = []; + for (const child of await listSubdirectories(repoRoot, currentRelative)) { + const childName = path.posix.basename(child); + if (matcher.test(childName)) { + results.push(...(await expandSegments(repoRoot, child, rest))); + } + } + return results; + } + + const nextRelative = currentRelative + ? path.posix.join(currentRelative, segment) + : segment; + const absolute = path.join(repoRoot, nextRelative); + const dirStat = await stat(absolute).catch(() => null); + if (!dirStat?.isDirectory()) { + return []; + } + return expandSegments(repoRoot, nextRelative, rest); +} + +async function listSubdirectories( + repoRoot: string, + currentRelative: string, +): Promise { + const absolute = path.join(repoRoot, currentRelative); + let entries; + try { + entries = await readdir(absolute, { withFileTypes: true }); + } catch { + return []; + } + + return entries + .filter( + (entry) => + entry.isDirectory() && + entry.name !== "node_modules" && + entry.name !== ".git" && + !entry.name.startsWith("."), + ) + .map((entry) => + currentRelative + ? path.posix.join(currentRelative, entry.name) + : entry.name, + ); +} + +function globSegmentToRegExp(segment: string): RegExp { + const escaped = segment + .replace(/[.+^${}()|[\]\\]/gu, "\\$&") + .replace(/\*/gu, "[^/]*"); + return new RegExp(`^${escaped}$`, "u"); +} + +async function readFileIfPresent(filePath: string): Promise { + try { + return await readFile(filePath, "utf8"); + } catch (error) { + // ENOENT: absent. EISDIR/ENOTDIR: the name exists but is a directory (or a + // path component is not a directory), which for manifest detection means + // "no readable manifest here" — treat as absent rather than letting it + // propagate and wedge detection for the whole repo. + if (isMissingFileError(error)) { + return null; + } + throw error; + } +} + +async function pathExists(filePath: string): Promise { + try { + await stat(filePath); + return true; + } catch { + return false; + } +} + +function dedupeStrings(values: string[]): string[] { + return [...new Set(values)]; +} + +/** + * Per-subproject generator state persisted at openwiki/.workspaces-state.json. + * Maps each subproject's repo-relative path to the git HEAD at the time its + * sub-wiki was last generated, so future runs can reason about which subprojects + * changed. This file is excluded from the content snapshot and generated + * indexes (see WORKSPACES_STATE_FILE in agent/utils.ts and EXCLUDED_FILES in + * okf/index-sync.ts). + */ +export interface WorkspacesState { + version: 1; + workspaces: Record; +} + +export const WORKSPACES_STATE_RELATIVE_PATH = "openwiki/.workspaces-state.json"; + +export async function readWorkspacesState( + repoRoot: string, +): Promise { + const statePath = path.join(repoRoot, WORKSPACES_STATE_RELATIVE_PATH); + const raw = await readFileIfPresent(statePath); + if (raw === null) { + return { version: 1, workspaces: {} }; + } + + try { + const parsed: unknown = JSON.parse(raw); + if (isRecord(parsed) && isRecord(parsed.workspaces)) { + const workspaces: WorkspacesState["workspaces"] = {}; + for (const [key, value] of Object.entries(parsed.workspaces)) { + if (isRecord(value) && typeof value.updatedAt === "string") { + workspaces[key] = { + gitHead: + typeof value.gitHead === "string" ? value.gitHead : undefined, + updatedAt: value.updatedAt, + }; + } + } + return { version: 1, workspaces }; + } + } catch { + // Corrupt state is non-fatal: treat as empty and let the run rewrite it. + } + + return { version: 1, workspaces: {} }; +} + +export async function writeWorkspacesState( + repoRoot: string, + state: WorkspacesState, +): Promise { + const statePath = path.join(repoRoot, WORKSPACES_STATE_RELATIVE_PATH); + await mkdir(path.dirname(statePath), { recursive: true }); + await writeFile(statePath, `${JSON.stringify(state, null, 2)}\n`, "utf8"); +} + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function isFileNotFoundError(error: unknown): boolean { + return ( + error instanceof Error && + "code" in error && + (error as NodeJS.ErrnoException).code === "ENOENT" + ); +} + +/** + * True when a read failed because the target is absent OR is not a readable + * file (a directory named like a manifest, or a non-directory path component). + * All three mean "no manifest to read here" for detection purposes. + */ +function isMissingFileError(error: unknown): boolean { + if (!(error instanceof Error) || !("code" in error)) { + return false; + } + const code = (error as NodeJS.ErrnoException).code; + return code === "ENOENT" || code === "EISDIR" || code === "ENOTDIR"; +} diff --git a/src/okf/index-sync.ts b/src/okf/index-sync.ts index c39d3a5c..fc231345 100644 --- a/src/okf/index-sync.ts +++ b/src/okf/index-sync.ts @@ -13,6 +13,8 @@ const EXCLUDED_FILES = new Set([ LOG_FILE, "_plan.md", "INSTRUCTIONS.md", + "workspaces.json", + ".workspaces-state.json", ]); /** diff --git a/test/code-mode.test.ts b/test/code-mode.test.ts index 11df0650..8d8cd8f5 100644 --- a/test/code-mode.test.ts +++ b/test/code-mode.test.ts @@ -112,6 +112,7 @@ describe("ensureCodeModeRepoSetup workflow", () => { expect(workflow).toContain("add-paths: |"); for (const managedPath of [ "openwiki", + "**/openwiki", "AGENTS.md", "CLAUDE.md", ".github/workflows/openwiki-update.yml", @@ -119,4 +120,31 @@ describe("ensureCodeModeRepoSetup workflow", () => { expect(workflow).toContain(managedPath); } }); + + test("non-recursive workflow uses the plain update command", async () => { + const repo = await createTempRepo(); + + await ensureCodeModeRepoSetup(repo); + + const workflow = await readIfPresent( + path.join(repo, ".github", "workflows", "openwiki-update.yml"), + ); + expect(workflow).toContain("run: openwiki code --update --print"); + expect(workflow).not.toContain("--recursive"); + }); + + test("recursive workflow reruns with --recursive", async () => { + const repo = await createTempRepo(); + + await ensureCodeModeRepoSetup(repo, { recursive: true }); + + const workflow = await readIfPresent( + path.join(repo, ".github", "workflows", "openwiki-update.yml"), + ); + expect(workflow).toContain( + "run: openwiki code --update --recursive --print", + ); + // Nested sub-wikis are staged in the PR. + expect(workflow).toContain("**/openwiki"); + }); }); diff --git a/test/commands.test.ts b/test/commands.test.ts index 397767a7..b3026657 100644 --- a/test/commands.test.ts +++ b/test/commands.test.ts @@ -445,3 +445,53 @@ describe("parseCommand — cron", () => { expect(result.kind).toBe("error"); }); }); + +describe("parseCommand — --recursive", () => { + test("recursive is undefined by default (auto-enable via manifest only)", () => { + expect(parseCommand(["--update"])).toMatchObject({ + kind: "run", + command: "update", + mode: "code", + recursive: undefined, + }); + }); + + test("--recursive forces recursion on", () => { + expect(parseCommand(["code", "--init", "--recursive"])).toMatchObject({ + kind: "run", + command: "init", + mode: "code", + recursive: true, + }); + expect(parseCommand(["--update", "--recursive=true"])).toMatchObject({ + recursive: true, + }); + }); + + test("--recursive=false and --no-recursive force recursion off", () => { + expect(parseCommand(["--update", "--recursive=false"])).toMatchObject({ + recursive: false, + }); + expect(parseCommand(["--update", "--no-recursive"])).toMatchObject({ + recursive: false, + }); + }); + + test("--recursive is rejected in personal mode", () => { + const result = parseCommand(["personal", "--update", "--recursive"]); + expect(result.kind).toBe("error"); + if (result.kind === "error") { + expect(result.message).toMatch(/only supported in code mode/); + } + }); + + test("--recursive=false is also rejected in personal mode", () => { + const result = parseCommand([ + "--mode", + "personal", + "--update", + "--no-recursive", + ]); + expect(result.kind).toBe("error"); + }); +}); diff --git a/test/docs-only-backend.test.ts b/test/docs-only-backend.test.ts index 08fa817c..fbe7b3b0 100644 --- a/test/docs-only-backend.test.ts +++ b/test/docs-only-backend.test.ts @@ -82,4 +82,31 @@ describe("OpenWikiLocalShellBackend", () => { readFile(path.join(rootDir, "notes.md"), "utf8"), ).resolves.toBe("ok"); }); + + test("rooted at a subproject, accepts /openwiki writes into that subproject", async () => { + // The recursive orchestrator re-roots the backend at packages/foo. The + // write guard keys off the virtual /openwiki prefix + cwd, so the same + // /openwiki/x.md path lands under packages/foo/openwiki with no guard change. + const repoRoot = await mkdtemp(path.join(os.tmpdir(), "openwiki-mono-")); + const subprojectDir = path.join(repoRoot, "packages", "foo"); + await mkdtemp(path.join(os.tmpdir(), "unused-")); + const backend = new OpenWikiLocalShellBackend({ + docsOnly: true, + outputMode: "repository", + rootDir: subprojectDir, + virtualMode: true, + }); + + const write = await backend.write("/openwiki/quickstart.md", "ok"); + expect(write).toEqual( + expect.objectContaining({ path: "/openwiki/quickstart.md" }), + ); + await expect( + readFile(path.join(subprojectDir, "openwiki", "quickstart.md"), "utf8"), + ).resolves.toBe("ok"); + + // A sibling escape is still refused just as at the repo root. + const refused = await backend.write("/src/code.ts", "nope"); + expect(refused.error).toContain("Refused path: /src/code.ts"); + }); }); diff --git a/test/orchestrator.test.ts b/test/orchestrator.test.ts new file mode 100644 index 00000000..f335a75a --- /dev/null +++ b/test/orchestrator.test.ts @@ -0,0 +1,288 @@ +import { mkdtemp, mkdir, rm, readFile, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, test, vi } from "vitest"; + +/** + * The orchestrator drives runs through the extracted runOne wrapper and does its + * once-per-process setup via loadOpenWikiEnv / syncBundledSkills / + * ensureCodeModeRepoSetup / resolveRunModel / refreshChatGptTokensIfNeeded. We + * mock all of those to record call order and counts without touching a model. + */ +const calls: string[] = []; + +const defaultRunOneImpl = ( + command: string, + cwd: string, + options: { recursionRole?: string }, +): Promise<{ command: string; model: string }> => { + calls.push(`runOne:${options.recursionRole ?? "none"}:${cwd}`); + return Promise.resolve({ command, model: "test-model" }); +}; + +const runOneMock = vi.fn(defaultRunOneImpl); +const resolveRunModelMock = vi.fn(() => { + calls.push("resolveRunModel"); + return { + provider: "openai", + modelId: "test-model", + providerRetryAttempts: 0, + }; +}); +const refreshMock = vi.fn((): Promise => { + calls.push("refreshChatGpt"); + return Promise.resolve(); +}); + +vi.mock("../src/agent/index.ts", () => ({ + runOne: runOneMock, + resolveRunModel: resolveRunModelMock, + refreshChatGptTokensIfNeeded: refreshMock, +})); + +const loadEnvMock = vi.fn((): Promise => { + calls.push("loadOpenWikiEnv"); + return Promise.resolve(); +}); +vi.mock("../src/env.ts", () => ({ + loadOpenWikiEnv: loadEnvMock, +})); + +const syncSkillsMock = vi.fn((): Promise => { + calls.push("syncBundledSkills"); + return Promise.resolve(); +}); +vi.mock("../src/agent/skills.ts", () => ({ + syncBundledSkills: syncSkillsMock, +})); + +const ensureSetupMock = vi.fn((): Promise => { + calls.push("ensureCodeModeRepoSetup"); + return Promise.resolve(); +}); +vi.mock("../src/code-mode.ts", () => ({ + ensureCodeModeRepoSetup: ensureSetupMock, +})); + +// Import AFTER mocks are registered. +const { runRecursiveOpenWiki } = + await import("../src/monorepo/orchestrator.ts"); +const { resolveWorkspaceRuns } = await import("../src/monorepo/workspaces.ts"); + +const tempDirs: string[] = []; + +async function createMonorepo(): Promise { + const repo = await mkdtemp(path.join(tmpdir(), "openwiki-orch-")); + tempDirs.push(repo); + for (const pkg of ["a", "b"]) { + await mkdir(path.join(repo, "packages", pkg), { recursive: true }); + await writeFile( + path.join(repo, "packages", pkg, "package.json"), + "{}", + "utf8", + ); + } + return repo; +} + +beforeEach(() => { + runOneMock.mockImplementation(defaultRunOneImpl); +}); + +afterEach(async () => { + calls.length = 0; + vi.clearAllMocks(); + await Promise.all( + tempDirs.splice(0).map((dir) => rm(dir, { force: true, recursive: true })), + ); +}); + +describe("runRecursiveOpenWiki ordering", () => { + test("runs subprojects then root, once each of the shared setup steps", async () => { + const repo = await createMonorepo(); + const manifest = { + version: 1, + workspaces: [{ path: "packages/a" }, { path: "packages/b" }], + } as const; + + const result = await runRecursiveOpenWiki( + "init", + repo, + { outputMode: "repository" }, + manifest, + ); + + // Shared once-only setup happens exactly once. + expect(loadEnvMock).toHaveBeenCalledTimes(1); + expect(syncSkillsMock).toHaveBeenCalledTimes(1); + expect(ensureSetupMock).toHaveBeenCalledTimes(1); + expect(resolveRunModelMock).toHaveBeenCalledTimes(1); + expect(refreshMock).toHaveBeenCalledTimes(1); + + // Order: setup → subproject a → subproject b → root. + const runOrder = calls.filter((c) => c.startsWith("runOne")); + expect(runOrder).toEqual([ + `runOne:subproject:${path.join(repo, "packages/a")}`, + `runOne:subproject:${path.join(repo, "packages/b")}`, + `runOne:root:${repo}`, + ]); + + // resolveRunModel and the refresh precede the first run. + expect(calls.indexOf("resolveRunModel")).toBeLessThan( + calls.findIndex((c) => c.startsWith("runOne")), + ); + + expect(result.subprojectResults).toHaveLength(2); + expect(result.rootResult.model).toBe("test-model"); + }); + + test("writes openwiki/workspaces.md BEFORE the root run", async () => { + const repo = await createMonorepo(); + + // Instrument runOne so we can observe the filesystem state at the root run. + let workspacesMdExistedAtRootRun = false; + runOneMock.mockImplementation( + async ( + command: string, + cwd: string, + options: { recursionRole?: string }, + ) => { + if (options.recursionRole === "root") { + workspacesMdExistedAtRootRun = await readFile( + path.join(repo, "openwiki", "workspaces.md"), + "utf8", + ) + .then(() => true) + .catch(() => false); + } + return { command, model: "test-model" }; + }, + ); + + await runRecursiveOpenWiki( + "init", + repo, + { outputMode: "repository" }, + { version: 1, workspaces: [{ path: "packages/a" }] }, + ); + + expect(workspacesMdExistedAtRootRun).toBe(true); + }); + + test("empty manifest falls back to a single plain run (no recursion role)", async () => { + const repo = await createMonorepo(); + + const result = await runRecursiveOpenWiki( + "update", + repo, + { outputMode: "repository" }, + { version: 1, workspaces: [] }, + ); + + const runOrder = calls.filter((c) => c.startsWith("runOne")); + expect(runOrder).toEqual([`runOne:none:${repo}`]); + expect(result.subprojectResults).toHaveLength(0); + }); + + test("skips a workspace with no documentable evidence", async () => { + const repo = await createMonorepo(); + await mkdir(path.join(repo, "packages", "empty"), { recursive: true }); + + const result = await runRecursiveOpenWiki( + "init", + repo, + { outputMode: "repository" }, + { + version: 1, + workspaces: [{ path: "packages/a" }, { path: "packages/empty" }], + }, + ); + + const runOrder = calls.filter( + (c) => c.startsWith("runOne") && c.includes("subproject"), + ); + expect(runOrder).toEqual([ + `runOne:subproject:${path.join(repo, "packages/a")}`, + ]); + expect(result.skippedWorkspaces.map((w) => w.path)).toEqual([ + "packages/empty", + ]); + }); + + test("continues past a failing subproject, still runs aggregation + root", async () => { + const repo = await createMonorepo(); + + // Fail the FIRST subproject (packages/a); packages/b and root must still run. + runOneMock.mockImplementation( + ( + command: string, + cwd: string, + options: { recursionRole?: string }, + ): Promise<{ command: string; model: string }> => { + calls.push(`runOne:${options.recursionRole ?? "none"}:${cwd}`); + if (cwd === path.join(repo, "packages/a")) { + return Promise.reject(new Error("boom in a")); + } + return Promise.resolve({ command, model: "test-model" }); + }, + ); + + const result = await runRecursiveOpenWiki( + "init", + repo, + { outputMode: "repository" }, + { + version: 1, + workspaces: [{ path: "packages/a" }, { path: "packages/b" }], + }, + ); + + // The failure is collected, not thrown. + expect(result.failedWorkspaces).toEqual([ + { path: "packages/a", error: "boom in a" }, + ]); + // packages/b succeeded and the root still ran. + expect(result.subprojectResults).toHaveLength(1); + const runOrder = calls.filter((c) => c.startsWith("runOne")); + expect(runOrder).toContain( + `runOne:subproject:${path.join(repo, "packages/b")}`, + ); + expect(runOrder).toContain(`runOne:root:${repo}`); + + // Aggregation was written and excludes the failed subproject. + const workspacesMd = await readFile( + path.join(repo, "openwiki", "workspaces.md"), + "utf8", + ); + expect(workspacesMd).toContain("packages/b/openwiki/quickstart.md"); + expect(workspacesMd).not.toContain("packages/a/openwiki/quickstart.md"); + }); +}); + +describe("writeRootAggregation content", () => { + test("aggregation links down to each subproject quickstart with OKF front matter", async () => { + const repo = await createMonorepo(); + const { writeRootAggregation } = + await import("../src/monorepo/orchestrator.ts"); + const plan = resolveWorkspaceRuns(repo, { + version: 1, + workspaces: [ + { path: "packages/a", name: "Alpha", goal: "the alpha pkg" }, + { path: "packages/b" }, + ], + }); + + await writeRootAggregation(repo, plan); + + const content = await readFile( + path.join(repo, "openwiki", "workspaces.md"), + "utf8", + ); + expect(content).toMatch(/^---\ntype: Reference/); + expect(content).toContain("[Alpha](../packages/a/openwiki/quickstart.md)"); + expect(content).toContain("the alpha pkg"); + expect(content).toContain( + "[packages/b](../packages/b/openwiki/quickstart.md)", + ); + }); +}); diff --git a/test/prompt.test.ts b/test/prompt.test.ts index 45050f03..2fe85b31 100644 --- a/test/prompt.test.ts +++ b/test/prompt.test.ts @@ -1,5 +1,6 @@ import { describe, expect, test } from "vitest"; -import { createSystemPrompt } from "../src/agent/prompt.ts"; +import { createSystemPrompt, createUserPrompt } from "../src/agent/prompt.ts"; +import type { RunContext } from "../src/agent/types.ts"; /** * Guards against the 0.2 regression where the shared "Canonical wiki location" @@ -63,3 +64,62 @@ describe("createSystemPrompt openwiki_generated enrichment guidance", () => { }); } }); + +describe("createSystemPrompt recursion roles", () => { + test("subproject role scopes to one subproject and forbids siblings", () => { + const prompt = createSystemPrompt("init", "repository", "subproject"); + expect(prompt).toContain("Monorepo subproject scope"); + expect(prompt).toMatch(/scoped to ONE subproject/); + expect(prompt).toMatch(/document, read into, or write to sibling/i); + }); + + test("root role links down and does not deep-document subtrees", () => { + const prompt = createSystemPrompt("init", "repository", "root"); + expect(prompt).toContain("Monorepo root scope"); + expect(prompt).toMatch(/link DOWN/); + expect(prompt).toContain("openwiki/workspaces.md"); + expect(prompt).toMatch(/Do NOT deep-document/); + }); + + test("absent role adds no recursion section (backward compatible)", () => { + const prompt = createSystemPrompt("init", "repository"); + expect(prompt).not.toContain("Monorepo subproject scope"); + expect(prompt).not.toContain("Monorepo root scope"); + }); +}); + +describe("createUserPrompt recursion reminders", () => { + const context: RunContext = { + lastUpdate: null, + gitSummary: "(git)", + wikiGoal: undefined, + }; + + test("subproject reminder appears in the init user prompt", () => { + const prompt = createUserPrompt( + "init", + context, + null, + "repository", + "subproject", + ); + expect(prompt).toMatch(/documenting a single subproject/); + }); + + test("root reminder appears in the update user prompt", () => { + const prompt = createUserPrompt( + "update", + context, + null, + "repository", + "root", + ); + expect(prompt).toMatch(/this is the monorepo root/i); + expect(prompt).toContain("openwiki/workspaces.md"); + }); + + test("absent role leaves the user prompt unchanged", () => { + const prompt = createUserPrompt("init", context, null, "repository"); + expect(prompt).not.toMatch(/Recursive monorepo run/); + }); +}); diff --git a/test/recursion-activation.test.ts b/test/recursion-activation.test.ts new file mode 100644 index 00000000..9aac0f0b --- /dev/null +++ b/test/recursion-activation.test.ts @@ -0,0 +1,124 @@ +import { mkdtemp, mkdir, rm, writeFile, readFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, test } from "vitest"; +import { + resolveRecursionActivation, + resolveWorkspaceRuns, +} from "../src/monorepo/workspaces.ts"; + +const tempDirs: string[] = []; + +async function createRepo(): Promise { + const repo = await mkdtemp(path.join(tmpdir(), "openwiki-activate-")); + tempDirs.push(repo); + return repo; +} + +afterEach(async () => { + await Promise.all( + tempDirs.splice(0).map((dir) => rm(dir, { force: true, recursive: true })), + ); +}); + +describe("resolveRecursionActivation", () => { + test("recurses when a manifest exists (default flag)", async () => { + const repo = await createRepo(); + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + await writeFile( + path.join(repo, "openwiki", "workspaces.json"), + JSON.stringify({ version: 1, workspaces: [{ path: "packages/a" }] }), + "utf8", + ); + + const activation = await resolveRecursionActivation(repo, undefined); + expect(activation.kind).toBe("recurse"); + if (activation.kind === "recurse") { + expect(activation.autoDetected).toBe(false); + expect(activation.manifest.workspaces).toHaveLength(1); + } + }); + + test("--recursive=false forces plain even with a manifest", async () => { + const repo = await createRepo(); + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + await writeFile( + path.join(repo, "openwiki", "workspaces.json"), + JSON.stringify({ version: 1, workspaces: [{ path: "packages/a" }] }), + "utf8", + ); + + const activation = await resolveRecursionActivation(repo, false); + expect(activation.kind).toBe("plain"); + }); + + test("default with no manifest is a plain run", async () => { + const repo = await createRepo(); + const activation = await resolveRecursionActivation(repo, undefined); + expect(activation.kind).toBe("plain"); + }); + + test("--recursive with no manifest auto-detects and writes a manifest", async () => { + const repo = await createRepo(); + await writeFile( + path.join(repo, "package.json"), + JSON.stringify({ workspaces: ["packages/*"] }), + "utf8", + ); + await mkdir(path.join(repo, "packages", "a"), { recursive: true }); + await writeFile(path.join(repo, "packages", "a", "package.json"), "{}"); + + const activation = await resolveRecursionActivation(repo, true); + expect(activation.kind).toBe("recurse"); + if (activation.kind === "recurse") { + expect(activation.autoDetected).toBe(true); + } + + // The manifest was written for the user to review. + const written = JSON.parse( + await readFile(path.join(repo, "openwiki", "workspaces.json"), "utf8"), + ) as { version: number; workspaces: { path: string }[] }; + expect(written.version).toBe(1); + expect(written.workspaces).toEqual([{ path: "packages/a" }]); + }); + + test("--recursive with no manifest and no detectable workspaces falls back to plain", async () => { + const repo = await createRepo(); + const activation = await resolveRecursionActivation(repo, true); + expect(activation.kind).toBe("plain"); + if (activation.kind === "plain") { + expect(activation.reason).toMatch(/no monorepo workspaces were detected/); + } + }); + + test("auto-detect on a packages/** repo writes a manifest that resolves cleanly (no self-poison)", async () => { + const repo = await createRepo(); + await writeFile( + path.join(repo, "package.json"), + JSON.stringify({ workspaces: ["packages/**"] }), + "utf8", + ); + await mkdir(path.join(repo, "packages", "a"), { recursive: true }); + await writeFile(path.join(repo, "packages", "a", "package.json"), "{}"); + await mkdir(path.join(repo, "packages", "b", "sub"), { recursive: true }); + await writeFile( + path.join(repo, "packages", "b", "sub", "package.json"), + "{}", + ); + + const activation = await resolveRecursionActivation(repo, true); + expect(activation.kind).toBe("recurse"); + + // A subsequent DEFAULT run (no flag) reads the written manifest and must not + // throw — this is the "stuck repo" regression. + const second = await resolveRecursionActivation(repo, undefined); + expect(second.kind).toBe("recurse"); + if (second.kind === "recurse") { + const plan = resolveWorkspaceRuns(repo, second.manifest); + expect(plan.runs.map((run) => run.relativePath).sort()).toEqual([ + "packages/a", + "packages/b/sub", + ]); + } + }); +}); diff --git a/test/workspace-git-scope.test.ts b/test/workspace-git-scope.test.ts new file mode 100644 index 00000000..477873ea --- /dev/null +++ b/test/workspace-git-scope.test.ts @@ -0,0 +1,292 @@ +import { + mkdtemp, + mkdir, + rm, + writeFile, + readFile, + chmod, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; +import { afterEach, describe, expect, test } from "vitest"; +import { createRunContext, getUpdateNoopStatus } from "../src/agent/utils.ts"; + +const execFileAsync = promisify(execFile); +const tempRepos: string[] = []; + +async function git(cwd: string, args: string[]): Promise { + const { stdout } = await execFileAsync("git", args, { cwd }); + return stdout.trim(); +} + +/** + * Builds a monorepo with two subprojects, each with its own openwiki/ sub-wiki + * and a .last-update.json pinned to the initial commit. + */ +async function createMonorepo(): Promise<{ + repo: string; + initialHead: string; +}> { + const repo = await mkdtemp(path.join(tmpdir(), "openwiki-scope-")); + tempRepos.push(repo); + await git(repo, ["init"]); + await git(repo, ["config", "user.email", "test@example.com"]); + await git(repo, ["config", "user.name", "OpenWiki Test"]); + + for (const pkg of ["foo", "bar"]) { + await mkdir(path.join(repo, "packages", pkg, "src"), { recursive: true }); + await mkdir(path.join(repo, "packages", pkg, "openwiki"), { + recursive: true, + }); + await writeFile( + path.join(repo, "packages", pkg, "src", "code.ts"), + `export const ${pkg} = 1;\n`, + "utf8", + ); + await writeFile( + path.join(repo, "packages", pkg, "openwiki", "quickstart.md"), + `# ${pkg}\n`, + "utf8", + ); + } + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + await writeFile(path.join(repo, "openwiki", "quickstart.md"), "# root\n"); + await writeFile(path.join(repo, "README.md"), "# root\n", "utf8"); + await git(repo, ["add", "."]); + await git(repo, ["commit", "-m", "initial"]); + const initialHead = await git(repo, ["rev-parse", "HEAD"]); + + return { repo, initialHead }; +} + +async function writeLastUpdate(dir: string, gitHead: string): Promise { + await mkdir(path.join(dir, "openwiki"), { recursive: true }); + await writeFile( + path.join(dir, "openwiki", ".last-update.json"), + `${JSON.stringify({ + updatedAt: new Date().toISOString(), + command: "update", + gitHead, + model: "test-model", + })}\n`, + "utf8", + ); +} + +afterEach(async () => { + await Promise.all( + tempRepos + .splice(0) + .map((repo) => rm(repo, { force: true, recursive: true })), + ); +}); + +describe("subproject-scoped git evidence (Variant B)", () => { + test("createGitSummary scoped to foo returns only foo-subtree paths", async () => { + const { repo, initialHead } = await createMonorepo(); + const fooDir = path.join(repo, "packages", "foo"); + await writeLastUpdate(fooDir, initialHead); + + // Commit changes in BOTH foo and bar. + await writeFile( + path.join(fooDir, "src", "code.ts"), + "export const foo = 2;\n", + ); + await writeFile( + path.join(repo, "packages", "bar", "src", "code.ts"), + "export const bar = 2;\n", + ); + await git(repo, ["add", "."]); + await git(repo, ["commit", "-m", "change foo and bar"]); + + const context = await createRunContext("update", fooDir, "repository", { + mode: "subproject", + }); + + // Foo's own source is present, subproject-relative (no packages/foo prefix). + expect(context.gitSummary).toContain("src/code.ts"); + // Bar's source must NOT leak into foo's evidence. + expect(context.gitSummary).not.toContain("packages/bar"); + expect(context.gitSummary).not.toContain("bar/src/code.ts"); + }); + + test("getUpdateNoopStatus(foo) SKIPS when only bar changed", async () => { + const { repo, initialHead } = await createMonorepo(); + const fooDir = path.join(repo, "packages", "foo"); + await writeLastUpdate(fooDir, initialHead); + + await writeFile( + path.join(repo, "packages", "bar", "src", "code.ts"), + "export const bar = 2;\n", + ); + await git(repo, ["add", "."]); + await git(repo, ["commit", "-m", "change bar only"]); + + const status = await getUpdateNoopStatus(fooDir, { mode: "subproject" }); + expect(status.shouldSkip).toBe(true); + }); + + test("getUpdateNoopStatus(foo) RUNS when foo changed", async () => { + const { repo, initialHead } = await createMonorepo(); + const fooDir = path.join(repo, "packages", "foo"); + await writeLastUpdate(fooDir, initialHead); + + await writeFile( + path.join(fooDir, "src", "code.ts"), + "export const foo = 2;\n", + ); + await git(repo, ["add", "."]); + await git(repo, ["commit", "-m", "change foo"]); + + const status = await getUpdateNoopStatus(fooDir, { mode: "subproject" }); + expect(status.shouldSkip).toBe(false); + }); + + test("getUpdateNoopStatus(foo) SKIPS when only foo's own openwiki changed", async () => { + const { repo, initialHead } = await createMonorepo(); + const fooDir = path.join(repo, "packages", "foo"); + await writeLastUpdate(fooDir, initialHead); + + await writeFile( + path.join(fooDir, "openwiki", "quickstart.md"), + "# foo updated\n", + ); + await git(repo, ["add", "."]); + await git(repo, ["commit", "-m", "update foo docs"]); + + const status = await getUpdateNoopStatus(fooDir, { mode: "subproject" }); + expect(status.shouldSkip).toBe(true); + }); +}); + +describe("root-excluding-nested git evidence (D1)", () => { + test("root summary excludes nested openwiki but keeps root openwiki", async () => { + const { repo, initialHead } = await createMonorepo(); + await writeLastUpdate(repo, initialHead); + + // Simulate subproject runs dirtying nested wikis, plus a real source change + // and a root-wiki change. + await writeFile( + path.join(repo, "packages", "foo", "openwiki", "quickstart.md"), + "# foo regenerated\n", + ); + await writeFile( + path.join(repo, "packages", "foo", "src", "code.ts"), + "export const foo = 3;\n", + ); + await writeFile( + path.join(repo, "openwiki", "quickstart.md"), + "# root regenerated\n", + ); + + const context = await createRunContext("update", repo, "repository", { + mode: "root-excluding-nested", + }); + + // Nested sub-wiki churn is excluded from the root's evidence. + expect(context.gitSummary).not.toContain("packages/foo/openwiki"); + // Real source change is still visible. + expect(context.gitSummary).toContain("packages/foo/src/code.ts"); + // Root's own wiki is preserved (it's the doc target and legitimate context). + expect(context.gitSummary).toContain("openwiki/quickstart.md"); + }); +}); + +/** + * Installs a fake `git` on PATH that appends its argv to a log file and then + * delegates to the real git. Lets a test assert the EXACT argument vectors util + * functions pass, proving the scope-absent contract byte-for-byte. + */ +async function withGitArgvCapture( + fn: (getArgs: () => Promise) => Promise, +): Promise { + const shimDir = await mkdtemp(path.join(tmpdir(), "openwiki-gitshim-")); + tempRepos.push(shimDir); + const logFile = path.join(shimDir, "argv.log"); + const realGit = (await execFileAsync("which", ["git"])).stdout.trim(); + const shimPath = path.join(shimDir, "git"); + await writeFile( + shimPath, + `#!/usr/bin/env bash\nprintf '%s\\0' "$@" >> ${JSON.stringify( + logFile, + )}\nprintf '\\n' >> ${JSON.stringify( + logFile, + )}\nexec ${JSON.stringify(realGit)} "$@"\n`, + "utf8", + ); + await chmod(shimPath, 0o755); + + const originalPath = process.env.PATH; + process.env.PATH = `${shimDir}${path.delimiter}${originalPath ?? ""}`; + try { + return await fn(async () => { + const raw = await readFile(logFile, "utf8").catch(() => ""); + return raw + .split("\n") + .filter((line) => line.length > 0) + .map((line) => line.split("\0").filter(Boolean)); + }); + } finally { + process.env.PATH = originalPath; + } +} + +describe("scope-absent commands are byte-identical to today", () => { + test("no scope issues no --relative and no pathspec", async () => { + const { repo, initialHead } = await createMonorepo(); + await writeLastUpdate(repo, initialHead); + await writeFile(path.join(repo, "README.md"), "# root changed\n"); + await git(repo, ["add", "."]); + await git(repo, ["commit", "-m", "change readme"]); + + await withGitArgvCapture(async (getArgs) => { + await createRunContext("update", repo, "repository"); + const calls = await getArgs(); + + // Every captured git invocation is prefixed with --no-pager (runGit). + const evidence = calls.filter((call) => call.includes("--no-pager")); + expect(evidence.length).toBeGreaterThan(0); + + for (const call of evidence) { + // The scope-absent contract: no --relative and no `--` pathspec. + expect(call).not.toContain("--relative"); + expect(call).not.toContain("--"); + } + + // status --short is exactly ["--no-pager","status","--short"]. + const statusCall = evidence.find( + (call) => call[1] === "status" && call[2] === "--short", + ); + expect(statusCall).toEqual(["--no-pager", "status", "--short"]); + }); + }); + + test("subproject scope adds --relative and `-- .` on diff", async () => { + const { repo, initialHead } = await createMonorepo(); + const fooDir = path.join(repo, "packages", "foo"); + await writeLastUpdate(fooDir, initialHead); + + await withGitArgvCapture(async (getArgs) => { + await createRunContext("update", fooDir, "repository", { + mode: "subproject", + }); + const calls = await getArgs(); + const diffCall = calls.find( + (call) => call.includes("diff") && call.includes("--name-status"), + ); + expect(diffCall).toContain("--relative"); + expect(diffCall).toContain("--"); + expect(diffCall).toContain("."); + + // status is scoped with `-- .` but NOT --relative. + const statusCall = calls.find( + (call) => call[1] === "status" && call.includes("--short"), + ); + expect(statusCall).toContain("--"); + expect(statusCall).toContain("."); + expect(statusCall).not.toContain("--relative"); + }); + }); +}); diff --git a/test/workspaces-aggregation.test.ts b/test/workspaces-aggregation.test.ts new file mode 100644 index 00000000..b3e3322e --- /dev/null +++ b/test/workspaces-aggregation.test.ts @@ -0,0 +1,69 @@ +import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, test } from "vitest"; +import { OpenWikiLocalShellBackend } from "../src/agent/docs-only-backend.ts"; +import { + migrateWikiToOkf, + synchronizeWikiIndexes, +} from "../src/okf/index-sync.ts"; +import { writeRootAggregation } from "../src/monorepo/orchestrator.ts"; +import { resolveWorkspaceRuns } from "../src/monorepo/workspaces.ts"; + +const tempDirs: string[] = []; + +async function createRepo(): Promise { + const repo = await mkdtemp(path.join(os.tmpdir(), "openwiki-agg-")); + tempDirs.push(repo); + await mkdir(path.join(repo, "packages", "a"), { recursive: true }); + return repo; +} + +afterEach(async () => { + await Promise.all( + tempDirs.splice(0).map((dir) => rm(dir, { force: true, recursive: true })), + ); +}); + +describe("workspaces.md aggregation + index sync (F3)", () => { + test("workspaces.md is linked into the root openwiki/index.md", async () => { + const repo = await createRepo(); + // A root quickstart so the root wiki has a normal page too. + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + await writeFile( + path.join(repo, "openwiki", "quickstart.md"), + `---\ntype: Reference\ntitle: Quickstart\ndescription: Root.\n---\n\n# Quickstart\n`, + "utf8", + ); + + const plan = resolveWorkspaceRuns(repo, { + version: 1, + workspaces: [{ path: "packages/a", name: "Alpha" }], + }); + await writeRootAggregation(repo, plan); + + const backend = new OpenWikiLocalShellBackend({ + docsOnly: true, + outputMode: "repository", + rootDir: repo, + virtualMode: true, + }); + + // migrateWikiToOkf (beforeAgent) must not rewrite the generated page's type. + await migrateWikiToOkf(backend, "repository"); + const afterMigrate = await readFile( + path.join(repo, "openwiki", "workspaces.md"), + "utf8", + ); + expect(afterMigrate).toMatch(/type: Reference/); + expect(afterMigrate).not.toContain("openwiki_generated"); + + // synchronizeWikiIndexes (afterAgent) links workspaces.md into index.md. + await synchronizeWikiIndexes(backend, "repository"); + const rootIndex = await readFile( + path.join(repo, "openwiki", "index.md"), + "utf8", + ); + expect(rootIndex).toContain("workspaces.md"); + }); +}); diff --git a/test/workspaces-state.test.ts b/test/workspaces-state.test.ts new file mode 100644 index 00000000..da48af3c --- /dev/null +++ b/test/workspaces-state.test.ts @@ -0,0 +1,106 @@ +import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, test } from "vitest"; +import { + createOpenWikiContentSnapshot, + persistRunMetadataIfChanged, +} from "../src/agent/utils.ts"; +import { OpenWikiLocalShellBackend } from "../src/agent/docs-only-backend.ts"; +import { synchronizeWikiIndexes } from "../src/okf/index-sync.ts"; +import { + readWorkspacesState, + writeWorkspacesState, +} from "../src/monorepo/workspaces.ts"; + +const tempDirs: string[] = []; + +async function createRepo(): Promise { + const repo = await mkdtemp(path.join(os.tmpdir(), "openwiki-wsstate-")); + tempDirs.push(repo); + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + await writeFile( + path.join(repo, "openwiki", "quickstart.md"), + `---\ntype: Reference\ntitle: Quickstart\ndescription: Root.\n---\n\n# Quickstart\n`, + "utf8", + ); + return repo; +} + +afterEach(async () => { + await Promise.all( + tempDirs.splice(0).map((dir) => rm(dir, { force: true, recursive: true })), + ); +}); + +describe(".workspaces-state.json exclusion", () => { + test("does not change the content snapshot (no spurious metadata writes)", async () => { + const repo = await createRepo(); + const before = await createOpenWikiContentSnapshot(repo, "repository"); + + await writeWorkspacesState(repo, { + version: 1, + workspaces: { "packages/a": { gitHead: "abc", updatedAt: "now" } }, + }); + + const after = await createOpenWikiContentSnapshot(repo, "repository"); + expect(after).toBe(before); + + // And persistRunMetadataIfChanged treats the wiki as unchanged. + const written = await persistRunMetadataIfChanged( + "update", + repo, + "test-model", + "repository", + before, + ); + expect(written).toBe(false); + }); + + test("is not linked into the generated openwiki/index.md", async () => { + const repo = await createRepo(); + await writeWorkspacesState(repo, { + version: 1, + workspaces: { "packages/a": { gitHead: "abc", updatedAt: "now" } }, + }); + // The manifest sibling must also be excluded. + await writeFile( + path.join(repo, "openwiki", "workspaces.json"), + JSON.stringify({ version: 1, workspaces: [{ path: "packages/a" }] }), + "utf8", + ); + + const backend = new OpenWikiLocalShellBackend({ + docsOnly: true, + outputMode: "repository", + rootDir: repo, + virtualMode: true, + }); + await synchronizeWikiIndexes(backend, "repository"); + + const index = await readFile( + path.join(repo, "openwiki", "index.md"), + "utf8", + ); + expect(index).not.toContain(".workspaces-state.json"); + expect(index).not.toContain("workspaces.json"); + }); + + test("read/write round-trips per-subproject gitHead", async () => { + const repo = await createRepo(); + expect((await readWorkspacesState(repo)).workspaces).toEqual({}); + + await writeWorkspacesState(repo, { + version: 1, + workspaces: { + "packages/a": { + gitHead: "deadbeef", + updatedAt: "2026-01-01T00:00:00Z", + }, + }, + }); + + const state = await readWorkspacesState(repo); + expect(state.workspaces["packages/a"].gitHead).toBe("deadbeef"); + }); +}); diff --git a/test/workspaces.test.ts b/test/workspaces.test.ts new file mode 100644 index 00000000..2bf62ab9 --- /dev/null +++ b/test/workspaces.test.ts @@ -0,0 +1,1041 @@ +import { mkdtemp, mkdir, rm, writeFile, symlink } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, test } from "vitest"; +import { + detectWorkspaces, + getWorkspaceSkipReason, + normalizeManifest, + readWorkspaceManifest, + resolveWorkspaceRuns, + type WorkspaceManifest, +} from "../src/monorepo/workspaces.ts"; + +const tempDirs: string[] = []; + +async function createTempRepo(): Promise { + const repo = await mkdtemp(path.join(tmpdir(), "openwiki-workspaces-")); + tempDirs.push(repo); + return repo; +} + +afterEach(async () => { + await Promise.all( + tempDirs.splice(0).map((dir) => rm(dir, { force: true, recursive: true })), + ); +}); + +describe("readWorkspaceManifest", () => { + test("returns null when no manifest exists", async () => { + const repo = await createTempRepo(); + expect(await readWorkspaceManifest(repo)).toBeNull(); + }); + + test("reads a valid manifest with goals and root brief", async () => { + const repo = await createTempRepo(); + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + await writeFile( + path.join(repo, "openwiki", "workspaces.json"), + JSON.stringify({ + version: 1, + workspaces: [{ path: "packages/core", goal: "the core", name: "Core" }], + root: { goal: "root brief" }, + }), + "utf8", + ); + + const manifest = await readWorkspaceManifest(repo); + expect(manifest).not.toBeNull(); + expect(manifest?.workspaces[0]).toEqual({ + path: "packages/core", + goal: "the core", + name: "Core", + }); + expect(manifest?.root?.goal).toBe("root brief"); + }); + + test("throws on malformed JSON", async () => { + const repo = await createTempRepo(); + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + await writeFile( + path.join(repo, "openwiki", "workspaces.json"), + "{ not json", + "utf8", + ); + await expect(readWorkspaceManifest(repo)).rejects.toThrow(/not valid JSON/); + }); +}); + +describe("normalizeManifest", () => { + test("rejects unsupported versions", () => { + expect(() => normalizeManifest({ version: 2, workspaces: [] })).toThrow( + /version 2 is unsupported/, + ); + }); + + test("rejects a missing workspaces array", () => { + expect(() => normalizeManifest({ version: 1 })).toThrow( + /must contain a `workspaces` array/, + ); + }); + + test("rejects a workspace entry without a string path", () => { + expect(() => + normalizeManifest({ version: 1, workspaces: [{ goal: "x" }] }), + ).toThrow(/must be an object with a string `path`/); + }); + + test("accepts a missing version (defaults to 1)", () => { + const manifest = normalizeManifest({ workspaces: [{ path: "a" }] }); + expect(manifest.version).toBe(1); + }); +}); + +describe("resolveWorkspaceRuns", () => { + const repoRoot = "/repo"; + + function manifest( + workspaces: WorkspaceManifest["workspaces"], + root?: WorkspaceManifest["root"], + ): WorkspaceManifest { + return { version: 1, workspaces, ...(root ? { root } : {}) }; + } + + test("normalizes paths and resolves absolute paths", () => { + const plan = resolveWorkspaceRuns( + repoRoot, + manifest([{ path: "packages/core/" }], { goal: " root " }), + ); + expect(plan.runs).toHaveLength(1); + expect(plan.runs[0].relativePath).toBe("packages/core"); + expect(plan.runs[0].absolutePath).toBe(path.resolve("/repo/packages/core")); + expect(plan.rootGoal).toBe("root"); + }); + + test("dedupes repeated paths", () => { + const plan = resolveWorkspaceRuns( + repoRoot, + manifest([{ path: "packages/a" }, { path: "packages/a/" }]), + ); + expect(plan.runs).toHaveLength(1); + }); + + test("rejects a workspace equal to the repo root", () => { + expect(() => + resolveWorkspaceRuns(repoRoot, manifest([{ path: "." }])), + ).toThrow(/may not be the repository root/); + expect(() => + resolveWorkspaceRuns(repoRoot, manifest([{ path: "" }])), + ).toThrow(/may not be the repository root/); + }); + + test("rejects absolute paths", () => { + expect(() => + resolveWorkspaceRuns(repoRoot, manifest([{ path: "/etc" }])), + ).toThrow(/absolute paths are not allowed/); + }); + + test("rejects .. traversal", () => { + expect(() => + resolveWorkspaceRuns(repoRoot, manifest([{ path: "../evil" }])), + ).toThrow(/may not escape the repository root/); + expect(() => + resolveWorkspaceRuns(repoRoot, manifest([{ path: "packages/../../x" }])), + ).toThrow(/may not escape the repository root/); + }); + + test("rejects nested/overlapping workspaces", () => { + expect(() => + resolveWorkspaceRuns( + repoRoot, + manifest([{ path: "packages/foo" }, { path: "packages/foo/bar" }]), + ), + ).toThrow(/is an ancestor of/); + }); + + test("allows sibling workspaces with a shared prefix name", () => { + const plan = resolveWorkspaceRuns( + repoRoot, + manifest([{ path: "packages/foo" }, { path: "packages/foobar" }]), + ); + expect(plan.runs).toHaveLength(2); + }); + + test("empty workspaces produce an empty plan", () => { + const plan = resolveWorkspaceRuns(repoRoot, manifest([])); + expect(plan.runs).toHaveLength(0); + }); +}); + +describe("getWorkspaceSkipReason", () => { + test("returns null for a workspace with a package.json", async () => { + const repo = await createTempRepo(); + await mkdir(path.join(repo, "packages/foo"), { recursive: true }); + await writeFile(path.join(repo, "packages/foo/package.json"), "{}", "utf8"); + const [run] = resolveWorkspaceRuns(repo, { + version: 1, + workspaces: [{ path: "packages/foo" }], + }).runs; + expect(await getWorkspaceSkipReason(repo, run)).toBeNull(); + }); + + test("skips a workspace with no manifest, no instructions, no goal, no source", async () => { + const repo = await createTempRepo(); + await mkdir(path.join(repo, "packages/empty"), { recursive: true }); + const [run] = resolveWorkspaceRuns(repo, { + version: 1, + workspaces: [{ path: "packages/empty" }], + }).runs; + expect(await getWorkspaceSkipReason(repo, run)).toMatch(/no source files/); + }); + + test("does not skip when a goal is provided even with no source", async () => { + const repo = await createTempRepo(); + await mkdir(path.join(repo, "packages/empty"), { recursive: true }); + const [run] = resolveWorkspaceRuns(repo, { + version: 1, + workspaces: [{ path: "packages/empty", goal: "document me" }], + }).runs; + expect(await getWorkspaceSkipReason(repo, run)).toBeNull(); + }); + + test("does not skip when openwiki/INSTRUCTIONS.md is present", async () => { + const repo = await createTempRepo(); + await mkdir(path.join(repo, "packages/foo/openwiki"), { recursive: true }); + await writeFile( + path.join(repo, "packages/foo/openwiki/INSTRUCTIONS.md"), + "brief\n", + "utf8", + ); + const [run] = resolveWorkspaceRuns(repo, { + version: 1, + workspaces: [{ path: "packages/foo" }], + }).runs; + expect(await getWorkspaceSkipReason(repo, run)).toBeNull(); + }); + + test("skips a missing directory", async () => { + const repo = await createTempRepo(); + const [run] = resolveWorkspaceRuns(repo, { + version: 1, + workspaces: [{ path: "packages/ghost" }], + }).runs; + expect(await getWorkspaceSkipReason(repo, run)).toMatch(/does not exist/); + }); + + test("rejects a symlink that escapes the repository", async () => { + const repo = await createTempRepo(); + const outside = await createTempRepo(); + await mkdir(path.join(repo, "packages"), { recursive: true }); + await symlink(outside, path.join(repo, "packages/escape")); + const [run] = resolveWorkspaceRuns(repo, { + version: 1, + workspaces: [{ path: "packages/escape" }], + }).runs; + expect(await getWorkspaceSkipReason(repo, run)).toMatch(/symlink escape/); + }); +}); + +describe("detectWorkspaces", () => { + test("returns [] with no monorepo manifests", async () => { + const repo = await createTempRepo(); + expect(await detectWorkspaces(repo)).toEqual([]); + }); + + test("does not wedge when a manifest name is a directory (EISDIR)", async () => { + const repo = await createTempRepo(); + // A directory named like a manifest makes readFile throw EISDIR; detection + // must treat it as "no manifest here" rather than propagating the error. + await mkdir(path.join(repo, "pom.xml"), { recursive: true }); + await mkdir(path.join(repo, "package.json"), { recursive: true }); + await expect(detectWorkspaces(repo)).resolves.toEqual([]); + }); + + test("expands package.json workspaces globs into existing dirs", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "package.json"), + JSON.stringify({ workspaces: ["packages/*"] }), + "utf8", + ); + await mkdir(path.join(repo, "packages/a"), { recursive: true }); + await mkdir(path.join(repo, "packages/b"), { recursive: true }); + await writeFile(path.join(repo, "packages/a/package.json"), "{}", "utf8"); + await writeFile(path.join(repo, "packages/b/package.json"), "{}", "utf8"); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path).sort()).toEqual([ + "packages/a", + "packages/b", + ]); + }); + + test("expands pnpm-workspace.yaml packages", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "pnpm-workspace.yaml"), + "packages:\n - 'apps/*'\n", + "utf8", + ); + await mkdir(path.join(repo, "apps/web"), { recursive: true }); + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path)).toEqual(["apps/web"]); + }); + + test("expands Cargo.toml [workspace] members", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "Cargo.toml"), + '[workspace]\nmembers = ["crates/one", "crates/two"]\n', + "utf8", + ); + await mkdir(path.join(repo, "crates/one"), { recursive: true }); + await mkdir(path.join(repo, "crates/two"), { recursive: true }); + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path).sort()).toEqual([ + "crates/one", + "crates/two", + ]); + }); + + test("expands go.work use directives", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "go.work"), + "go 1.22\n\nuse (\n\t./svc/a\n\t./svc/b\n)\n", + "utf8", + ); + await mkdir(path.join(repo, "svc/a"), { recursive: true }); + await mkdir(path.join(repo, "svc/b"), { recursive: true }); + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path).sort()).toEqual([ + "svc/a", + "svc/b", + ]); + }); + + test("expands ** globs to leaf package dirs, not intermediates", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "package.json"), + JSON.stringify({ workspaces: ["packages/**"] }), + "utf8", + ); + // A nested package inside a container that is itself NOT a package. Only the + // leaf package (has a manifest) should be detected; the container must not. + await mkdir(path.join(repo, "packages/group/nested"), { recursive: true }); + await writeFile( + path.join(repo, "packages/group/nested/package.json"), + "{}", + "utf8", + ); + + const detected = await detectWorkspaces(repo); + const paths = detected.map((entry) => entry.path); + expect(paths).toContain("packages/group/nested"); + expect(paths).not.toContain("packages/group"); + expect(paths).not.toContain("packages"); + }); + + test("common packages/* glob does not emit the container", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "package.json"), + JSON.stringify({ workspaces: ["packages/*"] }), + "utf8", + ); + await mkdir(path.join(repo, "packages/a"), { recursive: true }); + await writeFile(path.join(repo, "packages/a/package.json"), "{}", "utf8"); + + const detected = await detectWorkspaces(repo); + const paths = detected.map((entry) => entry.path); + expect(paths).toEqual(["packages/a"]); + expect(paths).not.toContain("packages"); + }); + + describe(".NET solution (.sln / .slnx)", () => { + test("expands classic .sln project directories", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "App.sln"), + [ + 'Microsoft Visual Studio Solution File, Format Version 12.00', + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Api", "src\\Api\\Api.csproj", "{11111111-1111-1111-1111-111111111111}"', + "EndProject", + 'Project("{F184B08F-C81C-45F6-A57F-5ABD9991F28F}") = "Lib", "src\\Lib\\Lib.vbproj", "{22222222-2222-2222-2222-222222222222}"', + "EndProject", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "src/Api"), { recursive: true }); + await mkdir(path.join(repo, "src/Lib"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path).sort()).toEqual([ + "src/Api", + "src/Lib", + ]); + }); + + test("skips solution-folder entries in a .sln", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "App.sln"), + [ + 'Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "SolutionItems", "SolutionItems", "{33333333-3333-3333-3333-333333333333}"', + "EndProject", + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Api", "src\\Api\\Api.csproj", "{11111111-1111-1111-1111-111111111111}"', + "EndProject", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "src/Api"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + const paths = detected.map((entry) => entry.path); + expect(paths).toEqual(["src/Api"]); + expect(paths).not.toContain("SolutionItems"); + }); + + test("expands .slnx entries and ignores ", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "App.slnx"), + [ + "", + ' ', + ' ', + " ", + ' ', + ' ', + "", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "src/Web"), { recursive: true }); + await mkdir(path.join(repo, "src/Core"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path).sort()).toEqual([ + "src/Core", + "src/Web", + ]); + }); + + test("parses a CRLF .sln (real solution files use CRLF)", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "App.sln"), + [ + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Api", "src\\Api\\Api.csproj", "{11111111-1111-1111-1111-111111111111}"', + "EndProject", + "", + ].join("\r\n"), + "utf8", + ); + await mkdir(path.join(repo, "src/Api"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path)).toEqual(["src/Api"]); + }); + + test("parses a .slnx with single-quoted Path", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "App.slnx"), + [ + "", + " ", + "", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "src/Api"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path)).toEqual(["src/Api"]); + }); + + test("malformed .sln returns no .NET workspaces", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "App.sln"), + "this is not a solution file at all", + "utf8", + ); + expect(await detectWorkspaces(repo)).toEqual([]); + }); + + test("coarsens an area's src/tests projects to the area root", async () => { + // The .NET DDD convention nests projects under an intermediate src/ or + // tests/ dir inside a product area. All three projects below belong to the + // ONE area `platform/admission`, so detection must collapse to it rather + // than emitting one workspace per .csproj. + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "App.sln"), + [ + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Api", "platform\\admission\\src\\Core.Admission.Api\\Core.Admission.Api.csproj", "{11111111-1111-1111-1111-111111111111}"', + "EndProject", + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Domain", "platform\\admission\\src\\Core.Admission.Domain\\Core.Admission.Domain.csproj", "{22222222-2222-2222-2222-222222222222}"', + "EndProject", + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "DomainTests", "platform\\admission\\tests\\Core.Admission.Domain.Tests\\Core.Admission.Domain.Tests.csproj", "{33333333-3333-3333-3333-333333333333}"', + "EndProject", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "platform/admission"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path)).toEqual([ + "platform/admission", + ]); + }); + + test("leaves a flat lib (project dir directly under a container) unchanged", async () => { + // A project whose immediate parent is neither src nor tests (a flatter lib + // like kernel/Core.Domain.Kernel) has nothing to trim and stays as-is. + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "App.sln"), + [ + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Kernel", "kernel\\Core.Domain.Kernel\\Core.Domain.Kernel.csproj", "{11111111-1111-1111-1111-111111111111}"', + "EndProject", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "kernel/Core.Domain.Kernel"), { + recursive: true, + }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path)).toEqual([ + "kernel/Core.Domain.Kernel", + ]); + }); + + test("mixes coarsened areas and flat libs into a deduped, non-overlapping set", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "App.sln"), + [ + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Api", "platform\\admission\\src\\Core.Admission.Api\\Core.Admission.Api.csproj", "{11111111-1111-1111-1111-111111111111}"', + "EndProject", + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Tests", "platform\\admission\\tests\\Core.Admission.Api.Tests\\Core.Admission.Api.Tests.csproj", "{22222222-2222-2222-2222-222222222222}"', + "EndProject", + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "BillingApi", "platform\\billing\\src\\Core.Billing.Api\\Core.Billing.Api.csproj", "{33333333-3333-3333-3333-333333333333}"', + "EndProject", + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Kernel", "kernel\\Core.Domain.Kernel\\Core.Domain.Kernel.csproj", "{44444444-4444-4444-4444-444444444444}"', + "EndProject", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "platform/admission"), { recursive: true }); + await mkdir(path.join(repo, "platform/billing"), { recursive: true }); + await mkdir(path.join(repo, "kernel/Core.Domain.Kernel"), { + recursive: true, + }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path).sort()).toEqual([ + "kernel/Core.Domain.Kernel", + "platform/admission", + "platform/billing", + ]); + }); + + test("does not coarsen a top-level src/Proj to the repository root", async () => { + // A project directly under a root `src/` (no area segment above the `src` + // parent) must NOT collapse to "" (the repo root); it stays at src/Proj. + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "App.sln"), + [ + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Api", "src\\Api\\Api.csproj", "{11111111-1111-1111-1111-111111111111}"', + "EndProject", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "src/Api"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path)).toEqual(["src/Api"]); + }); + + test("does not collapse an area named `src`/`tests` to a bare tree root", async () => { + // BUG 1 guard: when the area segment above the src/tests parent is itself + // `src` or `tests` (idiomatic: a tests folder directly under a top-level + // src/), coarsening must NOT produce a bare `src`/`tests` workspace that + // would span the entire source or test tree as one wiki. + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "App.sln"), + [ + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Tests", "src\\tests\\App.Tests\\App.Tests.csproj", "{11111111-1111-1111-1111-111111111111}"', + "EndProject", + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Inner", "src\\src\\Foo\\Foo.csproj", "{22222222-2222-2222-2222-222222222222}"', + "EndProject", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "src/tests/App.Tests"), { recursive: true }); + await mkdir(path.join(repo, "src/src/Foo"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + const paths = detected.map((entry) => entry.path); + expect(paths).not.toContain("src"); + expect(paths).not.toContain("tests"); + expect(paths.sort()).toEqual(["src/src/Foo", "src/tests/App.Tests"]); + }); + + test("keeps a top-level src/App and its src/tests/App.Tests both alive", async () => { + // BUG 2 guard: without the area!==src/tests check, `src/tests/App.Tests` + // coarsens to `src`, becomes an ancestor of `src/App`, and dropAncestor + // deletes it, silently losing the test project. Both must survive. + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "App.sln"), + [ + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "App", "src\\App\\App.csproj", "{11111111-1111-1111-1111-111111111111}"', + "EndProject", + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Tests", "src\\tests\\App.Tests\\App.Tests.csproj", "{22222222-2222-2222-2222-222222222222}"', + "EndProject", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "src/App"), { recursive: true }); + await mkdir(path.join(repo, "src/tests/App.Tests"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path).sort()).toEqual([ + "src/App", + "src/tests/App.Tests", + ]); + }); + + test("documented edge: a directly-in-area project is dropped when a deeper non-src/tests sibling keeps the area as an ancestor", async () => { + // BUG 3 (accepted, documented): `area/Y` coarsens to nothing (its parent + // `area` is not src/tests, so it stays `area/Y`)... but here `area/Api` + // sits directly in the area via a src/ project that coarsens to `area`, + // while `area/group/Z` (parent `group` != src/tests) stays granular. The + // coarsened `area` becomes an ancestor of `area/group/Z` and is dropped. + // This locks in the known behavior; the lost area is recoverable via the + // written manifest. + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "App.sln"), + [ + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Api", "area\\src\\Area.Api\\Area.Api.csproj", "{11111111-1111-1111-1111-111111111111}"', + "EndProject", + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Deep", "area\\group\\Area.Deep\\Area.Deep.csproj", "{22222222-2222-2222-2222-222222222222}"', + "EndProject", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "area/src/Area.Api"), { recursive: true }); + await mkdir(path.join(repo, "area/group/Area.Deep"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + // `area` (from area/src/Area.Api) is an ancestor of `area/group/Area.Deep` + // and is dropped; only the deeper granular project survives. + expect(detected.map((entry) => entry.path)).toEqual([ + "area/group/Area.Deep", + ]); + }); + }); + + describe("Maven (pom.xml modules)", () => { + test("expands directories", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "pom.xml"), + [ + '', + " ", + " service-a", + " libs/service-b", + " ", + "", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "service-a"), { recursive: true }); + await mkdir(path.join(repo, "libs/service-b"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path).sort()).toEqual([ + "libs/service-b", + "service-a", + ]); + }); + + test("ignores modules inside XML comments", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "pom.xml"), + [ + "", + " ", + " service-a", + " ", + " ", + "", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "service-a"), { recursive: true }); + await mkdir(path.join(repo, "disabled"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + const paths = detected.map((entry) => entry.path); + expect(paths).toEqual(["service-a"]); + expect(paths).not.toContain("disabled"); + }); + + test("detects only the first (root) block", async () => { + // Documented limitation: profile-scoped are not merged; the + // narrow regex matches the first block only. Lock in that the + // root modules are detected (profile modules are simply not added). + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "pom.xml"), + [ + "", + " ", + " always", + " ", + " ", + " ", + " ", + " only-in-profile", + " ", + " ", + " ", + "", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "always"), { recursive: true }); + await mkdir(path.join(repo, "only-in-profile"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path)).toEqual(["always"]); + }); + + test("malformed pom.xml returns no Maven workspaces", async () => { + const repo = await createTempRepo(); + await writeFile(path.join(repo, "pom.xml"), "no modules", "utf8"); + expect(await detectWorkspaces(repo)).toEqual([]); + }); + }); + + describe("Gradle (settings.gradle / .kts includes)", () => { + test("maps :foo:bar project paths to foo/bar (Groovy)", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "settings.gradle"), + [ + "rootProject.name = 'demo'", + "include ':app'", + "include ':libs:core', ':libs:util'", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "app"), { recursive: true }); + await mkdir(path.join(repo, "libs/core"), { recursive: true }); + await mkdir(path.join(repo, "libs/util"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path).sort()).toEqual([ + "app", + "libs/core", + "libs/util", + ]); + }); + + test("handles Kotlin include(...) form and merges both files", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "settings.gradle.kts"), + ['include(":api", ":shared")', ""].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "api"), { recursive: true }); + await mkdir(path.join(repo, "shared"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path).sort()).toEqual([ + "api", + "shared", + ]); + }); + + test("captures a multi-line comma-continued include (Groovy)", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "settings.gradle"), + ["include ':a',", " ':b',", " ':c'", ""].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "a"), { recursive: true }); + await mkdir(path.join(repo, "b"), { recursive: true }); + await mkdir(path.join(repo, "c"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path).sort()).toEqual([ + "a", + "b", + "c", + ]); + }); + + test("captures a multi-line Kotlin include(...) list", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "settings.gradle.kts"), + ["include(", ' ":api",', ' ":shared"', ")", ""].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "api"), { recursive: true }); + await mkdir(path.join(repo, "shared"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path).sort()).toEqual([ + "api", + "shared", + ]); + }); + + test("does not emit includeBuild composite builds", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "settings.gradle"), + ["include ':app'", "includeBuild '../build-logic'", ""].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "app"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path)).toEqual(["app"]); + }); + + test("ignores includes in trailing and block comments", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "settings.gradle"), + [ + "include ':app' // include ':trailingFake'", + "/*", + "include ':blockFake'", + "*/", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "app"), { recursive: true }); + await mkdir(path.join(repo, "trailingFake"), { recursive: true }); + await mkdir(path.join(repo, "blockFake"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + const paths = detected.map((entry) => entry.path); + expect(paths).toEqual(["app"]); + expect(paths).not.toContain("trailingFake"); + expect(paths).not.toContain("blockFake"); + }); + + test("malformed settings.gradle returns no Gradle workspaces", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "settings.gradle"), + "rootProject.name = 'demo'", + "utf8", + ); + expect(await detectWorkspaces(repo)).toEqual([]); + }); + }); + + describe("Python uv workspace (pyproject.toml)", () => { + test("expands [tool.uv.workspace] members globs", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "pyproject.toml"), + [ + "[project]", + 'name = "root"', + "", + "[tool.uv.workspace]", + 'members = ["packages/*", "tools/cli"]', + 'exclude = ["packages/seeds"]', + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "packages/a"), { recursive: true }); + await mkdir(path.join(repo, "packages/b"), { recursive: true }); + await mkdir(path.join(repo, "tools/cli"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path).sort()).toEqual([ + "packages/a", + "packages/b", + "tools/cli", + ]); + }); + + test("pyproject.toml without a uv workspace section returns []", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "pyproject.toml"), + '[project]\nname = "solo"\n', + "utf8", + ); + expect(await detectWorkspaces(repo)).toEqual([]); + }); + }); + + describe("Bazel (coarse top-level roots)", () => { + test("emits immediate child dirs with a BUILD file when MODULE.bazel exists", async () => { + const repo = await createTempRepo(); + await writeFile(path.join(repo, "MODULE.bazel"), "", "utf8"); + await mkdir(path.join(repo, "app"), { recursive: true }); + await writeFile(path.join(repo, "app/BUILD.bazel"), "", "utf8"); + await mkdir(path.join(repo, "lib"), { recursive: true }); + await writeFile(path.join(repo, "lib/BUILD"), "", "utf8"); + // A top-level dir with no BUILD file must not be emitted. + await mkdir(path.join(repo, "docs"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + const paths = detected.map((entry) => entry.path); + expect(paths.sort()).toEqual(["app", "lib"]); + expect(paths).not.toContain("docs"); + }); + + test("does not explode into deeply nested Bazel packages", async () => { + const repo = await createTempRepo(); + await writeFile(path.join(repo, "WORKSPACE"), "", "utf8"); + await mkdir(path.join(repo, "app/feature/impl"), { recursive: true }); + await writeFile(path.join(repo, "app/BUILD.bazel"), "", "utf8"); + // Nested BUILD files exist but must NOT each become a workspace. + await writeFile(path.join(repo, "app/feature/BUILD.bazel"), "", "utf8"); + await writeFile( + path.join(repo, "app/feature/impl/BUILD.bazel"), + "", + "utf8", + ); + + const detected = await detectWorkspaces(repo); + expect(detected.map((entry) => entry.path)).toEqual(["app"]); + }); + + test("Bazel with no top-level BUILD dir returns []", async () => { + const repo = await createTempRepo(); + await writeFile(path.join(repo, "MODULE.bazel"), "", "utf8"); + await mkdir(path.join(repo, "deep/nested/pkg"), { recursive: true }); + await writeFile(path.join(repo, "deep/nested/pkg/BUILD"), "", "utf8"); + + expect(await detectWorkspaces(repo)).toEqual([]); + }); + }); +}); + +// Regression seam: detectWorkspaces output must feed resolveWorkspaceRuns +// without throwing. The overlap bug lived exactly here — every other overlap +// test hand-crafts the manifest, so this exercises the real auto-detect path. +describe("detectWorkspaces -> resolveWorkspaceRuns seam", () => { + test("packages/** detection resolves without overlap errors", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "package.json"), + JSON.stringify({ workspaces: ["packages/**"] }), + "utf8", + ); + await mkdir(path.join(repo, "packages/a"), { recursive: true }); + await writeFile(path.join(repo, "packages/a/package.json"), "{}", "utf8"); + await mkdir(path.join(repo, "packages/b/sub"), { recursive: true }); + await writeFile( + path.join(repo, "packages/b/sub/package.json"), + "{}", + "utf8", + ); + + const detected = await detectWorkspaces(repo); + // Must not throw (previously threw "packages is an ancestor of ..."). + const plan = resolveWorkspaceRuns(repo, { + version: 1, + workspaces: detected, + }); + const relativePaths = plan.runs.map((run) => run.relativePath).sort(); + expect(relativePaths).toEqual(["packages/a", "packages/b/sub"]); + }); + + test(".NET .sln detection resolves without throwing", async () => { + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "App.sln"), + [ + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Api", "src\\Api\\Api.csproj", "{11111111-1111-1111-1111-111111111111}"', + "EndProject", + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Web", "src\\Web\\Web.csproj", "{22222222-2222-2222-2222-222222222222}"', + "EndProject", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "src/Api"), { recursive: true }); + await mkdir(path.join(repo, "src/Web"), { recursive: true }); + + const detected = await detectWorkspaces(repo); + const plan = resolveWorkspaceRuns(repo, { version: 1, workspaces: detected }); + expect(plan.runs.map((run) => run.relativePath).sort()).toEqual([ + "src/Api", + "src/Web", + ]); + }); + + test("coarsened .NET area detection resolves to product-area runs", async () => { + // The DDD src/tests granularity would otherwise yield one run per .csproj; + // detection must coarsen to area roots that resolve without overlap. + const repo = await createTempRepo(); + await writeFile( + path.join(repo, "App.sln"), + [ + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Api", "platform\\admission\\src\\Core.Admission.Api\\Core.Admission.Api.csproj", "{11111111-1111-1111-1111-111111111111}"', + "EndProject", + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Domain", "platform\\admission\\src\\Core.Admission.Domain\\Core.Admission.Domain.csproj", "{22222222-2222-2222-2222-222222222222}"', + "EndProject", + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Tests", "platform\\admission\\tests\\Core.Admission.Domain.Tests\\Core.Admission.Domain.Tests.csproj", "{33333333-3333-3333-3333-333333333333}"', + "EndProject", + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Kernel", "kernel\\Core.Domain.Kernel\\Core.Domain.Kernel.csproj", "{44444444-4444-4444-4444-444444444444}"', + "EndProject", + "", + ].join("\n"), + "utf8", + ); + await mkdir(path.join(repo, "platform/admission"), { recursive: true }); + await mkdir(path.join(repo, "kernel/Core.Domain.Kernel"), { + recursive: true, + }); + + const detected = await detectWorkspaces(repo); + const plan = resolveWorkspaceRuns(repo, { version: 1, workspaces: detected }); + expect(plan.runs.map((run) => run.relativePath).sort()).toEqual([ + "kernel/Core.Domain.Kernel", + "platform/admission", + ]); + }); +}); From 3108e827a7a3db374be9b298ef9963e4410a2e1e Mon Sep 17 00:00:00 2001 From: Brad Huffman Date: Wed, 22 Jul 2026 18:47:12 -0500 Subject: [PATCH 2/5] feat: self-maintaining workspace discovery for recursive monorepo docs Makes `--recursive` keep the workspace manifest current as the repo evolves, so a scheduled CI run picks up newly added projects and drops removed ones without manual manifest edits. - Discovery re-runs on every recursive run (deterministic, ~20ms even on a 400-project repo) and MERGES with the existing openwiki/workspaces.json rather than only auto-detecting when no manifest exists. - New manifest schema separates concerns: a managed `workspaces` list (detection-owned, path-only, sorted, regenerated each run) and a hand-authored `overrides` map keyed by path (goal / name / exclude / include), so customization and auto-discovery never collide. - Idempotent write: the manifest is serialized to canonical bytes (fixed key order, sorted entries) and rewritten only when it actually changed, so an unchanged repo produces no manifest diff and no spurious CI commit. - Manual groupings are preserved: a referenced path detection cannot surface whose directory still exists is kept and promoted to `include: true` (carrying any goal/name); one whose directory is gone is pruned, with a warning when it carried a hand-authored override. - An `include` override that would overlap a detected workspace is dropped with a warning instead of being persisted, preserving the "never write an unresolvable manifest" invariant. - Legacy flat manifests (workspaces:[{path,goal,name}]) are read and migrated in-memory, then re-emitted in the new shape without losing goals. Documents the manual-preservation and overlap-guard behavior in README.md. Co-Authored-By: Claude --- README.md | 49 +++- src/cli.tsx | 11 +- src/monorepo/orchestrator.ts | 5 + src/monorepo/workspaces.ts | 440 +++++++++++++++++++++++++--- test/recursion-activation.test.ts | 471 +++++++++++++++++++++++++++++- test/workspaces.test.ts | 86 +++++- 6 files changed, 1009 insertions(+), 53 deletions(-) diff --git a/README.md b/README.md index 810b5813..ef5295bd 100644 --- a/README.md +++ b/README.md @@ -195,14 +195,49 @@ repository root: ```json { "version": 1, - "workspaces": [ - { "path": "packages/api", "name": "API", "goal": "Optional per-subproject brief." }, - { "path": "packages/web" } - ], + "workspaces": [{ "path": "packages/api" }, { "path": "packages/web" }], + "overrides": { + "packages/api": { "name": "API", "goal": "Optional per-subproject brief." }, + "vendor/thirdparty": { "exclude": true }, + "tools/custom-grouping": { "include": true, "name": "Tools" } + }, "root": { "goal": "Optional brief for the aggregating root wiki." } } ``` +The manifest has two parts. The `workspaces` list is **managed** — it is +regenerated from detection on every recursive run, sorted and path-only. All +hand-authored customization lives in the `overrides` map, keyed by +repo-relative path, and is never auto-clobbered: + +- `goal` / `name`: a per-subproject wiki brief and display name. +- `exclude: true`: detection still lists the path (so it is not re-discovered as + new noise), but no sub-wiki is generated for it. +- `include: true`: force-add a path detection cannot find (e.g. a custom + grouping). This flag is also what distinguishes a deliberate manual addition + from an override left behind by a deleted project. + +The legacy flat shape (`goal`/`name` written directly on `workspaces` entries) +is still read: those fields are migrated into `overrides` in memory, and the +next write emits the current shape without losing them. + +- **Self-maintaining discovery.** Detection re-runs on **every** recursive run + (it is deterministic and fast) and is merged with the existing manifest, so + the manifest stays current as the repo evolves: newly added projects appear + automatically, deleted ones drop out, and your `overrides` survive + regeneration. The merged manifest is written back **only when it changed** — a + no-op run leaves `openwiki/workspaces.json` byte-for-byte unchanged, so + scheduled CI runs produce no spurious manifest diff. +- **Manual entries are preserved, stale ones are pruned.** On the first + self-maintaining run, any path you hand-listed that detection cannot surface + is kept **if its directory still exists** — it is promoted to an explicit + `"include": true` override (carrying any goal/name) so it is stable and + self-documenting. A referenced path whose directory is gone is treated as a + removed project and dropped; if it carried a hand-authored override, a warning + names the pruned path (a plain managed entry is removed quietly). An + `"include"` path that would overlap a detected workspace (for example an + ancestor of a detected leaf) is ignored with a warning rather than persisted, + so a stray include can never wedge future runs. - **Activation.** If `openwiki/workspaces.json` is present, recursive mode is enabled automatically. Passing `--recursive` with no manifest triggers auto-detection of common workspace layouts (pnpm/npm/yarn workspaces, Cargo, @@ -213,14 +248,14 @@ repository root: wiki gets a generated `openwiki/workspaces.md` index linking to every sub-wiki; do not hand-edit it. - **Incremental updates.** Each subproject is evaluated independently: on - `--update`, a subproject's sub-wiki regenerates only when files *within that - subproject's own subtree* have changed since it was last documented (tracked in + `--update`, a subproject's sub-wiki regenerates only when files _within that + subproject's own subtree_ have changed since it was last documented (tracked in its own `openwiki/.last-update.json`). Unchanged subprojects are skipped without a model call, so scheduled refreshes stay cheap. The root wiki regenerates on every run. - **Known limitation — no dependency cascade.** Updates do not propagate across subprojects. If a shared subproject (for example a common kernel or contracts - package) changes, only *that* sub-wiki and the root wiki are refreshed — the + package) changes, only _that_ sub-wiki and the root wiki are refreshed — the sibling subprojects that depend on it are **not** automatically regenerated, even though their documented context may reference the changed code. Until dependency-aware invalidation exists, force a full refresh after a significant diff --git a/src/cli.tsx b/src/cli.tsx index d2e91758..81864ee6 100644 --- a/src/cli.tsx +++ b/src/cli.tsx @@ -618,7 +618,9 @@ function App({ command }: AppProps) { const recursiveRunCommand = resolvedCommand; const activationPromise: Promise = runMode === "code" && recursiveRunCommand !== "chat" - ? resolveRecursionActivation(runtimeCwd, command.recursive) + ? resolveRecursionActivation(runtimeCwd, command.recursive, (message) => + runOptions.onEvent({ type: "text", text: `\n${message}\n` }), + ) : Promise.resolve({ kind: "plain", reason: "not a code init/update run", @@ -4135,7 +4137,12 @@ async function runPrintCommand( // to code (repository) mode and to init/update runs. const activation = command.mode === "code" && command.command !== "chat" - ? await resolveRecursionActivation(runtimeCwd, command.recursive) + ? await resolveRecursionActivation( + runtimeCwd, + command.recursive, + (message) => + runOptions.onEvent({ type: "text", text: `\n${message}\n` }), + ) : ({ kind: "plain", reason: "not a code init/update run" } as const); if (activation.kind === "recurse") { diff --git a/src/monorepo/orchestrator.ts b/src/monorepo/orchestrator.ts index 9303b4c9..9cc5b939 100644 --- a/src/monorepo/orchestrator.ts +++ b/src/monorepo/orchestrator.ts @@ -66,6 +66,11 @@ export interface RecursiveRunResult { * subprojects that depend on it. Dependency-aware invalidation is intentionally * out of scope here; see the "no dependency cascade" note in README.md. * + * The run set comes from resolveWorkspaceRuns, which applies the manifest's + * `overrides` map: a path marked `exclude: true` produces NO run here (it is + * documented nowhere yet stays listed so self-maintaining discovery does not + * re-surface it), and a run's goal/name come from its override when present. + * * Falls back to a single plain root run (no recursion role) when the manifest * resolves to zero workspaces. */ diff --git a/src/monorepo/workspaces.ts b/src/monorepo/workspaces.ts index a942681b..56e6d030 100644 --- a/src/monorepo/workspaces.ts +++ b/src/monorepo/workspaces.ts @@ -10,9 +10,16 @@ import path from "node:path"; import { parse as parseYaml } from "yaml"; /** - * One workspace entry as written in `openwiki/workspaces.json`. `path` is a - * repository-root-relative POSIX path (no globs at runtime; auto-detect expands - * globs before writing the manifest). `goal` and `name` are optional overrides. + * One entry in the managed `workspaces` list of `openwiki/workspaces.json`. + * `path` is a repository-root-relative POSIX path (no globs at runtime; + * auto-detect expands globs before writing the manifest). + * + * The `workspaces` list is DETECTION-OWNED: self-maintaining discovery + * regenerates it (path-only, sorted) on every recursive run. Hand-authored + * customization lives in the sibling `overrides` map, never on these entries. + * `goal`/`name` remain optional here only for backward-compatible reading of the + * legacy flat schema (see {@link normalizeManifest}); on read they are migrated + * into `overrides`, and the next write emits path-only entries. */ export interface WorkspaceEntry { path: string; @@ -21,12 +28,41 @@ export interface WorkspaceEntry { } /** - * The parsed `openwiki/workspaces.json` manifest. `version` is 1 for the current - * explicit-paths schema. + * A hand-authored customization for one workspace path, keyed by that path in + * the manifest's `overrides` map. This is the single place for all manual edits, + * so self-maintaining discovery can regenerate the `workspaces` list wholesale + * without clobbering user intent. + * + * - `goal` / `name`: per-subproject wiki brief and display name overrides. + * - `exclude`: when true, the path is NOT documented (no sub-wiki run) even if + * detection surfaces it. It remains listed in `workspaces` so it is not + * re-discovered as new noise; the run set filters it out. + * - `include`: when true, the path is force-added to the run set even if + * detection does not surface it (a custom grouping detection cannot find). + * This is also the signal that distinguishes a deliberate manual addition from + * an orphaned override left behind by a deleted project. + */ +export interface WorkspaceOverride { + goal?: string; + name?: string; + exclude?: boolean; + include?: boolean; +} + +/** + * The parsed `openwiki/workspaces.json` manifest. `version` is 1. + * + * The current schema separates the DETECTION-OWNED `workspaces` list (auto + * regenerated each recursive run, sorted, path-only) from the HAND-AUTHORED + * `overrides` map (keyed by repo-relative path, never auto-clobbered). The + * legacy flat shape (`workspaces:[{path, goal?, name?}]`, no `overrides`) is + * still read: {@link normalizeManifest} migrates any per-entry goal/name into + * `overrides` in memory, and the next write emits the current shape. */ export interface WorkspaceManifest { version: number; workspaces: WorkspaceEntry[]; + overrides?: Record; root?: { goal?: string }; } @@ -65,17 +101,36 @@ export type RecursionActivation = | { kind: "plain"; reason: string }; /** - * Decides whether to recurse, honoring the product rules: + * Decides whether to recurse, honoring the product rules and keeping the + * workspace manifest self-maintaining: * - `recursive === false`: force off, even if a manifest exists. - * - a manifest exists: recurse (auto-enabled). - * - `recursive === true` and no manifest: auto-detect workspaces; if any are - * found, WRITE openwiki/workspaces.json (for the user to review) and recurse; - * otherwise fall back to a plain run. - * - otherwise (default, no manifest): plain run. + * - otherwise recurse when `--recursive` is set OR a manifest already exists; + * `recursive` unset with no manifest is a plain run. + * + * On every recursive run, detection is re-run (it is deterministic and fast) and + * MERGED with the existing manifest so the manifest stays current as the repo + * evolves — new projects auto-appear, deleted ones drop out, and hand-authored + * customization survives: + * - The managed `workspaces` list is regenerated as the union of the freshly + * detected (pruned-to-resolvable) paths and any override paths marked + * `include: true` (manual additions detection cannot find), sorted, path-only. + * An excluded path detection still surfaces stays listed so it is not + * re-discovered as new noise; it is filtered out of the RUN set downstream. + * - The `overrides` map is preserved from the existing manifest, EXCEPT orphans: + * an override whose path is neither detected nor marked `include` (i.e. the + * project it customized was deleted or moved) is dropped and a warning is + * emitted via `onWarn`. + * - The result is written back ONLY if it differs from disk (idempotent), so a + * no-op run leaves the file byte-for-byte unchanged and CI produces no + * spurious manifest diff. + * + * `onWarn` receives a human-readable message for each pruned orphan override; + * callers wire it to the run's event stream. It defaults to a no-op. */ export async function resolveRecursionActivation( repoRoot: string, recursive: boolean | undefined, + onWarn: (message: string) => void = () => {}, ): Promise { if (recursive === false) { return { @@ -84,17 +139,39 @@ export async function resolveRecursionActivation( }; } - const manifest = await readWorkspaceManifest(repoRoot); - if (manifest) { - return { kind: "recurse", manifest, autoDetected: false }; - } + const existing = await readWorkspaceManifest(repoRoot); + const manifestPresent = existing !== null; - if (recursive !== true) { + if (!manifestPresent && recursive !== true) { return { kind: "plain", reason: "no openwiki/workspaces.json manifest" }; } - const detected = await detectWorkspaces(repoRoot); - if (detected.length === 0) { + // Detection is deterministic and cheap, so re-run it every recursive run and + // merge with the existing manifest. Prune to a resolvable (leaf-only) set: + // never persist a manifest that cannot be resolved, or a poisoned + // openwiki/workspaces.json would make every future default run auto-recurse + // into a throw. + const detected = pruneToResolvableWorkspaces( + repoRoot, + await detectWorkspaces(repoRoot), + ); + + // Pre-resolve directory existence for every non-detected path the existing + // manifest referenced, so mergeManifest can stay pure/synchronous. A manual + // grouping detection cannot find is preserved only when its directory really + // exists; a removed project's directory is gone, so it is dropped. + const existingDirs = await resolveExistingManualDirs( + repoRoot, + existing, + new Set(detected.map((entry) => entry.path)), + ); + const merged = mergeManifest(detected, existing, onWarn, (relativePath) => + existingDirs.has(relativePath), + ); + + if (!manifestPresent && merged.workspaces.length === 0) { + // `--recursive` was requested with no manifest and nothing to document: + // fall back to a plain run WITHOUT writing an empty manifest. return { kind: "plain", reason: @@ -102,17 +179,173 @@ export async function resolveRecursionActivation( }; } - // Never persist a manifest that cannot be resolved: a poisoned - // openwiki/workspaces.json would make every future default run auto-recurse - // into a throw, wedging the repo until the user hand-edits it. Prune to a - // resolvable (leaf-only) set before writing. - const writtenManifest: WorkspaceManifest = { + await writeWorkspaceManifestIfChanged(repoRoot, merged); + + return { kind: "recurse", manifest: merged, autoDetected: !manifestPresent }; +} + +/** + * Merges freshly detected paths with an existing manifest into the canonical, + * deterministic manifest to persist. See {@link resolveRecursionActivation} for + * the merge rules; this is the pure core (no I/O) so it is easy to reason about + * and test. + * + * `pathExists(relativePath)` reports whether a non-detected path still exists as + * a real directory on disk. It is the discriminator that keeps a manual manifest + * safe: a workspace path detection cannot surface but whose directory still + * exists is a deliberate manual grouping (preserved, promoted to an explicit + * `include`); one whose directory is gone is a removed project (dropped — and + * warned about only when it carried a hand-authored override, so removals of + * plain managed entries stay quiet as the feature intends). Injected (not read + * here) so this core stays pure; the caller backs it with a real stat. It + * defaults to "nothing extra exists", which matches pure detected-only merges. + */ +export function mergeManifest( + detected: WorkspaceEntry[], + existing: WorkspaceManifest | null, + onWarn: (message: string) => void = () => {}, + pathExists: (relativePath: string) => boolean = () => false, +): WorkspaceManifest { + const byPath = (left: string, right: string): number => + left.localeCompare(right); + + const detectedPaths = [...new Set(detected.map((entry) => entry.path))].sort( + byPath, + ); + const detectedSet = new Set(detectedPaths); + const overrides = existing?.overrides ?? {}; + + // Candidate manual paths: any path the existing manifest referenced (managed + // workspace entry OR override key) that detection no longer surfaces. A bare + // path-only legacy entry lives ONLY in `workspaces`, so it must be considered + // here too — otherwise it would be silently dropped without even a warning. + const candidatePaths = new Set(); + for (const entry of existing?.workspaces ?? []) { + if (!detectedSet.has(entry.path)) { + candidatePaths.add(entry.path); + } + } + for (const key of Object.keys(overrides)) { + if (!detectedSet.has(key)) { + candidatePaths.add(key); + } + } + + // Decide which candidates survive as manual includes. Keep a path that is an + // explicit include OR whose directory still exists (a manual grouping). Drop + // the rest: warn when the dropped path had a hand-authored override (intent is + // being discarded), stay quiet for a plain managed entry that was removed. + const survivingIncludes = new Set(); + for (const candidate of [...candidatePaths].sort(byPath)) { + const override = overrides[candidate]; + if (override?.include === true || pathExists(candidate)) { + survivingIncludes.add(candidate); + } else if (override) { + onWarn( + `Dropping orphaned workspace override for "${candidate}": the path is no longer detected or present. Mark it "include": true to keep it.`, + ); + } + } + + // Build the managed list detected-first (detected leaves are guaranteed + // non-overlapping and always win), then fold in surviving includes only when + // they do not overlap an already-accepted path. This restores the "never + // persist a manifest that cannot be resolved" invariant for the include union, + // which pruneToResolvableWorkspaces (detected-only, upstream) cannot cover. + const effectivePaths: string[] = [...detectedPaths]; + for (const candidate of [...survivingIncludes].sort(byPath)) { + const overlaps = effectivePaths.some( + (accepted) => + accepted === candidate || + isAncestorPath(candidate, accepted) || + isAncestorPath(accepted, candidate), + ); + if (overlaps) { + survivingIncludes.delete(candidate); + onWarn( + `Ignoring workspace include "${candidate}": it overlaps a detected workspace and would make the manifest unresolvable.`, + ); + continue; + } + effectivePaths.push(candidate); + } + + const workspaces: WorkspaceEntry[] = [...new Set(effectivePaths)] + .sort(byPath) + .map((relativePath) => ({ path: relativePath })); + + // Kept overrides (sorted for a stable, idempotent write): detected paths keep + // their override verbatim; surviving includes are persisted with include:true + // (promoting a bare manual entry, or a goal-only legacy entry, into an + // explicit include so it is stable and future removal routes through the + // orphan warning). Dropped paths carry no override forward. + const keptOverrides: Record = {}; + for (const key of [ + ...new Set([...Object.keys(overrides), ...survivingIncludes]), + ].sort(byPath)) { + if (detectedSet.has(key)) { + keptOverrides[key] = overrides[key]; + } else if (survivingIncludes.has(key)) { + keptOverrides[key] = { ...(overrides[key] ?? {}), include: true }; + } + } + + return { version: 1, - workspaces: pruneToResolvableWorkspaces(repoRoot, detected), + workspaces, + ...(Object.keys(keptOverrides).length > 0 + ? { overrides: keptOverrides } + : {}), + ...(existing?.root ? { root: existing.root } : {}), }; - await writeWorkspaceManifest(repoRoot, writtenManifest); +} + +/** + * Stats every non-detected path referenced by the existing manifest (managed + * entries and override keys) and returns the subset whose repo-relative + * directory actually exists. mergeManifest uses this to tell a real manual + * grouping (keep) from a removed project (drop). Best-effort and safe: paths + * that fail validation (absolute/`..`) or are absent simply do not appear. + */ +async function resolveExistingManualDirs( + repoRoot: string, + existing: WorkspaceManifest | null, + detectedPaths: Set, +): Promise> { + const candidates = new Set(); + for (const entry of existing?.workspaces ?? []) { + if (!detectedPaths.has(entry.path)) { + candidates.add(entry.path); + } + } + for (const key of Object.keys(existing?.overrides ?? {})) { + if (!detectedPaths.has(key)) { + candidates.add(key); + } + } - return { kind: "recurse", manifest: writtenManifest, autoDetected: true }; + const present = new Set(); + await Promise.all( + [...candidates].map(async (relativePath) => { + let normalized: string; + try { + normalized = normalizeWorkspacePath(relativePath); + } catch { + return; + } + if (normalized === "" || normalized === ".") { + return; + } + const dirStat = await stat(path.join(repoRoot, normalized)).catch( + () => null, + ); + if (dirStat?.isDirectory()) { + present.add(relativePath); + } + }), + ); + + return present; } /** @@ -136,9 +369,50 @@ function pruneToResolvableWorkspaces( } /** - * Writes an auto-detected manifest to openwiki/workspaces.json so the user can - * review and edit the detected workspace set. Written with a comment-free JSON - * body and a trailing newline. + * Serializes a manifest to its canonical, stable on-disk form: fixed top-level + * key order (version, workspaces, overrides, root), sorted `workspaces`, sorted + * `overrides` keys with fixed field order, undefined/empty fields omitted, and a + * trailing newline. Re-serializing an already-canonical manifest is a no-op, + * which is what makes the idempotent write check byte-exact. + */ +export function serializeManifest(manifest: WorkspaceManifest): string { + const canonical: Record = { version: 1 }; + + canonical.workspaces = [...manifest.workspaces] + .map((entry) => ({ path: entry.path })) + .sort((left, right) => left.path.localeCompare(right.path)); + + if (manifest.overrides && Object.keys(manifest.overrides).length > 0) { + const overrides: Record = {}; + for (const key of Object.keys(manifest.overrides).sort((left, right) => + left.localeCompare(right), + )) { + const override = manifest.overrides[key]; + const canonicalOverride: WorkspaceOverride = {}; + if (override.goal !== undefined) canonicalOverride.goal = override.goal; + if (override.name !== undefined) canonicalOverride.name = override.name; + if (override.exclude !== undefined) { + canonicalOverride.exclude = override.exclude; + } + if (override.include !== undefined) { + canonicalOverride.include = override.include; + } + overrides[key] = canonicalOverride; + } + canonical.overrides = overrides; + } + + if (manifest.root?.goal !== undefined) { + canonical.root = { goal: manifest.root.goal }; + } + + return `${JSON.stringify(canonical, null, 2)}\n`; +} + +/** + * Writes a manifest to openwiki/workspaces.json in canonical form so the user + * can review and edit it. Written with a comment-free JSON body and a trailing + * newline. */ export async function writeWorkspaceManifest( repoRoot: string, @@ -146,11 +420,30 @@ export async function writeWorkspaceManifest( ): Promise { const manifestPath = path.join(repoRoot, WORKSPACES_MANIFEST_RELATIVE_PATH); await mkdir(path.dirname(manifestPath), { recursive: true }); - await writeFile( - manifestPath, - `${JSON.stringify(manifest, null, 2)}\n`, - "utf8", - ); + await writeFile(manifestPath, serializeManifest(manifest), "utf8"); +} + +/** + * Idempotent write: serializes the target manifest and writes it ONLY when the + * bytes differ from what is already on disk. A no-op recursive run therefore + * leaves openwiki/workspaces.json untouched (same mtime, same bytes), so CI does + * not produce a spurious manifest diff. Returns true when a write happened. + */ +export async function writeWorkspaceManifestIfChanged( + repoRoot: string, + manifest: WorkspaceManifest, +): Promise { + const manifestPath = path.join(repoRoot, WORKSPACES_MANIFEST_RELATIVE_PATH); + const target = serializeManifest(manifest); + + const current = await readFileIfPresent(manifestPath); + if (current === target) { + return false; + } + + await mkdir(path.dirname(manifestPath), { recursive: true }); + await writeFile(manifestPath, target, "utf8"); + return true; } /** @@ -227,6 +520,29 @@ export function normalizeManifest(value: unknown): WorkspaceManifest { }; }); + const overrides = normalizeOverrides(value.overrides); + + // Backward compat: migrate any per-entry goal/name from the legacy flat schema + // into the overrides map (keyed by path). An explicit override wins; the entry + // only fills fields the override does not already set. This preserves goals a + // user hand-wrote in the old shape, since the next write emits path-only + // workspace entries and carries all customization in `overrides`. + for (const entry of workspaces) { + if (entry.goal === undefined && entry.name === undefined) { + continue; + } + const existing = overrides[entry.path] ?? {}; + overrides[entry.path] = { + ...existing, + ...(existing.goal === undefined && entry.goal !== undefined + ? { goal: entry.goal } + : {}), + ...(existing.name === undefined && entry.name !== undefined + ? { name: entry.name } + : {}), + }; + } + const rootGoal = isRecord(value.root) && typeof value.root.goal === "string" ? value.root.goal @@ -235,15 +551,49 @@ export function normalizeManifest(value: unknown): WorkspaceManifest { return { version: 1, workspaces, + ...(Object.keys(overrides).length > 0 ? { overrides } : {}), ...(rootGoal !== undefined ? { root: { goal: rootGoal } } : {}), }; } +/** + * Normalizes the untrusted `overrides` map, keeping only recognized fields with + * the right types. A missing or non-object `overrides` yields an empty map; a + * non-object override value is skipped rather than throwing (the overrides map + * is hand-authored and best-effort — a stray value must not wedge the run). + */ +function normalizeOverrides(value: unknown): Record { + const overrides: Record = {}; + if (!isRecord(value)) { + return overrides; + } + + for (const [key, raw] of Object.entries(value)) { + if (!isRecord(raw)) { + continue; + } + const override: WorkspaceOverride = {}; + if (typeof raw.goal === "string") override.goal = raw.goal; + if (typeof raw.name === "string") override.name = raw.name; + if (typeof raw.exclude === "boolean") override.exclude = raw.exclude; + if (typeof raw.include === "boolean") override.include = raw.include; + overrides[key] = override; + } + + return overrides; +} + /** * Validates, normalizes, dedupes, and rejects overlapping workspace entries, * returning an ordered recursion plan (leaves first is the caller's concern; * this function preserves manifest order after dedupe). * + * Overrides are applied per path (keyed by the normalized repo-relative path): + * - `exclude: true` produces NO run for that path (it stays listed in + * `workspaces` so it is not re-discovered, but is never documented). + * - `goal`/`name` take precedence over any legacy per-entry goal/name; the + * precedence chain is override → legacy entry → none. + * * Rejections (throw): * - absolute paths or paths containing `..` segments (escape attempts) * - a workspace equal to the repository root @@ -253,6 +603,7 @@ export function resolveWorkspaceRuns( repoRoot: string, manifest: WorkspaceManifest, ): ResolvedWorkspacePlan { + const overrides = manifest.overrides ?? {}; const seen = new Set(); const runs: ResolvedWorkspaceRun[] = []; @@ -272,11 +623,22 @@ export function resolveWorkspaceRuns( } seen.add(relativePath); + // An excluded path is validated and de-duped like any other (so overlap + // checks stay honest) but produces no run — it is listed only so detection + // does not re-surface it as new noise. + const override = overrides[relativePath]; + if (override?.exclude === true) { + continue; + } + + const goal = override?.goal ?? entry.goal; + const name = override?.name ?? entry.name; + runs.push({ relativePath, absolutePath: path.resolve(repoRoot, relativePath), - goal: entry.goal?.trim() ? entry.goal.trim() : undefined, - name: entry.name?.trim() ? entry.name.trim() : undefined, + goal: goal?.trim() ? goal.trim() : undefined, + name: name?.trim() ? name.trim() : undefined, }); } @@ -821,9 +1183,9 @@ async function readMavenModuleGlobs(repoRoot: string): Promise { } const withoutComments = modulesBlock[1].replace(//gu, ""); - return [...withoutComments.matchAll(/\s*([^<]+?)\s*<\/module>/gu)].map( - (match) => match[1].replace(/\\/gu, "/"), - ); + return [ + ...withoutComments.matchAll(/\s*([^<]+?)\s*<\/module>/gu), + ].map((match) => match[1].replace(/\\/gu, "/")); } /** diff --git a/test/recursion-activation.test.ts b/test/recursion-activation.test.ts index 9aac0f0b..58cd14b8 100644 --- a/test/recursion-activation.test.ts +++ b/test/recursion-activation.test.ts @@ -1,4 +1,11 @@ -import { mkdtemp, mkdir, rm, writeFile, readFile } from "node:fs/promises"; +import { + mkdtemp, + mkdir, + rm, + stat, + writeFile, + readFile, +} from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; import { afterEach, describe, expect, test } from "vitest"; @@ -7,6 +14,30 @@ import { resolveWorkspaceRuns, } from "../src/monorepo/workspaces.ts"; +/** Writes a package.json workspaces glob and a set of leaf packages. */ +async function writePackagesRepo( + repo: string, + packages: string[], +): Promise { + await writeFile( + path.join(repo, "package.json"), + JSON.stringify({ workspaces: ["packages/*"] }), + "utf8", + ); + for (const pkg of packages) { + await mkdir(path.join(repo, "packages", pkg), { recursive: true }); + await writeFile( + path.join(repo, "packages", pkg, "package.json"), + "{}", + "utf8", + ); + } +} + +function readManifest(repo: string): Promise { + return readFile(path.join(repo, "openwiki", "workspaces.json"), "utf8"); +} + const tempDirs: string[] = []; async function createRepo(): Promise { @@ -24,6 +55,17 @@ afterEach(async () => { describe("resolveRecursionActivation", () => { test("recurses when a manifest exists (default flag)", async () => { const repo = await createRepo(); + // The `workspaces` list is now detection-owned and regenerated every run, so + // the manifest path must be a real, detectable workspace to survive the + // merge (a bare path with no detection source and no override is treated as + // stale and pruned — see the include/orphan tests below). + await writeFile( + path.join(repo, "package.json"), + JSON.stringify({ workspaces: ["packages/*"] }), + "utf8", + ); + await mkdir(path.join(repo, "packages", "a"), { recursive: true }); + await writeFile(path.join(repo, "packages", "a", "package.json"), "{}"); await mkdir(path.join(repo, "openwiki"), { recursive: true }); await writeFile( path.join(repo, "openwiki", "workspaces.json"), @@ -122,3 +164,430 @@ describe("resolveRecursionActivation", () => { } }); }); + +describe("resolveRecursionActivation self-maintaining discovery", () => { + test("merge: a newly-added project dir appears; existing overrides preserved", async () => { + const repo = await createRepo(); + await writePackagesRepo(repo, ["a"]); + // Seed a manifest with an override for the existing project. + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + await writeFile( + path.join(repo, "openwiki", "workspaces.json"), + JSON.stringify({ + version: 1, + workspaces: [{ path: "packages/a" }], + overrides: { "packages/a": { goal: "the A", name: "Alpha" } }, + }), + "utf8", + ); + + // A later commit adds packages/b on disk. + await mkdir(path.join(repo, "packages", "b"), { recursive: true }); + await writeFile(path.join(repo, "packages", "b", "package.json"), "{}"); + + const activation = await resolveRecursionActivation(repo, undefined); + expect(activation.kind).toBe("recurse"); + if (activation.kind !== "recurse") return; + + expect(activation.manifest.workspaces.map((w) => w.path)).toEqual([ + "packages/a", + "packages/b", + ]); + expect(activation.manifest.overrides?.["packages/a"]).toEqual({ + goal: "the A", + name: "Alpha", + }); + }); + + test("deleted project: path removed from workspaces and its override pruned with a warning", async () => { + const repo = await createRepo(); + await writePackagesRepo(repo, ["a"]); + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + await writeFile( + path.join(repo, "openwiki", "workspaces.json"), + JSON.stringify({ + version: 1, + workspaces: [{ path: "packages/a" }, { path: "packages/gone" }], + overrides: { "packages/gone": { goal: "obsolete" } }, + }), + "utf8", + ); + + const warnings: string[] = []; + const activation = await resolveRecursionActivation( + repo, + undefined, + (message) => warnings.push(message), + ); + expect(activation.kind).toBe("recurse"); + if (activation.kind !== "recurse") return; + + expect(activation.manifest.workspaces.map((w) => w.path)).toEqual([ + "packages/a", + ]); + expect(activation.manifest.overrides?.["packages/gone"]).toBeUndefined(); + expect(warnings).toHaveLength(1); + expect(warnings[0]).toContain("packages/gone"); + }); + + test("idempotent: a second no-op activation does not rewrite the file", async () => { + const repo = await createRepo(); + await writePackagesRepo(repo, ["a", "b"]); + + await resolveRecursionActivation(repo, true); + const firstBytes = await readManifest(repo); + const firstMtime = ( + await stat(path.join(repo, "openwiki", "workspaces.json")) + ).mtimeMs; + + const second = await resolveRecursionActivation(repo, undefined); + expect(second.kind).toBe("recurse"); + const secondBytes = await readManifest(repo); + const secondMtime = ( + await stat(path.join(repo, "openwiki", "workspaces.json")) + ).mtimeMs; + + expect(secondBytes).toBe(firstBytes); + expect(secondMtime).toBe(firstMtime); + }); + + test("override preservation: a hand-authored goal/name survives re-detection", async () => { + const repo = await createRepo(); + await writePackagesRepo(repo, ["a"]); + await resolveRecursionActivation(repo, true); + + // User hand-edits the manifest to add an override. + const manifestPath = path.join(repo, "openwiki", "workspaces.json"); + const manifest = JSON.parse(await readFile(manifestPath, "utf8")) as { + version: number; + workspaces: { path: string }[]; + overrides?: Record; + }; + manifest.overrides = { "packages/a": { goal: "hand written", name: "A!" } }; + await writeFile(manifestPath, JSON.stringify(manifest, null, 2), "utf8"); + + const activation = await resolveRecursionActivation(repo, undefined); + expect(activation.kind).toBe("recurse"); + if (activation.kind !== "recurse") return; + expect(activation.manifest.overrides?.["packages/a"]).toEqual({ + goal: "hand written", + name: "A!", + }); + }); + + test("include/union: an override path detection does not find is still present and produces a run", async () => { + const repo = await createRepo(); + await writePackagesRepo(repo, ["a"]); + // A custom grouping detection cannot surface, forced in via include. Give it + // a goal so getWorkspaceSkipReason would not skip it either. + await mkdir(path.join(repo, "tools", "grouping"), { recursive: true }); + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + await writeFile( + path.join(repo, "openwiki", "workspaces.json"), + JSON.stringify({ + version: 1, + workspaces: [{ path: "packages/a" }], + overrides: { + "tools/grouping": { include: true, goal: "custom grouping" }, + }, + }), + "utf8", + ); + + const activation = await resolveRecursionActivation(repo, undefined); + expect(activation.kind).toBe("recurse"); + if (activation.kind !== "recurse") return; + + expect(activation.manifest.workspaces.map((w) => w.path)).toEqual([ + "packages/a", + "tools/grouping", + ]); + const plan = resolveWorkspaceRuns(repo, activation.manifest); + const grouping = plan.runs.find((r) => r.relativePath === "tools/grouping"); + expect(grouping).toBeDefined(); + expect(grouping?.goal).toBe("custom grouping"); + }); + + test("legacy flat schema: per-entry goal/name is migrated into overrides and re-emitted", async () => { + const repo = await createRepo(); + await writePackagesRepo(repo, ["a"]); + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + // Old flat shape: goal/name directly on the workspace entry, no overrides. + await writeFile( + path.join(repo, "openwiki", "workspaces.json"), + JSON.stringify({ + version: 1, + workspaces: [{ path: "packages/a", goal: "legacy goal", name: "LegA" }], + }), + "utf8", + ); + + const activation = await resolveRecursionActivation(repo, undefined); + expect(activation.kind).toBe("recurse"); + if (activation.kind !== "recurse") return; + + // The goal/name migrated into overrides; workspaces entries are path-only. + expect(activation.manifest.workspaces).toEqual([{ path: "packages/a" }]); + expect(activation.manifest.overrides?.["packages/a"]).toEqual({ + goal: "legacy goal", + name: "LegA", + }); + + // The written file is the new shape and still carries the goal. + const written = JSON.parse(await readManifest(repo)) as { + workspaces: { path: string; goal?: string }[]; + overrides?: Record; + }; + expect(written.workspaces).toEqual([{ path: "packages/a" }]); + expect(written.overrides?.["packages/a"]?.goal).toBe("legacy goal"); + }); + + test("determinism: two runs produce byte-identical, sorted output", async () => { + const repo = await createRepo(); + await writePackagesRepo(repo, ["b", "a", "c"]); + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + // Overrides deliberately out of key order; the write must sort them. + await writeFile( + path.join(repo, "openwiki", "workspaces.json"), + JSON.stringify({ + version: 1, + workspaces: [{ path: "packages/c" }], + overrides: { + "packages/c": { name: "C" }, + "packages/a": { name: "A" }, + }, + }), + "utf8", + ); + + await resolveRecursionActivation(repo, undefined); + const first = await readManifest(repo); + await resolveRecursionActivation(repo, undefined); + const second = await readManifest(repo); + + expect(second).toBe(first); + const parsed = JSON.parse(first) as { + workspaces: { path: string }[]; + overrides: Record; + }; + expect(parsed.workspaces.map((w) => w.path)).toEqual([ + "packages/a", + "packages/b", + "packages/c", + ]); + expect(Object.keys(parsed.overrides)).toEqual(["packages/a", "packages/c"]); + }); + + test("BUG1: a legacy path-only manual entry for a non-detected but existing dir survives (promoted to include), no data loss", async () => { + const repo = await createRepo(); + await writePackagesRepo(repo, ["a"]); + // A hand-added service OUTSIDE the packages/* glob: detection cannot surface + // it, but the directory really exists, so it is a deliberate manual grouping. + await mkdir(path.join(repo, "services", "legacy-svc"), { recursive: true }); + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + await writeFile( + path.join(repo, "openwiki", "workspaces.json"), + JSON.stringify({ + version: 1, + workspaces: [{ path: "packages/a" }, { path: "services/legacy-svc" }], + }), + "utf8", + ); + + const warnings: string[] = []; + const activation = await resolveRecursionActivation( + repo, + undefined, + (message) => warnings.push(message), + ); + expect(activation.kind).toBe("recurse"); + if (activation.kind !== "recurse") return; + + // The manual path is preserved (not silently dropped) and promoted to an + // explicit include so it is stable across future runs. + expect(activation.manifest.workspaces.map((w) => w.path)).toEqual([ + "packages/a", + "services/legacy-svc", + ]); + expect(activation.manifest.overrides?.["services/legacy-svc"]).toEqual({ + include: true, + }); + expect(warnings).toEqual([]); + + // And it produces a run downstream (it has source, so it is not skipped). + const plan = resolveWorkspaceRuns(repo, activation.manifest); + expect(plan.runs.map((r) => r.relativePath)).toContain( + "services/legacy-svc", + ); + }); + + test("BUG1: a legacy goal-only manual entry for a non-detected but existing dir keeps its goal (promoted to include)", async () => { + const repo = await createRepo(); + await writePackagesRepo(repo, ["a"]); + await mkdir(path.join(repo, "services", "legacy-svc"), { recursive: true }); + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + await writeFile( + path.join(repo, "openwiki", "workspaces.json"), + JSON.stringify({ + version: 1, + workspaces: [ + { path: "packages/a" }, + { path: "services/legacy-svc", goal: "keep me" }, + ], + }), + "utf8", + ); + + const activation = await resolveRecursionActivation(repo, undefined); + expect(activation.kind).toBe("recurse"); + if (activation.kind !== "recurse") return; + + expect(activation.manifest.overrides?.["services/legacy-svc"]).toEqual({ + goal: "keep me", + include: true, + }); + const plan = resolveWorkspaceRuns(repo, activation.manifest); + const run = plan.runs.find((r) => r.relativePath === "services/legacy-svc"); + expect(run?.goal).toBe("keep me"); + }); + + test("BUG1 counterpart: a legacy manual entry for a dir that no longer exists is dropped (managed entry: quiet; override: warned)", async () => { + const repo = await createRepo(); + await writePackagesRepo(repo, ["a"]); + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + // Neither path exists on disk. One is a bare managed entry (quiet removal), + // the other carries an override (intent discarded → warn). + await writeFile( + path.join(repo, "openwiki", "workspaces.json"), + JSON.stringify({ + version: 1, + workspaces: [ + { path: "packages/a" }, + { path: "services/bare-gone" }, + { path: "services/customized-gone" }, + ], + overrides: { "services/customized-gone": { goal: "obsolete" } }, + }), + "utf8", + ); + + const warnings: string[] = []; + const activation = await resolveRecursionActivation( + repo, + undefined, + (message) => warnings.push(message), + ); + expect(activation.kind).toBe("recurse"); + if (activation.kind !== "recurse") return; + + expect(activation.manifest.workspaces.map((w) => w.path)).toEqual([ + "packages/a", + ]); + expect(warnings).toHaveLength(1); + expect(warnings[0]).toContain("services/customized-gone"); + }); + + test("BUG2: an include override overlapping a detected path does NOT wedge — it is pruned + warned and the run proceeds", async () => { + const repo = await createRepo(); + await writePackagesRepo(repo, ["a"]); + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + // `packages` is an ancestor of the detected `packages/a`; a naive union + // would persist both and make resolveWorkspaceRuns throw on every future + // default run (manifest present ⇒ auto-recurse ⇒ throw). + await writeFile( + path.join(repo, "openwiki", "workspaces.json"), + JSON.stringify({ + version: 1, + workspaces: [{ path: "packages/a" }], + overrides: { packages: { include: true } }, + }), + "utf8", + ); + + const warnings: string[] = []; + const activation = await resolveRecursionActivation( + repo, + undefined, + (message) => warnings.push(message), + ); + expect(activation.kind).toBe("recurse"); + if (activation.kind !== "recurse") return; + + // The overlapping include is pruned; only the detected leaf remains. + expect(activation.manifest.workspaces.map((w) => w.path)).toEqual([ + "packages/a", + ]); + expect(activation.manifest.overrides?.packages).toBeUndefined(); + expect(warnings.some((w) => w.includes("packages"))).toBe(true); + + // The written manifest must resolve cleanly (invariant restored): a + // subsequent default run does not throw. + const second = await resolveRecursionActivation(repo, undefined); + expect(second.kind).toBe("recurse"); + if (second.kind === "recurse") { + expect(() => resolveWorkspaceRuns(repo, second.manifest)).not.toThrow(); + } + }); + + test("#7 .NET coarsening: a per-project override orphaned when detection collapses to the area is dropped + warned, never wedges", async () => { + const repo = await createRepo(); + // Detection coarsens the src/tests projects of an area to the area root, so + // it emits `platform/admission`, not `platform/admission/src/...`. + await writeFile( + path.join(repo, "App.sln"), + [ + 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Api", "platform\\admission\\src\\Core.Admission.Api\\Core.Admission.Api.csproj", "{11111111-1111-1111-1111-111111111111}"', + "EndProject", + "", + ].join("\n"), + "utf8", + ); + await mkdir( + path.join(repo, "platform", "admission", "src", "Core.Admission.Api"), + { + recursive: true, + }, + ); + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + // A stale manifest keyed at the per-PROJECT granularity (pre-coarsening). + await writeFile( + path.join(repo, "openwiki", "workspaces.json"), + JSON.stringify({ + version: 1, + workspaces: [{ path: "platform/admission/src/Core.Admission.Api" }], + overrides: { + "platform/admission/src/Core.Admission.Api": { goal: "stale" }, + }, + }), + "utf8", + ); + + const warnings: string[] = []; + const activation = await resolveRecursionActivation( + repo, + undefined, + (message) => warnings.push(message), + ); + expect(activation.kind).toBe("recurse"); + if (activation.kind !== "recurse") return; + + // Only the coarsened area survives; the per-project path (whose dir exists + // but overlaps the detected area) is pruned + warned rather than persisted + // into an unresolvable manifest. + expect(activation.manifest.workspaces.map((w) => w.path)).toEqual([ + "platform/admission", + ]); + expect( + activation.manifest.overrides?.[ + "platform/admission/src/Core.Admission.Api" + ], + ).toBeUndefined(); + expect(warnings.length).toBeGreaterThanOrEqual(1); + expect( + warnings.some((w) => + w.includes("platform/admission/src/Core.Admission.Api"), + ), + ).toBe(true); + expect(() => resolveWorkspaceRuns(repo, activation.manifest)).not.toThrow(); + }); +}); diff --git a/test/workspaces.test.ts b/test/workspaces.test.ts index 2bf62ab9..5ef4ddf0 100644 --- a/test/workspaces.test.ts +++ b/test/workspaces.test.ts @@ -89,6 +89,46 @@ describe("normalizeManifest", () => { const manifest = normalizeManifest({ workspaces: [{ path: "a" }] }); expect(manifest.version).toBe(1); }); + + test("parses an overrides map, keeping only recognized typed fields", () => { + const manifest = normalizeManifest({ + version: 1, + workspaces: [{ path: "a" }], + overrides: { + a: { + goal: "g", + name: "N", + exclude: true, + include: false, + bogus: 42, + }, + }, + }); + expect(manifest.overrides?.a).toEqual({ + goal: "g", + name: "N", + exclude: true, + include: false, + }); + }); + + test("migrates legacy per-entry goal/name into overrides (explicit override wins)", () => { + const manifest = normalizeManifest({ + version: 1, + workspaces: [ + { path: "a", goal: "legacy a", name: "LegacyA" }, + { path: "b", goal: "legacy b" }, + ], + overrides: { a: { goal: "explicit a" } }, + }); + // Explicit override goal wins; the missing name is filled from the entry. + expect(manifest.overrides?.a).toEqual({ + goal: "explicit a", + name: "LegacyA", + }); + // A legacy entry with no override migrates wholesale. + expect(manifest.overrides?.b).toEqual({ goal: "legacy b" }); + }); }); describe("resolveWorkspaceRuns", () => { @@ -165,6 +205,34 @@ describe("resolveWorkspaceRuns", () => { const plan = resolveWorkspaceRuns(repoRoot, manifest([])); expect(plan.runs).toHaveLength(0); }); + + test("an override with exclude:true produces no run", () => { + const plan = resolveWorkspaceRuns(repoRoot, { + version: 1, + workspaces: [{ path: "packages/a" }, { path: "packages/b" }], + overrides: { "packages/b": { exclude: true } }, + }); + expect(plan.runs.map((run) => run.relativePath)).toEqual(["packages/a"]); + }); + + test("an override goal/name takes precedence over a legacy per-entry value", () => { + const plan = resolveWorkspaceRuns(repoRoot, { + version: 1, + workspaces: [{ path: "packages/a", goal: "entry goal", name: "Entry" }], + overrides: { "packages/a": { goal: "override goal", name: "Override" } }, + }); + expect(plan.runs[0].goal).toBe("override goal"); + expect(plan.runs[0].name).toBe("Override"); + }); + + test("falls back to a legacy per-entry goal/name when no override exists", () => { + const plan = resolveWorkspaceRuns(repoRoot, { + version: 1, + workspaces: [{ path: "packages/a", goal: "entry goal", name: "Entry" }], + }); + expect(plan.runs[0].goal).toBe("entry goal"); + expect(plan.runs[0].name).toBe("Entry"); + }); }); describe("getWorkspaceSkipReason", () => { @@ -359,7 +427,7 @@ describe("detectWorkspaces", () => { await writeFile( path.join(repo, "App.sln"), [ - 'Microsoft Visual Studio Solution File, Format Version 12.00', + "Microsoft Visual Studio Solution File, Format Version 12.00", 'Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Api", "src\\Api\\Api.csproj", "{11111111-1111-1111-1111-111111111111}"', "EndProject", 'Project("{F184B08F-C81C-45F6-A57F-5ABD9991F28F}") = "Lib", "src\\Lib\\Lib.vbproj", "{22222222-2222-2222-2222-222222222222}"', @@ -738,7 +806,11 @@ describe("detectWorkspaces", () => { test("malformed pom.xml returns no Maven workspaces", async () => { const repo = await createTempRepo(); - await writeFile(path.join(repo, "pom.xml"), "no modules", "utf8"); + await writeFile( + path.join(repo, "pom.xml"), + "no modules", + "utf8", + ); expect(await detectWorkspaces(repo)).toEqual([]); }); }); @@ -1000,7 +1072,10 @@ describe("detectWorkspaces -> resolveWorkspaceRuns seam", () => { await mkdir(path.join(repo, "src/Web"), { recursive: true }); const detected = await detectWorkspaces(repo); - const plan = resolveWorkspaceRuns(repo, { version: 1, workspaces: detected }); + const plan = resolveWorkspaceRuns(repo, { + version: 1, + workspaces: detected, + }); expect(plan.runs.map((run) => run.relativePath).sort()).toEqual([ "src/Api", "src/Web", @@ -1032,7 +1107,10 @@ describe("detectWorkspaces -> resolveWorkspaceRuns seam", () => { }); const detected = await detectWorkspaces(repo); - const plan = resolveWorkspaceRuns(repo, { version: 1, workspaces: detected }); + const plan = resolveWorkspaceRuns(repo, { + version: 1, + workspaces: detected, + }); expect(plan.runs.map((run) => run.relativePath).sort()).toEqual([ "kernel/Core.Domain.Kernel", "platform/admission", From 8871cc54acc76c3627e45fc166c365429406f1b4 Mon Sep 17 00:00:00 2001 From: Brad Huffman Date: Wed, 22 Jul 2026 19:06:44 -0500 Subject: [PATCH 3/5] fix: guard generated-file writes against symlink following (security) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The recursive monorepo generator writes files inside the repository being documented (openwiki/workspaces.md, workspaces.json, .workspaces-state.json). Both the destination paths and their contents are attacker-influenced — the repo under documentation is untrusted input — and `writeFile` follows symlinks by default. A malicious repo could commit one of these paths as a symlink to a file outside the repo (e.g. ~/.bashrc); running recursive OpenWiki would then follow the link and overwrite that target with generated content. Adds `writeGeneratedFile`, which before writing: - refuses if the destination is a symlink (lstat, does not follow it), and - refuses if the resolved parent directory escapes the repo root (realpath), catching a symlinked ancestor directory. Routes all four generated-file writes through it (the aggregation index plus the manifest and state writers, which had the same exposure the reporter's finding did not enumerate). Fails loudly on a rejected path rather than writing through the link. Addresses the Corridor review finding on #439. Co-Authored-By: Claude --- src/monorepo/orchestrator.ts | 9 +++-- src/monorepo/workspaces.ts | 72 +++++++++++++++++++++++++++++++++--- test/workspaces.test.ts | 48 ++++++++++++++++++++++++ 3 files changed, 120 insertions(+), 9 deletions(-) diff --git a/src/monorepo/orchestrator.ts b/src/monorepo/orchestrator.ts index 9cc5b939..da0efd40 100644 --- a/src/monorepo/orchestrator.ts +++ b/src/monorepo/orchestrator.ts @@ -1,5 +1,4 @@ import { execFile } from "node:child_process"; -import { mkdir, writeFile } from "node:fs/promises"; import path from "node:path"; import { promisify } from "node:util"; import { @@ -20,6 +19,7 @@ import { getWorkspaceSkipReason, readWorkspacesState, resolveWorkspaceRuns, + writeGeneratedFile, writeWorkspacesState, type ResolvedWorkspacePlan, type ResolvedWorkspaceRun, @@ -219,7 +219,6 @@ export async function writeRootAggregation( plan: ResolvedWorkspacePlan, ): Promise { const openWikiDir = path.join(repoRoot, "openwiki"); - await mkdir(openWikiDir, { recursive: true }); const rows = plan.runs .map((run) => { @@ -243,7 +242,11 @@ This monorepo documents each subproject in its own OpenWiki sub-wiki. This page ${rows || "No documented subprojects."} `; - await writeFile(path.join(openWikiDir, "workspaces.md"), content, "utf8"); + await writeGeneratedFile( + repoRoot, + path.join(openWikiDir, "workspaces.md"), + content, + ); } /** diff --git a/src/monorepo/workspaces.ts b/src/monorepo/workspaces.ts index 56e6d030..fde32d76 100644 --- a/src/monorepo/workspaces.ts +++ b/src/monorepo/workspaces.ts @@ -1,4 +1,5 @@ import { + lstat, mkdir, readFile, readdir, @@ -414,13 +415,70 @@ export function serializeManifest(manifest: WorkspaceManifest): string { * can review and edit it. Written with a comment-free JSON body and a trailing * newline. */ +/** + * Safely writes a generated file that lives inside a repository OpenWiki is + * documenting. Both the path AND its contents are attacker-influenced (the repo + * under documentation is untrusted input), so a plain `writeFile` is unsafe: + * `writeFile` follows symlinks, so a repo that commits the destination as a + * symlink (e.g. `openwiki/workspaces.md` -> `~/.bashrc`) would have that target + * silently overwritten with generated content when a developer runs recursive + * OpenWiki. Guard against it: + * + * 1. `lstat` the destination and refuse if it is a symlink (do NOT follow it). + * 2. `realpath` the parent directory and refuse if it resolves outside the repo + * root (a symlinked ancestor directory would otherwise escape the repo). + * + * Throws on a rejected path so a malicious layout fails loudly rather than + * writing through the link. + */ +export async function writeGeneratedFile( + repoRoot: string, + absolutePath: string, + content: string, +): Promise { + const parentDir = path.dirname(absolutePath); + await mkdir(parentDir, { recursive: true }); + + // The parent must resolve to a real location inside the repo. realpath follows + // symlinks, so a symlinked ancestor pointing outside the repo is caught here. + const realRepoRoot = await realpath(repoRoot); + const realParent = await realpath(parentDir); + const relativeParent = path.relative(realRepoRoot, realParent); + if ( + relativeParent !== "" && + (relativeParent.startsWith("..") || path.isAbsolute(relativeParent)) + ) { + throw new Error( + `Refusing to write ${JSON.stringify( + path.relative(repoRoot, absolutePath), + )}: its directory resolves outside the repository (symlink escape).`, + ); + } + + // The destination itself must not be a symlink; writeFile would follow it and + // clobber the link target. lstat does not follow the final component. + const existing = await lstat(absolutePath).catch(() => null); + if (existing?.isSymbolicLink()) { + throw new Error( + `Refusing to write ${JSON.stringify( + path.relative(repoRoot, absolutePath), + )}: the destination is a symlink; writing would follow it and overwrite an arbitrary file.`, + ); + } + + await writeFile(absolutePath, content, "utf8"); +} + export async function writeWorkspaceManifest( repoRoot: string, manifest: WorkspaceManifest, ): Promise { const manifestPath = path.join(repoRoot, WORKSPACES_MANIFEST_RELATIVE_PATH); - await mkdir(path.dirname(manifestPath), { recursive: true }); - await writeFile(manifestPath, serializeManifest(manifest), "utf8"); + await writeGeneratedFile( + repoRoot, + manifestPath, + serializeManifest(manifest), + ); } /** @@ -441,8 +499,7 @@ export async function writeWorkspaceManifestIfChanged( return false; } - await mkdir(path.dirname(manifestPath), { recursive: true }); - await writeFile(manifestPath, target, "utf8"); + await writeGeneratedFile(repoRoot, manifestPath, target); return true; } @@ -1588,8 +1645,11 @@ export async function writeWorkspacesState( state: WorkspacesState, ): Promise { const statePath = path.join(repoRoot, WORKSPACES_STATE_RELATIVE_PATH); - await mkdir(path.dirname(statePath), { recursive: true }); - await writeFile(statePath, `${JSON.stringify(state, null, 2)}\n`, "utf8"); + await writeGeneratedFile( + repoRoot, + statePath, + `${JSON.stringify(state, null, 2)}\n`, + ); } function isRecord(value: unknown): value is Record { diff --git a/test/workspaces.test.ts b/test/workspaces.test.ts index 5ef4ddf0..1d3ba5e4 100644 --- a/test/workspaces.test.ts +++ b/test/workspaces.test.ts @@ -2,12 +2,14 @@ import { mkdtemp, mkdir, rm, writeFile, symlink } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; import { afterEach, describe, expect, test } from "vitest"; +import { lstat, readFile } from "node:fs/promises"; import { detectWorkspaces, getWorkspaceSkipReason, normalizeManifest, readWorkspaceManifest, resolveWorkspaceRuns, + writeGeneratedFile, type WorkspaceManifest, } from "../src/monorepo/workspaces.ts"; @@ -1117,3 +1119,49 @@ describe("detectWorkspaces -> resolveWorkspaceRuns seam", () => { ]); }); }); + +describe("writeGeneratedFile (symlink-following guard)", () => { + test("writes a normal file inside the repo", async () => { + const repo = await createTempRepo(); + const target = path.join(repo, "openwiki", "workspaces.md"); + await writeGeneratedFile(repo, target, "hello\n"); + expect(await readFile(target, "utf8")).toBe("hello\n"); + }); + + test("refuses to follow a symlinked destination and does not clobber its target", async () => { + const repo = await createTempRepo(); + const outside = await createTempRepo(); + const victim = path.join(outside, "victim.txt"); + await writeFile(victim, "original\n"); + + // A malicious repo commits openwiki/workspaces.md as a symlink to a file + // outside the repo; the write must refuse rather than follow it. + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + const link = path.join(repo, "openwiki", "workspaces.md"); + await symlink(victim, link); + + await expect( + writeGeneratedFile(repo, link, "attacker content\n"), + ).rejects.toThrow(/symlink/); + // The link target outside the repo is untouched. + expect(await readFile(victim, "utf8")).toBe("original\n"); + // The path on disk is still a symlink, never overwritten as a real file. + expect((await lstat(link)).isSymbolicLink()).toBe(true); + }); + + test("refuses when the parent directory resolves outside the repo", async () => { + const repo = await createTempRepo(); + const outside = await createTempRepo(); + // openwiki/ itself is a symlink to a directory outside the repo, so the + // resolved write parent escapes the repository. + await symlink(outside, path.join(repo, "openwiki")); + + await expect( + writeGeneratedFile( + repo, + path.join(repo, "openwiki", "workspaces.md"), + "x\n", + ), + ).rejects.toThrow(/outside the repository/); + }); +}); From 1f8588e4e640e7b8936b510ba75e4ab9d7286284 Mon Sep 17 00:00:00 2001 From: Brad Huffman Date: Wed, 22 Jul 2026 19:18:53 -0500 Subject: [PATCH 4/5] fix: guard code-mode workflow/agent-file writes against symlinks (security) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second Corridor finding on #439: `ensureCodeModeRepoSetup` wrote .github/workflows/openwiki-update.yml and AGENTS.md/CLAUDE.md into the target repo with bare `writeFile`, which follows symlinks — the same CWE-59 class as the workspaces.md finding. A malicious repo could commit the workflow path as a symlink to e.g. ~/.ssh/authorized_keys and have it overwritten when a developer runs OpenWiki in code mode. Extracts the `writeGeneratedFile` guard (introduced for the workspaces writes) into a shared `src/safe-write.ts` — it has nothing monorepo-specific about it, and code-mode is a lower layer that should not depend on the workspaces module. Routes both code-mode write paths through it, so every file OpenWiki writes into an untrusted target repo now refuses a symlinked destination (lstat) or a parent that escapes the repo root (realpath). Moves the guard's tests to test/safe-write.test.ts and adds a workflow-file (repo-root) attack case. Co-Authored-By: Claude --- src/code-mode.ts | 13 +++--- src/monorepo/orchestrator.ts | 2 +- src/monorepo/workspaces.ts | 65 +--------------------------- src/safe-write.ts | 56 ++++++++++++++++++++++++ test/safe-write.test.ts | 83 ++++++++++++++++++++++++++++++++++++ test/workspaces.test.ts | 48 --------------------- 6 files changed, 149 insertions(+), 118 deletions(-) create mode 100644 src/safe-write.ts create mode 100644 test/safe-write.test.ts diff --git a/src/code-mode.ts b/src/code-mode.ts index a9651cb8..87ec792f 100644 --- a/src/code-mode.ts +++ b/src/code-mode.ts @@ -1,7 +1,8 @@ -import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { readFile } from "node:fs/promises"; import path from "node:path"; import { OPENWIKI_VERSION } from "./constants.js"; import { isFileNotFoundError } from "./fs-errors.js"; +import { writeGeneratedFile } from "./safe-write.js"; const OPENWIKI_AGENTS_SNIPPET_START = ""; const OPENWIKI_AGENTS_SNIPPET_END = ""; @@ -44,11 +45,10 @@ async function writeCodeModeWorkflow( "workflows", "openwiki-update.yml", ); - await mkdir(path.dirname(workflowPath), { recursive: true }); - await writeFile( + await writeGeneratedFile( + cwd, workflowPath, createCodeModeWorkflow(cronExpression, recursive), - "utf8", ); } @@ -57,12 +57,13 @@ async function writeCodeModeAgentSnippets(cwd: string): Promise { await Promise.all( CODE_MODE_AGENT_FILES.map((fileName) => - writeCodeModeAgentSnippet(path.join(cwd, fileName), snippet), + writeCodeModeAgentSnippet(cwd, path.join(cwd, fileName), snippet), ), ); } async function writeCodeModeAgentSnippet( + repoRoot: string, agentsPath: string, snippet: string, ): Promise { @@ -83,7 +84,7 @@ async function writeCodeModeAgentSnippet( ? `${currentContent.slice(0, startIndex)}${snippet}${currentContent.slice(endIndex + OPENWIKI_AGENTS_SNIPPET_END.length)}` : `${currentContent.trimEnd()}${currentContent.trim().length > 0 ? "\n\n" : ""}${snippet}\n`; - await writeFile(agentsPath, nextContent, "utf8"); + await writeGeneratedFile(repoRoot, agentsPath, nextContent); } function createCodeModeWorkflow( diff --git a/src/monorepo/orchestrator.ts b/src/monorepo/orchestrator.ts index da0efd40..f6724636 100644 --- a/src/monorepo/orchestrator.ts +++ b/src/monorepo/orchestrator.ts @@ -19,13 +19,13 @@ import { getWorkspaceSkipReason, readWorkspacesState, resolveWorkspaceRuns, - writeGeneratedFile, writeWorkspacesState, type ResolvedWorkspacePlan, type ResolvedWorkspaceRun, type WorkspaceManifest, type WorkspacesState, } from "./workspaces.js"; +import { writeGeneratedFile } from "../safe-write.js"; const execFileAsync = promisify(execFile); diff --git a/src/monorepo/workspaces.ts b/src/monorepo/workspaces.ts index fde32d76..bcf8f6a3 100644 --- a/src/monorepo/workspaces.ts +++ b/src/monorepo/workspaces.ts @@ -1,14 +1,7 @@ -import { - lstat, - mkdir, - readFile, - readdir, - realpath, - stat, - writeFile, -} from "node:fs/promises"; +import { readFile, readdir, realpath, stat } from "node:fs/promises"; import path from "node:path"; import { parse as parseYaml } from "yaml"; +import { writeGeneratedFile } from "../safe-write.js"; /** * One entry in the managed `workspaces` list of `openwiki/workspaces.json`. @@ -415,60 +408,6 @@ export function serializeManifest(manifest: WorkspaceManifest): string { * can review and edit it. Written with a comment-free JSON body and a trailing * newline. */ -/** - * Safely writes a generated file that lives inside a repository OpenWiki is - * documenting. Both the path AND its contents are attacker-influenced (the repo - * under documentation is untrusted input), so a plain `writeFile` is unsafe: - * `writeFile` follows symlinks, so a repo that commits the destination as a - * symlink (e.g. `openwiki/workspaces.md` -> `~/.bashrc`) would have that target - * silently overwritten with generated content when a developer runs recursive - * OpenWiki. Guard against it: - * - * 1. `lstat` the destination and refuse if it is a symlink (do NOT follow it). - * 2. `realpath` the parent directory and refuse if it resolves outside the repo - * root (a symlinked ancestor directory would otherwise escape the repo). - * - * Throws on a rejected path so a malicious layout fails loudly rather than - * writing through the link. - */ -export async function writeGeneratedFile( - repoRoot: string, - absolutePath: string, - content: string, -): Promise { - const parentDir = path.dirname(absolutePath); - await mkdir(parentDir, { recursive: true }); - - // The parent must resolve to a real location inside the repo. realpath follows - // symlinks, so a symlinked ancestor pointing outside the repo is caught here. - const realRepoRoot = await realpath(repoRoot); - const realParent = await realpath(parentDir); - const relativeParent = path.relative(realRepoRoot, realParent); - if ( - relativeParent !== "" && - (relativeParent.startsWith("..") || path.isAbsolute(relativeParent)) - ) { - throw new Error( - `Refusing to write ${JSON.stringify( - path.relative(repoRoot, absolutePath), - )}: its directory resolves outside the repository (symlink escape).`, - ); - } - - // The destination itself must not be a symlink; writeFile would follow it and - // clobber the link target. lstat does not follow the final component. - const existing = await lstat(absolutePath).catch(() => null); - if (existing?.isSymbolicLink()) { - throw new Error( - `Refusing to write ${JSON.stringify( - path.relative(repoRoot, absolutePath), - )}: the destination is a symlink; writing would follow it and overwrite an arbitrary file.`, - ); - } - - await writeFile(absolutePath, content, "utf8"); -} - export async function writeWorkspaceManifest( repoRoot: string, manifest: WorkspaceManifest, diff --git a/src/safe-write.ts b/src/safe-write.ts new file mode 100644 index 00000000..744ea6cc --- /dev/null +++ b/src/safe-write.ts @@ -0,0 +1,56 @@ +import { lstat, mkdir, realpath, writeFile } from "node:fs/promises"; +import path from "node:path"; + +/** + * Safely writes a file that lives inside a repository OpenWiki is documenting. + * + * OpenWiki writes several files into the target repo (the recursive-monorepo + * manifest/index/state, plus the code-mode workflow and AGENTS.md/CLAUDE.md). + * Both the destination paths and their contents are attacker-influenced — the + * repo under documentation is untrusted input — and `writeFile` follows symlinks + * by default. A malicious repo could commit one of these paths as a symlink to a + * file outside the repo (e.g. `~/.ssh/authorized_keys` or `~/.bashrc`); running + * OpenWiki would then follow the link and overwrite that target with generated + * content (CWE-59). Guard against it before writing: + * + * 1. `realpath` the parent directory and refuse if it resolves outside the repo + * root — catches a symlinked ancestor directory pointing out of the repo. + * 2. `lstat` the destination and refuse if it is a symlink (lstat does not + * follow the final component), so the write can never follow the link. + * + * Throws on a rejected path so a malicious layout fails loudly rather than + * writing through the link. + */ +export async function writeGeneratedFile( + repoRoot: string, + absolutePath: string, + content: string, +): Promise { + const parentDir = path.dirname(absolutePath); + await mkdir(parentDir, { recursive: true }); + + const realRepoRoot = await realpath(repoRoot); + const realParent = await realpath(parentDir); + const relativeParent = path.relative(realRepoRoot, realParent); + if ( + relativeParent !== "" && + (relativeParent.startsWith("..") || path.isAbsolute(relativeParent)) + ) { + throw new Error( + `Refusing to write ${JSON.stringify( + path.relative(repoRoot, absolutePath), + )}: its directory resolves outside the repository (symlink escape).`, + ); + } + + const existing = await lstat(absolutePath).catch(() => null); + if (existing?.isSymbolicLink()) { + throw new Error( + `Refusing to write ${JSON.stringify( + path.relative(repoRoot, absolutePath), + )}: the destination is a symlink; writing would follow it and overwrite an arbitrary file.`, + ); + } + + await writeFile(absolutePath, content, "utf8"); +} diff --git a/test/safe-write.test.ts b/test/safe-write.test.ts new file mode 100644 index 00000000..bb9dfc11 --- /dev/null +++ b/test/safe-write.test.ts @@ -0,0 +1,83 @@ +import { lstat, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, test } from "vitest"; +import { writeGeneratedFile } from "../src/safe-write.ts"; + +const tempDirs: string[] = []; + +async function createTempRepo(): Promise { + const repo = await mkdtemp(path.join(tmpdir(), "openwiki-safe-write-")); + tempDirs.push(repo); + return repo; +} + +afterEach(async () => { + await Promise.all( + tempDirs.splice(0).map((dir) => rm(dir, { force: true, recursive: true })), + ); +}); + +describe("writeGeneratedFile (symlink-following guard)", () => { + test("writes a normal file inside the repo, creating parents", async () => { + const repo = await createTempRepo(); + const target = path.join(repo, "openwiki", "workspaces.md"); + await writeGeneratedFile(repo, target, "hello\n"); + expect(await readFile(target, "utf8")).toBe("hello\n"); + }); + + test("refuses to follow a symlinked destination and does not clobber its target", async () => { + const repo = await createTempRepo(); + const outside = await createTempRepo(); + const victim = path.join(outside, "victim.txt"); + await writeFile(victim, "original\n"); + + // A malicious repo commits the destination as a symlink to a file outside + // the repo; the write must refuse rather than follow it (CWE-59). + await mkdir(path.join(repo, "openwiki"), { recursive: true }); + const link = path.join(repo, "openwiki", "workspaces.md"); + await symlink(victim, link); + + await expect( + writeGeneratedFile(repo, link, "attacker content\n"), + ).rejects.toThrow(/symlink/); + // The link target outside the repo is untouched. + expect(await readFile(victim, "utf8")).toBe("original\n"); + // The path on disk is still a symlink, never overwritten as a real file. + expect((await lstat(link)).isSymbolicLink()).toBe(true); + }); + + test("refuses a symlinked destination at the repo root (e.g. a workflow file)", async () => { + const repo = await createTempRepo(); + const outside = await createTempRepo(); + const victim = path.join(outside, "authorized_keys"); + await writeFile(victim, "original-key\n"); + + // Mirrors the .github/workflows/openwiki-update.yml attack: a repo-root + // path committed as a symlink pointing outside the repo. + await mkdir(path.join(repo, ".github", "workflows"), { recursive: true }); + const link = path.join(repo, ".github", "workflows", "openwiki-update.yml"); + await symlink(victim, link); + + await expect( + writeGeneratedFile(repo, link, "name: pwned\n"), + ).rejects.toThrow(/symlink/); + expect(await readFile(victim, "utf8")).toBe("original-key\n"); + }); + + test("refuses when the parent directory resolves outside the repo", async () => { + const repo = await createTempRepo(); + const outside = await createTempRepo(); + // openwiki/ itself is a symlink to a directory outside the repo, so the + // resolved write parent escapes the repository. + await symlink(outside, path.join(repo, "openwiki")); + + await expect( + writeGeneratedFile( + repo, + path.join(repo, "openwiki", "workspaces.md"), + "x\n", + ), + ).rejects.toThrow(/outside the repository/); + }); +}); diff --git a/test/workspaces.test.ts b/test/workspaces.test.ts index 1d3ba5e4..5ef4ddf0 100644 --- a/test/workspaces.test.ts +++ b/test/workspaces.test.ts @@ -2,14 +2,12 @@ import { mkdtemp, mkdir, rm, writeFile, symlink } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; import { afterEach, describe, expect, test } from "vitest"; -import { lstat, readFile } from "node:fs/promises"; import { detectWorkspaces, getWorkspaceSkipReason, normalizeManifest, readWorkspaceManifest, resolveWorkspaceRuns, - writeGeneratedFile, type WorkspaceManifest, } from "../src/monorepo/workspaces.ts"; @@ -1119,49 +1117,3 @@ describe("detectWorkspaces -> resolveWorkspaceRuns seam", () => { ]); }); }); - -describe("writeGeneratedFile (symlink-following guard)", () => { - test("writes a normal file inside the repo", async () => { - const repo = await createTempRepo(); - const target = path.join(repo, "openwiki", "workspaces.md"); - await writeGeneratedFile(repo, target, "hello\n"); - expect(await readFile(target, "utf8")).toBe("hello\n"); - }); - - test("refuses to follow a symlinked destination and does not clobber its target", async () => { - const repo = await createTempRepo(); - const outside = await createTempRepo(); - const victim = path.join(outside, "victim.txt"); - await writeFile(victim, "original\n"); - - // A malicious repo commits openwiki/workspaces.md as a symlink to a file - // outside the repo; the write must refuse rather than follow it. - await mkdir(path.join(repo, "openwiki"), { recursive: true }); - const link = path.join(repo, "openwiki", "workspaces.md"); - await symlink(victim, link); - - await expect( - writeGeneratedFile(repo, link, "attacker content\n"), - ).rejects.toThrow(/symlink/); - // The link target outside the repo is untouched. - expect(await readFile(victim, "utf8")).toBe("original\n"); - // The path on disk is still a symlink, never overwritten as a real file. - expect((await lstat(link)).isSymbolicLink()).toBe(true); - }); - - test("refuses when the parent directory resolves outside the repo", async () => { - const repo = await createTempRepo(); - const outside = await createTempRepo(); - // openwiki/ itself is a symlink to a directory outside the repo, so the - // resolved write parent escapes the repository. - await symlink(outside, path.join(repo, "openwiki")); - - await expect( - writeGeneratedFile( - repo, - path.join(repo, "openwiki", "workspaces.md"), - "x\n", - ), - ).rejects.toThrow(/outside the repository/); - }); -}); From 63c4dfd5541d6f6f377347188a4080a1c10c1c41 Mon Sep 17 00:00:00 2001 From: Brad Huffman Date: Wed, 22 Jul 2026 19:41:02 -0500 Subject: [PATCH 5/5] docs: explain why recursive monorepo docs have no dependency cascade Reframe the "no dependency cascade" note from a limitation-awaiting-a-fix into a by-design boundary. A subproject run is isolated to its own subtree (filesystem rooted at the subproject dir; git evidence scoped with `-- .`), so it cannot read a dependency's source. Re-running a dependent after an unrelated dependency change would have no new information to act on. Public API changes already surface in the dependent's own subtree diff and regenerate normally; only dependency internals (invisible to dependents) are skipped, which is the desired behavior. Closes ppsplus-bradh#2. Co-Authored-By: Claude --- README.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 9f998898..6117c832 100644 --- a/README.md +++ b/README.md @@ -253,14 +253,22 @@ next write emits the current shape without losing them. its own `openwiki/.last-update.json`). Unchanged subprojects are skipped without a model call, so scheduled refreshes stay cheap. The root wiki regenerates on every run. -- **Known limitation — no dependency cascade.** Updates do not propagate across - subprojects. If a shared subproject (for example a common kernel or contracts - package) changes, only _that_ sub-wiki and the root wiki are refreshed — the - sibling subprojects that depend on it are **not** automatically regenerated, - even though their documented context may reference the changed code. Until - dependency-aware invalidation exists, force a full refresh after a significant - shared-code change by removing the relevant `openwiki/.last-update.json` files - (or running each affected subproject explicitly). +- **No dependency cascade (by design).** Updates do not propagate across + subprojects: if a shared subproject (for example a common kernel or contracts + package) changes, only _that_ sub-wiki and the root wiki are refreshed, not the + siblings that depend on it. This is intentional, not a missing feature. Each + subproject run is **isolated to its own subtree** — the filesystem tools are + rooted at the subproject directory and its git evidence is scoped with a `-- .` + pathspec — so a run cannot read a dependency's source even if it wanted to. + Re-running a dependent after an unrelated dependency change would therefore + have no new information to incorporate and would just reproduce the same + sub-wiki. In practice this is not a gap: when a dependency's _public API_ + changes, the dependent's own call sites change too, which shows up in the + dependent's own subtree diff and regenerates it normally. Only changes to a + dependency's _internals_ (invisible to dependents) are skipped, which is the + desired behavior. If you still want to force a refresh after a significant + shared-code change, remove the relevant `openwiki/.last-update.json` files (or + run each affected subproject explicitly). On the first interactive run, OpenWiki will have you configure your inference provider, API key, and LLM. You will also be able to set a LangSmith API key to trace your OpenWiki runs to a LangSmith tracing project named "openwiki" (optional).