feat: recursive documentation for monorepos (#162) - #439
Open
Brad Huffman (ppsplus-bradh) wants to merge 12 commits into
Open
feat: recursive documentation for monorepos (#162)#439Brad Huffman (ppsplus-bradh) wants to merge 12 commits into
Brad Huffman (ppsplus-bradh) wants to merge 12 commits into
Conversation
Adds a recursive documentation strategy so large monorepos can be documented as a tree of wikis: each subproject gets its own nested `<subproject>/openwiki/` sub-wiki documenting its subtree in depth, while the repository-root wiki links down to them and covers only cross-cutting concerns. Closes the request in langchain-ai#162. How it works: - Subprojects come from an `openwiki/workspaces.json` manifest, or from `--recursive` auto-detection of common workspace layouts (pnpm/npm/yarn, Cargo, Go, uv, Gradle, Maven, .NET solutions, Bazel), which writes a manifest for review then proceeds. Manifest presence auto-enables recursion; `--recursive=false` forces a single run. - An orchestrator runs the existing per-repo core once per subproject (backend rooted at the subproject so the docs-only write guard, snapshot, metadata, plan cleanup, and index-sync all scope for free), then writes a generated `openwiki/workspaces.md` aggregation index, then the root run last so its index-sync links the aggregation in. One model is resolved and reused across all runs; runs are sequential. - Git evidence and the incremental no-op check are scoped per subproject (subproject-relative diff), so `--update` regenerates only subprojects whose own subtree changed and skips the rest without a model call. Root git evidence excludes nested `**/openwiki`. - .NET solution detection coarsens per-project paths to product-area roots (`area/src/X`, `area/tests/X` -> `area`) to avoid one wiki per .csproj. - Backward compatible: with no manifest and no flag, behavior is unchanged. Documents the current limitation that updates do not cascade across subprojects (a shared-dependency change refreshes only that sub-wiki and the root, not its dependents) in README.md and the orchestrator JSDoc. Co-Authored-By: Claude <noreply@anthropic.com>
Makes `--recursive` keep the workspace manifest current as the repo evolves,
so a scheduled CI run picks up newly added projects and drops removed ones
without manual manifest edits.
- Discovery re-runs on every recursive run (deterministic, ~20ms even on a
400-project repo) and MERGES with the existing openwiki/workspaces.json
rather than only auto-detecting when no manifest exists.
- New manifest schema separates concerns: a managed `workspaces` list
(detection-owned, path-only, sorted, regenerated each run) and a
hand-authored `overrides` map keyed by path (goal / name / exclude /
include), so customization and auto-discovery never collide.
- Idempotent write: the manifest is serialized to canonical bytes (fixed key
order, sorted entries) and rewritten only when it actually changed, so an
unchanged repo produces no manifest diff and no spurious CI commit.
- Manual groupings are preserved: a referenced path detection cannot surface
whose directory still exists is kept and promoted to `include: true`
(carrying any goal/name); one whose directory is gone is pruned, with a
warning when it carried a hand-authored override.
- An `include` override that would overlap a detected workspace is dropped
with a warning instead of being persisted, preserving the "never write an
unresolvable manifest" invariant.
- Legacy flat manifests (workspaces:[{path,goal,name}]) are read and migrated
in-memory, then re-emitted in the new shape without losing goals.
Documents the manual-preservation and overlap-guard behavior in README.md.
Co-Authored-By: Claude <noreply@anthropic.com>
![corridor-security[bot]](https://avatars.githubusercontent.com/in/1177339?s=60&v=4){kind=small}
There was a problem hiding this comment.
The recursive monorepo aggregation writes openwiki/workspaces.md via writeFile() without checking whether the destination is a symlink; a malicious repo can commit that path as a symlink to an arbitrary file outside the repo, causing the write to follow it and overwrite that target with attacker-influenced content when a developer runs recursive OpenWiki.
…repo-docs # Conflicts: # test/code-mode.test.ts # test/prompt.test.ts
The recursive monorepo generator writes files inside the repository being documented (openwiki/workspaces.md, workspaces.json, .workspaces-state.json). Both the destination paths and their contents are attacker-influenced — the repo under documentation is untrusted input — and `writeFile` follows symlinks by default. A malicious repo could commit one of these paths as a symlink to a file outside the repo (e.g. ~/.bashrc); running recursive OpenWiki would then follow the link and overwrite that target with generated content. Adds `writeGeneratedFile`, which before writing: - refuses if the destination is a symlink (lstat, does not follow it), and - refuses if the resolved parent directory escapes the repo root (realpath), catching a symlinked ancestor directory. Routes all four generated-file writes through it (the aggregation index plus the manifest and state writers, which had the same exposure the reporter's finding did not enumerate). Fails loudly on a rejected path rather than writing through the link. Addresses the Corridor review finding on langchain-ai#439. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
This PR introduces an arbitrary file overwrite vulnerability via symlink following in src/code-mode.ts, where the GitHub Actions workflow file is written with an unguarded writeFile() call that will follow a symlink committed by a malicious repository. The safe writeGeneratedFile() helper introduced elsewhere in the PR (which checks for symlinks via lstat and validates parent directory containment) is not applied to this write or to the AGENTS.md/CLAUDE.md writes in the same file.
…urity) Second Corridor finding on langchain-ai#439: `ensureCodeModeRepoSetup` wrote .github/workflows/openwiki-update.yml and AGENTS.md/CLAUDE.md into the target repo with bare `writeFile`, which follows symlinks — the same CWE-59 class as the workspaces.md finding. A malicious repo could commit the workflow path as a symlink to e.g. ~/.ssh/authorized_keys and have it overwritten when a developer runs OpenWiki in code mode. Extracts the `writeGeneratedFile` guard (introduced for the workspaces writes) into a shared `src/safe-write.ts` — it has nothing monorepo-specific about it, and code-mode is a lower layer that should not depend on the workspaces module. Routes both code-mode write paths through it, so every file OpenWiki writes into an untrusted target repo now refuses a symlinked destination (lstat) or a parent that escapes the repo root (realpath). Moves the guard's tests to test/safe-write.test.ts and adds a workflow-file (repo-root) attack case. Co-Authored-By: Claude <noreply@anthropic.com>
Collaborator
|
Nice! Excited for this. |
Reframe the "no dependency cascade" note from a limitation-awaiting-a-fix into a by-design boundary. A subproject run is isolated to its own subtree (filesystem rooted at the subproject dir; git evidence scoped with `-- .`), so it cannot read a dependency's source. Re-running a dependent after an unrelated dependency change would have no new information to act on. Public API changes already surface in the dependent's own subtree diff and regenerate normally; only dependency internals (invisible to dependents) are skipped, which is the desired behavior. Closes #2. Co-Authored-By: Claude <noreply@anthropic.com>
…repo-docs # Conflicts: # src/agent/index.ts # src/cli.tsx # src/code-mode.ts # test/code-mode.test.ts
…to feat/recursive-monorepo-docs
…repo-docs # Conflicts: # src/agent/index.ts # src/cli.tsx # src/code-mode.ts
Contributor
Author
|
Colin Francis (@colifran) happy to take any feedback or make adjustments to approach based on consensus/discussion if you've had any. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a recursive documentation strategy so large monorepos can be documented as a tree of wikis instead of one flat wiki. Each subproject gets its own nested
<subproject>/openwiki/sub-wiki documenting its subtree in depth, while the repository-root wiki links down to them and covers only cross-cutting concerns.Closes the request in #162.
Motivation
For a large monorepo, a single wiki either drowns in detail or stays too shallow to be useful. #162 asked for OpenWiki to "run recursively so it supports monorepos/large systems that can't otherwise be documented in a single wiki." This implements exactly that: granular per-subproject docs, with the root wiki as a map that references them.
How it works
openwiki/workspaces.jsonmanifest at the repo root, or — with--recursiveand no manifest — from auto-detection of common workspace layouts (pnpm/npm/yarn workspaces, Cargo, Gogo.work, uv, Gradle, Maven, .NET.sln/.slnx, Bazel). Auto-detection writes a manifest for review, then proceeds. Manifest presence auto-enables recursion;--recursive=falseforces a single-repo run.workspaceslist (detection-owned, path-only, regenerated) plus a hand-authoredoverridesmap keyed by path (goal/name/exclude/include), so customization and auto-discovery never collide. Manual groupings detection can't surface are preserved as long as their directory exists..last-update.jsonmetadata, plan cleanup, and index-sync all scope to that subproject with no special-casing — then writes a generatedopenwiki/workspaces.mdaggregation index, then the root run last (so the root's index-sync links the aggregation in). One model is resolved and reused across all runs; runs are sequential.--update, a subproject regenerates only when files in its own subtree changed since it was last documented (its ownopenwiki/.last-update.json), skipping unchanged subprojects without a model call. The root wiki regenerates every run..sln/.slnxdetection coarsens per-project paths to product-area roots (area/src/X,area/tests/X→area) to avoid one wiki per.csproj, guarded so it never collapses to a whole-tree wiki.Backward compatibility
With no manifest and no
--recursiveflag, behavior is unchanged — the single-repo path is byte-for-byte identical. Legacy flat manifests are read and migrated in-memory, preserving hand-authored goals.No dependency cascade (by design)
Updates do not cascade across subprojects: a change to a shared subproject (e.g. a common kernel) refreshes only that sub-wiki and the root, not the sibling subprojects that depend on it. This is a deliberate design boundary, not a missing feature.
A subproject run is isolated to its own subtree — the filesystem tools are rooted at the subproject directory and its git evidence is scoped with a
-- .pathspec — so a run cannot read a dependency's source. Re-running a dependent after an unrelated dependency change would have no new information to incorporate and would just reproduce the same sub-wiki. And when a dependency's public API changes, the dependent's own call sites change with it, which already shows up in the dependent's own subtree diff and regenerates it normally; only changes to a dependency's internals (invisible to dependents) are skipped, which is the desired behavior.This supersedes the earlier "dependency-aware invalidation" follow-up (ppsplus-bradh#2), which was closed as by-design once the isolation model made clear that a cascade would only trigger runs that can't act on the trigger. The README's "no dependency cascade" note explains the same rationale.
Testing