feat: native OpenClaw gateway plugin (in-process analysis pipeline + dashboard route) - #569
feat: native OpenClaw gateway plugin (in-process analysis pipeline + dashboard route)#569zeroaltitude wants to merge 9 commits into
Conversation
…pipeline, agent tools, dashboard route Adds .openclaw-plugin/, a real OpenClaw gateway plugin (beyond the existing install.sh skill-symlink integration). The gateway process itself gains: - Agent tools any connected agent can call mid-conversation: understand_list_projects / understand_analyze_project (background job) / understand_status / understand_search / understand_get_node - HTTP routes: /understand-anything project picker that spawns and redirects to the token-protected understand-anything-viewer per project - A self-contained analysis pipeline built entirely on @understand-anything/core's public API (createIgnoreFilter, LanguageRegistry, TreeSitterPlugin, buildFileAnalysisPrompt / parseFileAnalysisResponse, GraphBuilder, validateGraph, saveGraph) with single-turn Anthropic Messages API calls for the LLM enrichment phase — no Claude Code or Task-tool dispatch involved. Output is byte-compatible .ua/knowledge-graph.json. The import-edge resolver handles the TS-ESM convention where "./util.js" refers to util.ts on disk (covered by a regression test; without this every intra-project import edge in a typical TS codebase is silently dropped). Analysis is restricted to a configured project allowlist; concurrency and file-count caps bound LLM cost; the API key comes from plugin config or the gateway process env only. .openclaw-plugin joins the pnpm workspace so it can depend on @understand-anything/core via workspace:*.
…agent tools Current OpenClaw refuses to register plugin agent tools unless the manifest declares them up front in contracts.tools (registry.ts:617 "plugin must declare contracts.tools before registering agent tools"). Verified via `openclaw plugins inspect understand-anything --runtime` against an isolated state dir: before this, activate() hit 5 error diagnostics and registered 0 tools; after, all 5 tools + both dashboard routes register with 0 errors.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 305bc0af0b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| let stat; | ||
| try { | ||
| stat = statSync(abs); |
There was a problem hiding this comment.
When a configured project contains a symlinked directory, statSync follows it and the recursion below can walk outside the configured project allowlist (for example a link to /) or repeatedly revisit the same tree until ELOOP; those files can then be sent to the LLM despite not being under an allowed project. Use lstat/Dirent and skip symlinked directories before recursing.
Useful? React with 👍 / 👎.
|
|
||
| log("Walking project tree..."); | ||
| const allRelFiles = walkProject(projectRoot, ignoreFilter); | ||
| const codeFiles = allRelFiles.filter((f) => languageRegistry.getForFile(f) !== null); |
There was a problem hiding this comment.
After the first successful run, walkProject will return .ua/knowledge-graph.json and .ua/meta.json because the default ignore filter does not exclude the UA data dirs and JSON is a registered language. On subsequent runs those generated artifacts enter codeFiles, get LLM analysis, add self-referential nodes, and can count against maxFiles; exclude .ua/ and .understand-anything/ before language filtering.
Useful? React with 👍 / 👎.
| } | ||
|
|
||
| export function matchesAnyPattern(relPath: string, patterns: string[]): boolean { | ||
| return patterns.some((p) => globToRegExp(p).test(relPath)); |
There was a problem hiding this comment.
The project-summary prompt asks the model for "glob patterns or path prefixes", but a prefix such as src/api is converted to ^src/api$ here and therefore does not match files like src/api/users.ts. In that common response shape, the layer's nodeIds stay empty even though the files belong to the layer; treat non-glob patterns as directory prefixes or normalize them to /**.
Useful? React with 👍 / 👎.
An independent GPT-5.6-Sol review of the plugin (report in ~/reports/understand-anything-plugin-review-2026-07-12.md, not part of this repo) found 4 must-fix and 4 should-fix issues, spot-checked and confirmed against the source before addressing: Must-fix: - walk.ts: walkProject followed symlinks with no containment check or cycle detection. A symlink under an allowed project root (not covered by the default ignore patterns) could read files outside the projects allowlist — the plugin's entire security boundary — and a symlink cycle could recurse forever and crash the whole gateway process (this runs in-process, not in a subprocess). Fixed via realpathSync containment checks plus a visited-dirs set seeded with the root itself (a naive per-child-only visited set still let a *self*-referential top-level symlink walk root's contents one extra time before the guard caught it). Added src/__tests__/walk.test.ts covering escape, deep cycle, and self-cycle cases with real filesystem symlinks. - dashboard-route.ts: getOrStartViewer only wrote to the `viewers` map once spawn() produced its dashboard URL (up to 15s later), so two concurrent requests for the same project's dashboard both saw a cache miss and both spawned a viewer — the loser leaked as an unreferenced, never-killed process. Fixed by storing the in-flight *promise* in the map synchronously before spawn() returns. Verified live (not just reasoned about): fired two concurrent requests against the built module and confirmed both resolved to the identical viewer instance (same port, same token). - dashboard-route.ts / index.ts: no lifecycle management at all for spawned understand-anything-viewer processes — no shutdown hook, no idle eviction, no cap. Every project dashboard opened once left a Node child process bound to a port for the life of the gateway, and a plugin/gateway reload orphaned all of them (module state resets, PIDs become unreachable). Added shutdownAllViewers() wired to the gateway_stop hook, plus a 30-minute idle-eviction sweep (unref'd so it can't itself keep the process alive). - llm.ts: no timeout on the Anthropic HTTPS call. A single stalled TCP connection never settled, permanently consuming a mapWithConcurrency worker slot and wedging that analysis job in "running" state forever — the only recovery was a full gateway restart, since understand_analyze_project refuses to start a new job while one is "running". Added a 60s request timeout via the `timeout` socket option + a `timeout` event handler that destroys the request and rejects cleanly. Should-fix: - index.ts: getGraph's loadGraph/JSON.parse path had no try/catch. saveGraph (in @understand-anything/core) does a plain writeFileSync with no temp-file+rename, and the gateway now serves reads continuously while a background analyzeProject() job can write at any moment — a read landing mid-write threw an uncaught exception. Now falls back to the previously cached graph on parse failure instead of propagating. - llm.ts: resolveAnthropicApiKey defaulted to the "main" agent's auth-profile path when OPENCLAW_AGENT_DIR wasn't set. A gateway hosting multiple agents with separate homes could silently read the wrong agent's key/account. The auth-profile fallback now only applies when the host has actually told us which agent this is. - index.ts: nodeBrief did `n.tags.length` unconditionally despite graphs being loaded with `validate: false` specifically to tolerate off-schema data. Guarded with an Array.isArray check. - dashboard-route.ts: /understand-anything/open with no `project` query param at all passed validation (Number(null) === 0) and silently opened project index 0. Now requires the param to be present. Nice-to-have also applied: pipeline.ts called builder.build() twice (once to preview layer node IDs, once for the persisted graph) — build() is a pure, deterministic snapshot, so this was a needless double array-copy on potentially thousands of nodes/edges. Now built once and mutated in place. Full pipeline re-verified end-to-end after these changes (real Anthropic key, real project, matching node/edge counts, 0 warnings, no regression). 17/17 unit tests pass (13 existing + 4 new).
|
Ran an independent adversarial code review of this PR (GPT-5.6-Sol) and pushed a follow-up commit addressing everything at must-fix/should-fix severity: Must-fix
Should-fix: non-atomic graph writes could throw on concurrent reads (now falls back to cached graph), the API-key fallback no longer silently defaults to reading the Nice-to-have: removed a redundant duplicate Full pipeline re-verified end-to-end after these changes (real Anthropic key, real project) with no regression — matching node/edge counts, 0 warnings. 17/17 unit tests pass (13 existing + 4 new). |
Adds a floating "Ask" chat widget to the dashboard, backed by a new POST /ask.json endpoint — the missing piece between "generate a graph" and "actually talk to it," without leaving the browser for a CLI/chat session. Architecture, deliberately not a fork of understand-anything-viewer: that package's whole reason to exist is staying LLM-free for team-sharing without an API key, so it's never modified. Instead: - src/interactive-server.ts is a new, separate server (spawned as its own subprocess, same pattern as the existing viewer proxy) that reuses the viewer's static dashboard build and JSON API routes read-only, injects a small vanilla-JS widget script into index.html, and adds the /ask.json endpoint. Same token/security model as the existing endpoints throughout. - src/ask.ts is the actual Q&A logic: SearchEngine finds graph nodes relevant to the question, a bounded number of their source files are read for grounding, and the same llm.ts caller used for analysis answers using only that context. Mirrors upstream's /understand-chat skill, just reachable from the browser. - src/ask-widget.js is the floating chat UI: vanilla JS/CSS, no build step, so it's independent of the dashboard's React app and build pipeline. - dashboard-route.ts now branches on whether an API key is resolvable (same resolution order the analysis pipeline already uses): with a key, serves the interactive server; without one, falls back to the plain viewer exactly as before — verified both paths live, including that the plain-viewer fallback still returns the correct redirect with no widget injected. Verified end-to-end with a real Anthropic key against a real analyzed project: the widget serves, /ask.json answers a real question accurately (correctly identified and cited the actual ingestion code paths across multiple files), and the existing graph/file-content API is unaffected. 3 new unit tests for ask.ts's grounding logic (no-graph case, node-matching, snippet inclusion) — 20/20 total tests pass.
|
Added the interactive Ask chat panel discussed as a natural extension of this plugin — the dashboard can now be talked to directly instead of needing a CLI/chat session for Q&A. What's new:
Verification:
|
…board Two picker-page additions: 1. Unanalyzed projects get an "Understand this project" button (POST /understand-anything/analyze?project=<idx>) instead of a plain "call the tool" note. Shares one job-tracking path with the understand_analyze_project tool via a new startAnalysis() function extracted from the tool body — not a second, divergent implementation. The picker auto-refreshes (plain <meta refresh>, no JS) while any job is running, and shows a "Retry analysis" button if one failed. 2. An "Add a project" form (gated behind a new allowAddProject config flag, default false) lets the dashboard register a new project at runtime by GitHub URL or local path, then immediately starts analyzing it. src/project-store.ts is the new ProjectStore: combines config-declared projects (fixed) with dynamically-added ones (persisted to dynamic-projects.json under ~/.local/share/understand-anything-plugin, so they survive a restart) into one index-stable list — new entries only ever append, so existing tool/dashboard references by index never break. GitHub URL handling only recognizes an actual https://github.com/... or git@github.com:... URL — deliberately NOT a bare "owner/repo" shorthand (ambiguous with a genuine relative local path) or any non-GitHub git/ file:// URL (which could otherwise smuggle an arbitrary local path or host into a "clone", defeating the projects allowlist this feature sits inside). Clones shallow (--depth 1) with a 3-minute timeout via a real array-args spawn() (no shell, no injection risk). allowAddProject defaults to false and is documented as a real security tradeoff in the README: the projects allowlist is otherwise the only way to expand what this plugin can read/analyze/serve, and this setting relaxes that for anyone who can reach the dashboard route. Verified end-to-end with a real live GitHub clone (not just unit tests): activated the full plugin, clicked "Understand this project" on a fresh unanalyzed project (job started, status tool confirmed "running"), then added octocat/Hello-World by its real GitHub URL — genuinely cloned to ~/.local/share/understand-anything-plugin/clones/octocat-Hello-World, appeared in both the picker and understand_list_projects at the correct index, and auto-started analysis. Also verified the allowAddProject: false default correctly hides the form and rejects the POST endpoint. 6 new unit tests for ProjectStore (persistence round-trip, duplicate handling, non-existent-path rejection, GitHub shorthand/non-GitHub-URL rejection) — 26/26 total tests pass.
|
Two more picker-page additions: 1. Click to analyze. Unanalyzed projects get an "Understand this project" button right on the picker page instead of a "call the tool" note — shares one job-tracking path with the 2. Add a project from the dashboard, gated behind a new
Verified end-to-end with a real live GitHub clone, not just unit tests: activated the full plugin, clicked "Understand this project" on a fresh project (job started, confirmed via 6 new tests for |
Three additions, plus a real bug fix uncovered along the way: 1. Re-analyze button: already-analyzed projects on the picker page now get a "Re-analyze" button (analysis was always a full fresh re-run, just no UI trigger existed for the already-analyzed case) — this is how a project's tours/summaries reset to whatever the pipeline currently produces as it improves. 2. Auto-generated tours at analysis time (src/tour-generation.ts, src/tour-store.ts): a free, deterministic module-walkthrough tour (wraps core's generateHeuristicTour, zero LLM cost) is synced into the standard graph.tour field — upstream's schema only supports one tour, so this keeps the existing Learn persona/LearnPanel working with zero changes. A second, LLM-narrated code-review tour ranks nodes by complexity + how central they are in the dependency graph (fan-in/fan-out), then asks the model to explain *why* a reviewer should care about each, not just what the code does. Both persist to a plugin-owned .ua/tours.json sidecar (upstream's schema has no room for multiple named tours). Lifecycle/request-flow tours are explicitly deferred — that needs call-graph/endpoint traversal work, a separate research task. 3. Custom tour generation (src/custom-tour.ts) + a new Tours widget (src/tours-widget.js, same vanilla-JS/no-build-step pattern as the Ask widget): select node(s) in the live React Flow canvas — read via a MutationObserver on .react-flow__node.selected, zero patches to the dashboard's React source — type a prompt, and get back a tour scoped to just those nodes and that request. New endpoints on interactive-server.ts: GET /tours.json (list) and POST /generate-tour.json (generate + persist), same per-instance token model as the existing endpoints. Bug fix found while testing tour generation live: llm.ts's createLlmCaller assumed content[0] was always the text block, but extended thinking (on by default for some models/accounts) puts a "thinking" block first, with the actual response in a later block — every LLM call in the plugin (analysis, Ask, tours) shared this latent bug, it just hadn't been triggered before now. Fixed to find the first block with type "text" instead of blindly indexing [0]. Added 4 regression tests mocking node:https (common case, thinking-first case, no-text-block case, API-error case). Verified end-to-end with real live Anthropic calls, not just unit tests: analyzed a real project (module + code-review tours both generated correctly, zero warnings after the llm.ts fix), started the interactive server, confirmed both widget scripts inject, hit GET /tours.json and saw both auto-tours, then POST /generate-tour.json with real node ids and a real prompt — got back an accurate 5-step tour correctly tracing the actual CLI entry point through to post-ingestion cleanup, persisted alongside the other two tours. Confirmed all validation/security paths (empty nodeIds, empty prompt, missing token → 403). 14 new tests (custom-tour, tour-generation, tour-store, llm regression) — 44/44 total tests pass.
|
Three more additions, plus a real bug fix uncovered while testing: 1. Re-analyze button on already-analyzed projects (analysis was always a full fresh re-run — no UI trigger existed for that case). 2. Auto-generated tours at analysis time. A free, deterministic module walkthrough (wraps core's 3. Custom tour generation + a new Tours widget (same vanilla-JS pattern as the Ask widget): select node(s) in the live graph canvas — read via a Bug fix found along the way: Verified end-to-end with real live Anthropic calls: analyzed a real project (both tours generated correctly, zero warnings post-fix), started the interactive server, confirmed both widgets inject, hit the tours API, then generated a custom tour from real node ids + a real prompt — got back an accurate 5-step walkthrough correctly tracing the actual code path. All validation/security paths confirmed. 14 new tests — 44/44 total pass. |
Formalizes an on-disk contract that already existed informally: the understand-diff skill has always read/written <ua-dir>/diff-overlay.json in this exact shape (SKILL.md step 8), and the dashboard's store/GraphView already consume changedNodeIds/affectedNodeIds for the diff-mode overlay UI — but core had no first-class type or persistence pair for it, unlike the sibling domain-graph.json (saveDomainGraph/loadDomainGraph). - DiffOverlay type in types.ts. - saveDiffOverlay/loadDiffOverlay in persistence/index.ts, mirroring the domain-graph save/load pattern exactly (same directory resolution, same file-path sanitization so absolute paths never leak to disk — extracted the per-path sanitization logic out of sanitiseFilePaths into a reusable sanitiseFilePath so both the node-array and plain-string-array shapes can share it, no behavior change to the existing function). - computeDiffOverlay(graph, changedFiles, baseBranch?) in the new diff-overlay.ts: pure, deterministic graph traversal — maps changed file paths to their file/function/class nodes (graph-builder stamps the same filePath on all three), then walks one hop of edges in both directions to find the blast radius, excluding the changed nodes themselves. No LLM involved, mirrors the skill's own documented algorithm (grep node IDs in edges, 1-hop) exactly. 11 new tests (5 persistence round-trip/sanitization, 6 traversal logic). Full suite: 942/942 tests pass (931 existing + 11 new).
…ntrol) Resolves changed files via `gh pr diff` or `git diff <base>...HEAD` (falling back to uncommitted working-tree changes), computes the blast radius against the persisted graph with core's computeDiffOverlay (pure, no LLM), then asks the model to narrate a tour grounded only in the changed + affected nodes. Exposed as both the understand_analyze_pr agent tool and a POST /generate-pr-tour.json endpoint, with a matching input + button in the Tours widget. Node lists sent to the model are capped (40 changed / 20 affected) so the required output stays bounded on large diffs — confirmed via live E2E testing against a real branch diff that an uncapped prompt plus the previous 2000 max_tokens ceiling produced a mid-JSON truncation (stop_reason: max_tokens). Raised the budget to 8000 and added the caps together, then re-verified a 7-step walkthrough generates cleanly against the same real diff. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Fourth addition: PR/diff walkthroughs. Answers "can this framework understand a change, not just a static codebase?" — yes: This one touched Bug found + fixed via live E2E testing, not just unit tests: the LLM call's 5 new tests (real git-repo integration tests for changed-file resolution + stub-LLM grounding tests) — 49/49 total pass. |
…ry script
Ask (src/ask.ts) previously had zero notion of what the user is currently
looking at — pure free-text SearchEngine relevance. Custom-tour generation
already scoped to an explicit node selection (read via a private
MutationObserver on .react-flow__node.selected in tours-widget.js), so a
question asked while a specific file was selected in the canvas could ignore
that file entirely if the question's wording didn't happen to match it.
Extracted the selection-reading logic into src/selection.js, a single shared
discovery script (window.uaSelection.get()/.subscribe()) loaded before both
widgets, so there's exactly one MutationObserver instead of two independently
watching the same DOM. Both ask-widget.js and tours-widget.js now read from
it instead of keeping private copies.
askAboutProject gained an optional selectedNodeIds param: selected nodes are
always included as primary grounding context (deduped against SearchEngine's
matches), ahead of and independent from whatever the question text happens to
match. The Ask panel also shows a "Grounded in: ..." bar so it's visible
whether a selection was picked up.
Verified live against a real analyzed project: the same vague question ("what
does this file do?") answered about whatever file ranked highest generically
without a selection, and correctly answered about the selected file
(src/db.ts) once one was set — despite an identical question in both cases.
3 new tests (selection grounding, selected+matched dedup, unknown node id
ignored) — 52/52 total pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Fifth addition: Ask + Tours now ground on the user's live graph selection, through one shared mechanism. Prompted by watching it fail in practice: selected a file via the dashboard's Files breadcrumb, asked the Ask panel a question, and the answer never referenced the selected file at all — because
Verified live, not just unit-tested: asked the identical vague question ("what does this file do?") against a real analyzed project twice — once with nothing selected (answered about whatever ranked highest generically) and once with 3 new tests — 52/52 total pass. |
1 participant
Summary
Adds
.openclaw-plugin/— a real OpenClaw gateway plugin that runs the Understand-Anything knowledge-graph pipeline inside the gateway process itself, as a deeper alternative to the existinginstall.sh openclawskill-symlink integration (which is untouched and keeps working).understand_list_projects,understand_analyze_project(background job: tree-sitter structural pass + one LLM call per file, persisted to.ua/knowledge-graph.json),understand_status,understand_search(via@understand-anything/core'sSearchEngine),understand_get_node./understand-anything(project picker) and/understand-anything/open?project=<idx>, which starts (or reuses) anunderstand-anything-viewerinstance per project and redirects into its token-protected, 127.0.0.1-only dashboard — reusing the existing viewer's security model as-is.src/pipeline.ts) is built entirely on@understand-anything/core's already-public API —createIgnoreFilter/LanguageRegistryfor the walk,TreeSitterPlugin.analyzeFilefor structure,buildFileAnalysisPrompt/parseFileAnalysisResponse+buildProjectSummaryPrompt/parseProjectSummaryResponsefor LLM enrichment,GraphBuilder→validateGraph→saveGraph/saveMetafor persistence. No new parsing/prompt logic — just new orchestration and a new host..ua/knowledge-graph.jsonshape the skills produce, same dashboard renders it.projectsallowlist (required, no default). Anthropic API key comes from plugin config or the gateway process env only — never written to disk by the plugin..openclaw-pluginjoins the pnpm workspace (workspace:*on@understand-anything/coreandunderstand-anything-viewer) so it builds against the existing packages with no duplication.Why
OpenClaw's current support here is a skill symlink (
install.sh openclaw→~/.openclaw/skills/), same mechanism as every other CLI platform. That's great for the Claude-Code-style slash-command flow, but it means the analysis pipeline only runs when a Task-tool-capable coding agent drives it step-by-step through the 848-line/understandskill. This plugin instead runs the pipeline as ordinary backend code attached to the gateway's own lifecycle — any agent connected to that gateway getsunderstand_*tools mid-conversation and a persistent dashboard route, with no CLI-specific dispatch required.Test plan
pnpm --filter @understand-anything/openclaw-plugin build(tsc) — cleanpnpm --filter @understand-anything/openclaw-plugin test— 13/13 unit tests pass (import-resolution + glob-matching)./util.js→util.tscase), 0 warnings.ua/knowledge-graph.jsonvalidates against the existing schema; file paths correctly relativized (no absolute-path leaks)understand-anything-viewerserves the resulting graph: dashboard loads, graph JSON accessible with token,403withoutopenclaw plugins inspect <id> --runtime) in an isolated config: plugin statusloaded, all 5 tools + 2 HTTP routes registered, 0 error diagnosticsopenclaw-pluginsmonorepo through an actual gateway + chat session — analysis completed successfully end-to-end (52/52 files) via a real tool call from an agent conversationNotes for reviewers
main; an earlier iteration of the branch had unrelated repo-tooling commits (a localbdissue-tracker init) mixed in — those are intentionally excluded from this PR.