diff --git a/docs/integrations/pi.md b/docs/integrations/pi.md new file mode 100644 index 0000000..c4078f9 --- /dev/null +++ b/docs/integrations/pi.md @@ -0,0 +1,43 @@ +# PI integration (example) + +This project ships a small HTTP server that exposes engram's query, learn, +and context streaming endpoints. To integrate with external tooling (like +`pi`), you can call the server from your agent process to run the same +hook handlers engram uses internally. + +Quick example (requires Node 20+ and the engram HTTP server running): + +1. Start the engram HTTP server for your project: + + - From a built installation: `engramx serve --port 7337 --project /path/to/project` + - Or run the server via the CLI: `node dist/cli.js serve --port 7337 --project /path/to/project` + + The server writes an auth token to `~/.engram/http-server.token` on first start. + +2. Run the example PI client that demonstrates the flow: + + ```bash + node examples/pi-client.js /path/to/project + ``` + + The script will: + - Call `POST /hook` with a `SessionStart` payload (project brief + provider warmup) + - Call `POST /hook` with a `UserPromptSubmit` payload (pre-query injection) + - POST a session summary to `/learn` so engram can persist session learnings + +3. Production integration suggestions: + + - Call `POST /hook` with `SessionStart` at the beginning of a user session + to warm provider caches and inject both project and global memory. + - During request processing, call `POST /hook` for `UserPromptSubmit` and + `PreToolUse` events (or use `GET /context/stream` and `/query`) to + resolve context on-demand. + - When the request completes and you have a session summary, call `POST /learn` + with the summary text so engram persists the learning in its graph. + +Security: all HTTP endpoints require an auth token (read from +`~/.engram/http-server.token` or supplied via `Authorization: Bearer `). +This keeps the server local-only and fail-closed. + +If you want, I can add a small PI-side wrapper that calls these endpoints +from the pi harness directly (example: a `pi` skill or integration). \ No newline at end of file diff --git a/docs/operations/dev-reload-port-audit.md b/docs/operations/dev-reload-port-audit.md new file mode 100644 index 0000000..23319ea --- /dev/null +++ b/docs/operations/dev-reload-port-audit.md @@ -0,0 +1,66 @@ +Dev-reload: port wait and startup audit + +Summary + +This document records the recent changes made to reduce transient EADDRINUSE errors during development reloads and to add a small startup audit line when the HTTP server writes its PID file. + +What was changed + +1) scripts/dev-reload.sh + +- After building the project, the script now waits (best-effort) for the configured PORT to be free before starting the server. This reduces races where nodemon spawns a new process while the old one is still closing and listening on the port. +- Behavior is configurable via environment variables: + - WAIT_RETRIES (default: 16) + - SLEEP_INTERVAL (default: 0.5 seconds) +- If the port remains busy after the retry loop the script proceeds anyway (non-blocking fallback) to avoid hanging CI or developer shells. + +2) src/server/http.ts + +- The writePid(projectRoot) helper still writes .engram/http-server.pid but now also appends an audit line with timestamp, pid, port, and project to two locations: + - /.engram/http-server.start.log (project-scoped audit log) + - ~/.engram/http-server.log (user-level co-log) +- Audit line format: START pid= port= project= +- Audit writes are best-effort; failures are swallowed so they don't block server startup. + +Why + +- Frequent EADDRINUSE errors were observed during rapid dev reloads. These are usually harmless (old process still closing) but they spam stderr and can hide real issues. Waiting briefly for the port to be released reduces noise and stabilizes local dev workflow. +- The PID file used by the HUD / component-status can sometimes be missing during restart windows. The audit log makes it trivial to confirm when the server wrote the PID (and which pid/port it recorded), which aids debugging. + +Files modified + +- scripts/dev-reload.sh +- src/server/http.ts + +How to test locally + +1) Force a dev rebuild (touch a source file and allow nodemon/dev watcher to run dev-reload) +2) Watch the script output: you should see lines like: + [dev-reload] waiting for port 7337 to be free (still: ) — attempt X/Y + [dev-reload] starting server via: node dist/cli.js ui ... +3) After server starts, verify: + - cat /.engram/http-server.pid → should contain the pid + - tail -n 50 /.engram/http-server.start.log → last line is the START audit line + - tail -n 50 ~/.engram/http-server.log → contains the START audit line +4) Exercise the dashboard/UI or POST a SessionStart hook to confirm the updated server responds with the injected project brief. + +Environment knobs + +- WAIT_RETRIES= number of attempts waiting for port to be free (default 16) +- SLEEP_INTERVAL= seconds to sleep between attempts (default 0.5) + +Todo / Follow-ups (issues created) + +- [ ] Add a small integration test that verifies the audit line is written on server startup (requires test harness that can spawn the server). +- [ ] Consider making the dev-reload script fail-hard if the port remains busy (configurable via env var) to avoid accidental double-runs in CI. +- [ ] Rotate/trim the audit logs (project-level and user-level) to avoid unbounded disk growth. +- [ ] Improve portability for environments without lsof (Windows) — consider a Node-based port probe helper or use netstat variations. +- [ ] Expose the startup audit via component-status so the UI can show "Last startup at (pid)". + +Link to changes + +- Branch: feat/sessionstart-resume-auto-memory +- Commits: + - dev: wait for port free in dev-reload; http: append startup audit line when writing pid + +If you'd like any of the follow-ups implemented now (tests, log rotation, stricter dev-reload behavior), tell me which ones and I'll implement them in follow-up commits/PRs. diff --git a/examples/pi-client.js b/examples/pi-client.js new file mode 100644 index 0000000..f982550 --- /dev/null +++ b/examples/pi-client.js @@ -0,0 +1,72 @@ +#!/usr/bin/env node +/** + * Example PI client that demonstrates calling engram's /hook and /learn endpoints. + * + * Usage: node examples/pi-client.js /path/to/project + */ + +import { readFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; + +const PORT = process.env.ENGRAM_HTTP_PORT ?? 7337; +const TOKEN_PATH = join(homedir(), ".engram", "http-server.token"); + +function readToken() { + try { + const t = readFileSync(TOKEN_PATH, "utf-8").trim(); + return t; + } catch (err) { + console.error("Failed to read token at", TOKEN_PATH, err?.message ?? err); + process.exit(1); + } +} + +async function postHook(payload) { + const resp = await fetch(`http://127.0.0.1:${PORT}/hook`, { + method: "POST", + headers: { + Authorization: `Bearer ${readToken()}`, + "Content-Type": "application/json", + }, + body: JSON.stringify(payload), + }); + if (resp.status === 204) return null; + return await resp.json(); +} + +async function postLearn(content, file) { + const resp = await fetch(`http://127.0.0.1:${PORT}/learn`, { + method: "POST", + headers: { + Authorization: `Bearer ${readToken()}`, + "Content-Type": "application/json", + }, + body: JSON.stringify({ content, file }), + }); + if (resp.status >= 400) { + const txt = await resp.text(); + throw new Error(`learn failed: ${txt}`); + } + return await resp.json(); +} + +const projectRootArg = process.argv[2] ?? process.cwd(); + +(async () => { + console.log("Project root:", projectRootArg); + console.log("Calling SessionStart..."); + const session = await postHook({ hook_event_name: "SessionStart", cwd: projectRootArg, source: "startup" }); + console.log("SessionStart result:", session); + + console.log("Calling UserPromptSubmit..."); + const prompt = await postHook({ hook_event_name: "UserPromptSubmit", cwd: projectRootArg, prompt: "How does authentication work?" }); + console.log("UserPromptSubmit result:", prompt); + + console.log("Posting a summary to /learn..."); + const learn = await postLearn("Session summary: example decision to prefer JWT tokens", "pi-example"); + console.log("Learn result:", learn); +})().catch((err) => { + console.error("Error", err); + process.exit(1); +}); diff --git a/package-lock.json b/package-lock.json index 31abca3..1d1e2a7 100644 --- a/package-lock.json +++ b/package-lock.json @@ -7,6 +7,7 @@ "": { "name": "engramx", "version": "3.4.0", + "hasInstallScript": true, "license": "Apache-2.0", "dependencies": { "@modelcontextprotocol/sdk": "^1.29.0", @@ -27,6 +28,7 @@ }, "devDependencies": { "@types/node": "^25.5.2", + "nodemon": "^3.0.2", "tsup": "^8.5.1", "typescript": "^6.0.2", "vitest": "^4.1.4" @@ -1534,6 +1536,33 @@ "dev": true, "license": "MIT" }, + "node_modules/anymatch": { + "version": "3.1.3", + "resolved": "https://registry.npmjs.org/anymatch/-/anymatch-3.1.3.tgz", + "integrity": "sha512-KMReFUr0B4t+D+OBkjR3KYqvocp2XaSzO55UcB6mgQMd3KbcE+mWTyvVV7D/zsdEbNnV6acZUutkiHQXvTr1Rw==", + "dev": true, + "license": "ISC", + "dependencies": { + "normalize-path": "^3.0.0", + "picomatch": "^2.0.4" + }, + "engines": { + "node": ">= 8" + } + }, + "node_modules/anymatch/node_modules/picomatch": { + "version": "2.3.2", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.2.tgz", + "integrity": "sha512-V7+vQEJ06Z+c5tSye8S+nHUfI51xoXIXjHQ99cQtKUkQqqO1kO/KCJUfZXuB47h/YBlDhah2H3hdUGXn8ie0oA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8.6" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, "node_modules/assertion-error": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/assertion-error/-/assertion-error-2.0.1.tgz", @@ -1544,6 +1573,29 @@ "node": ">=12" } }, + "node_modules/balanced-match": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz", + "integrity": "sha512-BLrgEcRTwX2o6gGxGOCNyMvGSp35YofuYzw9h1IMTRmKqttAZZVU67bdb9Pr2vUHA8+j3i2tJfjO6C6+4myGTA==", + "dev": true, + "license": "MIT", + "engines": { + "node": "18 || 20 || >=22" + } + }, + "node_modules/binary-extensions": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/binary-extensions/-/binary-extensions-2.3.0.tgz", + "integrity": "sha512-Ceh+7ox5qe7LJuLHoY0feh3pHuUDHAcRUeyL2VYghZwfpkNIy/+8Ocg0a3UuSoYzavmylwuLWQOf3hl0jjMMIw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/body-parser": { "version": "2.2.2", "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-2.2.2.tgz", @@ -1568,6 +1620,32 @@ "url": "https://opencollective.com/express" } }, + "node_modules/brace-expansion": { + "version": "5.0.5", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.5.tgz", + "integrity": "sha512-VZznLgtwhn+Mact9tfiwx64fA9erHH/MCXEUfB/0bX/6Fz6ny5EGTXYltMocqg4xFAQZtnO3DHWWXi8RiuN7cQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "balanced-match": "^4.0.2" + }, + "engines": { + "node": "18 || 20 || >=22" + } + }, + "node_modules/braces": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/braces/-/braces-3.0.3.tgz", + "integrity": "sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==", + "dev": true, + "license": "MIT", + "dependencies": { + "fill-range": "^7.1.1" + }, + "engines": { + "node": ">=8" + } + }, "node_modules/bundle-require": { "version": "5.1.0", "resolved": "https://registry.npmjs.org/bundle-require/-/bundle-require-5.1.0.tgz", @@ -2075,6 +2153,19 @@ } } }, + "node_modules/fill-range": { + "version": "7.1.1", + "resolved": "https://registry.npmjs.org/fill-range/-/fill-range-7.1.1.tgz", + "integrity": "sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==", + "dev": true, + "license": "MIT", + "dependencies": { + "to-regex-range": "^5.0.1" + }, + "engines": { + "node": ">=8" + } + }, "node_modules/finalhandler": { "version": "2.1.1", "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-2.1.1.tgz", @@ -2187,6 +2278,19 @@ "node": ">= 0.4" } }, + "node_modules/glob-parent": { + "version": "5.1.2", + "resolved": "https://registry.npmjs.org/glob-parent/-/glob-parent-5.1.2.tgz", + "integrity": "sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow==", + "dev": true, + "license": "ISC", + "dependencies": { + "is-glob": "^4.0.1" + }, + "engines": { + "node": ">= 6" + } + }, "node_modules/gopd": { "version": "1.2.0", "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz", @@ -2199,6 +2303,16 @@ "url": "https://github.com/sponsors/ljharb" } }, + "node_modules/has-flag": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/has-flag/-/has-flag-3.0.0.tgz", + "integrity": "sha512-sKJf1+ceQBr4SMkvQnBDNDtf4TXpVhVGateu0t918bl30FnbE2m4vNLX+VWe/dpjlb+HugGYzW7uQXH98HPEYw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=4" + } + }, "node_modules/has-symbols": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz", @@ -2268,6 +2382,13 @@ "url": "https://opencollective.com/express" } }, + "node_modules/ignore-by-default": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/ignore-by-default/-/ignore-by-default-1.0.1.tgz", + "integrity": "sha512-Ius2VYcGNk7T90CppJqcIkS5ooHUZyIQK+ClZfMfMNFEF9VSE73Fq+906u/CWu92x4gzZMWOwfFYckPObzdEbA==", + "dev": true, + "license": "ISC" + }, "node_modules/inherits": { "version": "2.0.4", "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", @@ -2292,6 +2413,52 @@ "node": ">= 0.10" } }, + "node_modules/is-binary-path": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/is-binary-path/-/is-binary-path-2.1.0.tgz", + "integrity": "sha512-ZMERYes6pDydyuGidse7OsHxtbI7WVeUEozgR/g7rd0xUimYNlvZRE/K2MgZTjWy725IfelLeVcEM97mmtRGXw==", + "dev": true, + "license": "MIT", + "dependencies": { + "binary-extensions": "^2.0.0" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/is-extglob": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/is-extglob/-/is-extglob-2.1.1.tgz", + "integrity": "sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/is-glob": { + "version": "4.0.3", + "resolved": "https://registry.npmjs.org/is-glob/-/is-glob-4.0.3.tgz", + "integrity": "sha512-xelSayHH36ZgE7ZWhli7pW34hNbNl8Ojv5KVmkJD4hBdD3th8Tfk9vYasLM+mXWOZhFkgZfxhLSnrwRr4elSSg==", + "dev": true, + "license": "MIT", + "dependencies": { + "is-extglob": "^2.1.1" + }, + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/is-number": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/is-number/-/is-number-7.0.0.tgz", + "integrity": "sha512-41Cifkg6e8TylSpdtTpeLVMqvSBEVzTttHvERD741+pnZ8ANv0004MRL43QKPDlK9cGvNp6NZWZUBlbGXYxxng==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.12.0" + } + }, "node_modules/is-promise": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/is-promise/-/is-promise-4.0.0.tgz", @@ -2703,6 +2870,22 @@ "url": "https://opencollective.com/express" } }, + "node_modules/minimatch": { + "version": "10.2.5", + "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.5.tgz", + "integrity": "sha512-MULkVLfKGYDFYejP07QOurDLLQpcjk7Fw+7jXS2R2czRQzR56yHRveU5NDJEOviH+hETZKSkIk5c+T23GjFUMg==", + "dev": true, + "license": "BlueOak-1.0.0", + "dependencies": { + "brace-expansion": "^5.0.5" + }, + "engines": { + "node": "18 || 20 || >=22" + }, + "funding": { + "url": "https://github.com/sponsors/isaacs" + } + }, "node_modules/mlly": { "version": "1.8.2", "resolved": "https://registry.npmjs.org/mlly/-/mlly-1.8.2.tgz", @@ -2782,6 +2965,96 @@ "node-gyp-build-test": "build-test.js" } }, + "node_modules/nodemon": { + "version": "3.1.14", + "resolved": "https://registry.npmjs.org/nodemon/-/nodemon-3.1.14.tgz", + "integrity": "sha512-jakjZi93UtB3jHMWsXL68FXSAosbLfY0In5gtKq3niLSkrWznrVBzXFNOEMJUfc9+Ke7SHWoAZsiMkNP3vq6Jw==", + "dev": true, + "license": "MIT", + "dependencies": { + "chokidar": "^3.5.2", + "debug": "^4", + "ignore-by-default": "^1.0.1", + "minimatch": "^10.2.1", + "pstree.remy": "^1.1.8", + "semver": "^7.5.3", + "simple-update-notifier": "^2.0.0", + "supports-color": "^5.5.0", + "touch": "^3.1.0", + "undefsafe": "^2.0.5" + }, + "bin": { + "nodemon": "bin/nodemon.js" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/nodemon" + } + }, + "node_modules/nodemon/node_modules/chokidar": { + "version": "3.6.0", + "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-3.6.0.tgz", + "integrity": "sha512-7VT13fmjotKpGipCW9JEQAusEPE+Ei8nl6/g4FBAmIm0GOOLMua9NDDo/DWp0ZAxCr3cPq5ZpBqmPAQgDda2Pw==", + "dev": true, + "license": "MIT", + "dependencies": { + "anymatch": "~3.1.2", + "braces": "~3.0.2", + "glob-parent": "~5.1.2", + "is-binary-path": "~2.1.0", + "is-glob": "~4.0.1", + "normalize-path": "~3.0.0", + "readdirp": "~3.6.0" + }, + "engines": { + "node": ">= 8.10.0" + }, + "funding": { + "url": "https://paulmillr.com/funding/" + }, + "optionalDependencies": { + "fsevents": "~2.3.2" + } + }, + "node_modules/nodemon/node_modules/picomatch": { + "version": "2.3.2", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.2.tgz", + "integrity": "sha512-V7+vQEJ06Z+c5tSye8S+nHUfI51xoXIXjHQ99cQtKUkQqqO1kO/KCJUfZXuB47h/YBlDhah2H3hdUGXn8ie0oA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8.6" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, + "node_modules/nodemon/node_modules/readdirp": { + "version": "3.6.0", + "resolved": "https://registry.npmjs.org/readdirp/-/readdirp-3.6.0.tgz", + "integrity": "sha512-hOS089on8RduqdbhvQ5Z37A0ESjsqz6qnRcffsMU3495FuTdqSm+7bhJ29JvIOsBDEEnan5DPu9t3To9VRlMzA==", + "dev": true, + "license": "MIT", + "dependencies": { + "picomatch": "^2.2.1" + }, + "engines": { + "node": ">=8.10.0" + } + }, + "node_modules/normalize-path": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/normalize-path/-/normalize-path-3.0.0.tgz", + "integrity": "sha512-6eZs5Ls3WtCisHWp9S2GUy8dqkpGi4BVSz3GaqiE6ezub0512ESztXUwUB6C6IKbQkY2Pnb/mD4WYojCRwcwLA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, "node_modules/object-assign": { "version": "4.1.1", "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz", @@ -3006,6 +3279,13 @@ "node": ">= 0.10" } }, + "node_modules/pstree.remy": { + "version": "1.1.8", + "resolved": "https://registry.npmjs.org/pstree.remy/-/pstree.remy-1.1.8.tgz", + "integrity": "sha512-77DZwxQmxKnu3aR542U+X8FypNzbfJ+C5XQDk3uWjWxn6151aIMGthWYRXTqT1E5oJvg+ljaa2OJi+VfvCOQ8w==", + "dev": true, + "license": "MIT" + }, "node_modules/qs": { "version": "6.15.1", "resolved": "https://registry.npmjs.org/qs/-/qs-6.15.1.tgz", @@ -3179,6 +3459,19 @@ "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==", "license": "MIT" }, + "node_modules/semver": { + "version": "7.7.4", + "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.4.tgz", + "integrity": "sha512-vFKC2IEtQnVhpT78h1Yp8wzwrf8CM+MzKMHGJZfBtzhZNycRFnXsHk6E5TxIkkMsgNS7mdX3AGB7x2QM2di4lA==", + "dev": true, + "license": "ISC", + "bin": { + "semver": "bin/semver.js" + }, + "engines": { + "node": ">=10" + } + }, "node_modules/send": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/send/-/send-1.2.1.tgz", @@ -3330,6 +3623,19 @@ "dev": true, "license": "ISC" }, + "node_modules/simple-update-notifier": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/simple-update-notifier/-/simple-update-notifier-2.0.0.tgz", + "integrity": "sha512-a2B9Y0KlNXl9u/vsW6sTIu9vGEpfKu2wRV6l1H3XEas/0gUIzGzBoP/IouTcUQbm9JWZLH3COxyn03TYlFax6w==", + "dev": true, + "license": "MIT", + "dependencies": { + "semver": "^7.5.3" + }, + "engines": { + "node": ">=10" + } + }, "node_modules/source-map": { "version": "0.7.6", "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.7.6.tgz", @@ -3412,6 +3718,19 @@ "node": ">= 6" } }, + "node_modules/supports-color": { + "version": "5.5.0", + "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-5.5.0.tgz", + "integrity": "sha512-QjVjwdXIt408MIiAqCX4oUKsgU2EqAGzs2Ppkm4aQYbjm+ZEWEcW4SfFNTr4uMNZma0ey4f5lgLrkB0aX0QMow==", + "dev": true, + "license": "MIT", + "dependencies": { + "has-flag": "^3.0.0" + }, + "engines": { + "node": ">=4" + } + }, "node_modules/thenify": { "version": "3.3.1", "resolved": "https://registry.npmjs.org/thenify/-/thenify-3.3.1.tgz", @@ -3476,6 +3795,19 @@ "node": ">=14.0.0" } }, + "node_modules/to-regex-range": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/to-regex-range/-/to-regex-range-5.0.1.tgz", + "integrity": "sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "is-number": "^7.0.0" + }, + "engines": { + "node": ">=8.0" + } + }, "node_modules/toidentifier": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz", @@ -3485,6 +3817,16 @@ "node": ">=0.6" } }, + "node_modules/touch": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/touch/-/touch-3.1.1.tgz", + "integrity": "sha512-r0eojU4bI8MnHr8c5bNo7lJDdI2qXlWWJk6a9EAFG7vbhTjElYhBVS3/miuE0uOuoLdb8Mc/rVfsmm6eo5o9GA==", + "dev": true, + "license": "ISC", + "bin": { + "nodetouch": "bin/nodetouch.js" + } + }, "node_modules/tree-kill": { "version": "1.2.2", "resolved": "https://registry.npmjs.org/tree-kill/-/tree-kill-1.2.2.tgz", @@ -3713,6 +4055,13 @@ "dev": true, "license": "MIT" }, + "node_modules/undefsafe": { + "version": "2.0.5", + "resolved": "https://registry.npmjs.org/undefsafe/-/undefsafe-2.0.5.tgz", + "integrity": "sha512-WxONCrssBM8TSPRqN5EmsjVrsv4A8X12J4ArBiiayv3DyyG3ZlIg6yysuuSYdZsVz3TKcTg2fd//Ujd4CHV1iA==", + "dev": true, + "license": "MIT" + }, "node_modules/undici-types": { "version": "7.18.2", "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.18.2.tgz", diff --git a/package.json b/package.json index a29468a..fe119e9 100644 --- a/package.json +++ b/package.json @@ -22,6 +22,7 @@ "build": "tsup && node scripts/bundle-grammars.mjs", "build:nogrammars": "tsup", "dev": "tsup --watch", + "dev:reload": "nodemon --watch src --watch package.json --ext ts,js,json --exec \"bash scripts/dev-reload.sh\"", "test": "vitest", "lint": "tsc --noEmit", "prepublishOnly": "npm run build", @@ -76,6 +77,7 @@ "@types/node": "^25.5.2", "tsup": "^8.5.1", "typescript": "^6.0.2", - "vitest": "^4.1.4" + "vitest": "^4.1.4", + "nodemon": "^3.0.2" } } diff --git a/scripts/dev-reload.sh b/scripts/dev-reload.sh new file mode 100644 index 0000000..8f26ae8 --- /dev/null +++ b/scripts/dev-reload.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +set -euo pipefail + +PORT=${PORT:-7337} +PROJECT_ROOT="$(pwd)" +PIDFILE="$PROJECT_ROOT/.engram/http-server.pid" + +echo "[dev-reload] port=$PORT project=$PROJECT_ROOT" + +# Kill anything listening on the target port. Try a portable lsof invocation. +# Some lsof builds dislike the 127.0.0.1:PORT form — try a couple of variants. +pids=$(lsof -ti "tcp:${PORT}" -sTCP:LISTEN 2>/dev/null || true) +if [ -z "$pids" ]; then + pids=$(lsof -ti "TCP:${PORT}" -sTCP:LISTEN 2>/dev/null || true) +fi +if [ -n "$pids" ]; then + echo "[dev-reload] killing processes listening on 127.0.0.1:${PORT}: $pids" + echo "$pids" | xargs -r kill -9 || true +fi + +# Also remove pidfile if present +if [ -f "$PIDFILE" ]; then + echo "[dev-reload] found pidfile: $PIDFILE" + pidfile_pid=$(cat "$PIDFILE" 2>/dev/null || true) + if [ -n "$pidfile_pid" ]; then + echo "[dev-reload] killing pid from pidfile: $pidfile_pid" + kill -9 "$pidfile_pid" 2>/dev/null || true + fi + rm -f "$PIDFILE" || true +fi + +# Build artifacts +echo "[dev-reload] running npm run build" +npm run build + +# Wait for the target port to be free before starting the server. This +# avoids a race where nodemon spawns a new process while the old one is +# still closing, producing EADDRINUSE errors. We retry a few times with +# short sleeps and proceed even if the port remains busy (best-effort). +WAIT_RETRIES=${WAIT_RETRIES:-16} +SLEEP_INTERVAL=${SLEEP_INTERVAL:-0.5} +for i in $(seq 1 "$WAIT_RETRIES"); do + pids=$(lsof -ti "tcp:${PORT}" -sTCP:LISTEN 2>/dev/null || true) + if [ -z "$pids" ]; then + echo "[dev-reload] port ${PORT} is free" + break + fi + echo "[dev-reload] waiting for port ${PORT} to be free (still: $pids) — attempt $i/$WAIT_RETRIES" + sleep "$SLEEP_INTERVAL" +done + +# Start server via the CLI auto-start entrypoint (ui) so PID/lock semantics are consistent +# Use --no-open to avoid launching a browser from dev script +echo "[dev-reload] starting server via: node dist/cli.js ui --no-open --port ${PORT} -p ${PROJECT_ROOT}" +node dist/cli.js ui --no-open --port "${PORT}" -p "${PROJECT_ROOT}" + +echo "[dev-reload] done" diff --git a/src/autogen.ts b/src/autogen.ts index 8fd6d32..3785524 100644 --- a/src/autogen.ts +++ b/src/autogen.ts @@ -14,8 +14,8 @@ import { join } from "node:path"; import type { GraphStore } from "./graph/store.js"; import type { GraphNode } from "./graph/schema.js"; -const AUTOGEN_START = ""; -const AUTOGEN_END = ""; +const AUTOGEN_START = ""; +const AUTOGEN_END = ""; // ─── View data model ──────────────────────────────────────────────────────── @@ -333,7 +333,7 @@ export function writeToFile(filePath: string, summary: string): void { if (analysis.state === "unbalanced") { throw new Error( - `engram: cannot safely update ${filePath}: ${analysis.error} Re-run engram gen after fixing the markers.` + `engramx: cannot safely update ${filePath}: ${analysis.error} Re-run engram gen after fixing the markers.` ); } diff --git a/src/cli.ts b/src/cli.ts index 2e9e964..cfafecb 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -48,6 +48,7 @@ import { getComponentStatus, formatHudStatus } from "./intercept/component-statu import { buildEngramSection, writeEngramSectionToMemoryMd, + writeEngramSectionToPath, } from "./intercept/memory-md.js"; import { basename } from "node:path"; @@ -279,7 +280,7 @@ program const root = pathResolve(opts.project); if (!existsSync(join(root, ".engram", "graph.db"))) { console.error( - `engram: no graph found at ${root}. Run 'engram init' first.` + `engramx: no graph found at ${root}. Run 'engram init' first.` ); process.exit(1); } @@ -291,7 +292,7 @@ program process.exitCode = 0; } catch (err) { const msg = err instanceof Error ? err.message : String(err); - console.error(`engram: ${msg}`); + console.error(`engramx: ${msg}`); if (opts.verbose && err instanceof Error && err.stack) { console.error(err.stack); } @@ -548,8 +549,9 @@ program .description("Teach engram a decision, pattern, or lesson") .argument("", "What to remember (e.g., 'We chose JWT over sessions for horizontal scaling')") .option("-p, --project ", "Project directory", ".") - .action(async (text: string, opts: { project: string }) => { - const result = await learn(opts.project, text); + .option("--scope ", "Memory scope: project|global|entity", "project") + .action(async (text: string, opts: { project: string; scope: string }) => { + const result = await learn(opts.project, text, "manual", opts.scope); if (result.nodesAdded > 0) { console.log(chalk.green(`🧠 Learned ${result.nodesAdded} new insight(s).`)); } else { @@ -1463,9 +1465,10 @@ program "Write engram's structural facts into MEMORY.md (complementary to Anthropic Auto-Dream)" ) .option("-p, --project ", "Project directory", ".") + .option("--scope ", "Memory scope: project|global|entity", "project") .option("--dry-run", "Print what would be written without writing", false) .action( - async (opts: { project: string; dryRun: boolean }) => { + async (opts: { project: string; scope: string; dryRun: boolean }) => { const absProject = pathResolve(opts.project); const projectRoot = findProjectRoot(absProject); if (!projectRoot) { @@ -1476,87 +1479,189 @@ program process.exit(1); } - // Gather facts from core APIs - const [gods, mistakeList, graphStats] = await Promise.all([ - godNodes(projectRoot, 10).catch(() => []), - mistakes(projectRoot, { limit: 5 }).catch(() => []), - stats(projectRoot).catch(() => null), - ]); + const scope = typeof opts.scope === "string" && opts.scope ? opts.scope : "project"; - if (!graphStats) { - console.error(chalk.red("Failed to read graph stats.")); - process.exit(1); - } - - // Read git branch from .git/HEAD (reuses the logic pattern - // from the SessionStart handler) - let branch: string | null = null; + const { getStore } = await import("./core.js"); + const store = await getStore(projectRoot); try { - const headPath = join(projectRoot, ".git", "HEAD"); - if (existsSync(headPath)) { - const content = readFileSync(headPath, "utf-8").trim(); - const m = content.match(/^ref:\s+refs\/heads\/(.+)$/); - if (m) branch = m[1]; + const graphStats = store.getStats(scope === "project" ? projectRoot : undefined); + + // Helper to read current branch for project-scoped writes + let branch: string | null = null; + if (scope === "project") { + try { + const headPath = join(projectRoot, ".git", "HEAD"); + if (existsSync(headPath)) { + const content = readFileSync(headPath, "utf-8").trim(); + const m = content.match(/^ref:\s+refs\/heads\/(.+)$/); + if (m) branch = m[1]; + } + } catch { + /* ignore */ + } } - } catch { - /* branch stays null */ - } - const section = buildEngramSection({ - projectName: basename(projectRoot), - branch, - stats: { - nodes: graphStats.nodes, - edges: graphStats.edges, - extractedPct: graphStats.extractedPct, - }, - godNodes: gods, - landmines: mistakeList.map((m) => ({ - label: m.label, - sourceFile: m.sourceFile, - })), - lastMined: graphStats.lastMined, - }); + const slugify = (s: string) => + s + .toLowerCase() + .replace(/[^a-z0-9]+/g, "-") + .replace(/(^-|-$)/g, "") || "entity"; + + if (scope === "project") { + const godsRows = store.getGodNodes(10, projectRoot); + const gods = godsRows + .filter((g) => { + const ms = (g.node.metadata as Record).memoryScope as string | null | undefined; + return ms === "project" || ms === null || ms === undefined; + }) + .slice(0, 10) + .map((g) => ({ label: g.node.label, kind: g.node.kind, sourceFile: g.node.sourceFile })); + + let mistakeList = store.getAllNodes(projectRoot).filter((n) => n.kind === "mistake"); + mistakeList = mistakeList.filter((m) => { + const ms = (m.metadata as Record).memoryScope as string | null | undefined; + return ms === "project" || ms === null || ms === undefined; + }); + mistakeList.sort((a, b) => b.lastVerified - a.lastVerified); + mistakeList = mistakeList.slice(0, 5); + + const section = buildEngramSection({ + projectName: basename(projectRoot), + branch, + stats: { + nodes: graphStats.nodes, + edges: graphStats.edges, + extractedPct: graphStats.extractedPct, + }, + godNodes: gods, + landmines: mistakeList.map((m) => ({ label: m.label, sourceFile: m.sourceFile })), + lastMined: graphStats.lastMined, + }); + + console.log(chalk.bold(`\n📝 engram memory-sync`)); + console.log(chalk.dim(` Target: ${join(projectRoot, "MEMORY.md")}`)); + + if (opts.dryRun) { + console.log(chalk.cyan("\n Section to write (dry-run):\n")); + console.log(section.split("\n").map((l) => " " + l).join("\n")); + console.log(chalk.dim("\n (dry-run — no changes written)")); + return; + } + + const ok = writeEngramSectionToMemoryMd(projectRoot, section); + if (!ok) { + console.error(chalk.red("\n ❌ Write failed. MEMORY.md may be too large, or the engram section exceeded its size cap.")); + process.exit(1); + } + + console.log(chalk.green(`\n ✅ Synced ${gods.length} god nodes${mistakeList.length > 0 ? ` and ${mistakeList.length} landmines` : ""} to MEMORY.md`)); + console.log(chalk.dim(`\n Next: Anthropic's Auto-Dream will consolidate this alongside its prose entries.\n`)); + return; + } - console.log( - chalk.bold(`\n📝 engram memory-sync`) - ); - console.log( - chalk.dim(` Target: ${join(projectRoot, "MEMORY.md")}`) - ); + if (scope === "global") { + const godsRows = store.getGodNodes(200); + const gods = godsRows + .filter((g) => (g.node.metadata as Record).memoryScope === "global") + .slice(0, 10) + .map((g) => ({ label: g.node.label, kind: g.node.kind, sourceFile: g.node.sourceFile })); + + let mistakeList = store.getAllNodes().filter((n) => n.kind === "mistake"); + mistakeList = mistakeList.filter((m) => (m.metadata as Record).memoryScope === "global"); + mistakeList.sort((a, b) => b.lastVerified - a.lastVerified); + mistakeList = mistakeList.slice(0, 5); + + const section = buildEngramSection({ + projectName: "Global engramx memory", + branch: null, + stats: { + nodes: graphStats.nodes, + edges: graphStats.edges, + extractedPct: graphStats.extractedPct, + }, + godNodes: gods, + landmines: mistakeList.map((m) => ({ label: m.label, sourceFile: m.sourceFile })), + lastMined: graphStats.lastMined, + }); + + const globalPath = join(homedir(), ".engramx", "MEMORY.md"); + mkdirSync(dirname(globalPath), { recursive: true }); + + console.log(chalk.bold(`\n📝 engram memory-sync (global)`)); + console.log(chalk.dim(` Target: ${globalPath}`)); + + if (opts.dryRun) { + console.log(chalk.cyan("\n Section to write (dry-run):\n")); + console.log(section.split("\n").map((l) => " " + l).join("\n")); + console.log(chalk.dim("\n (dry-run — no changes written)")); + return; + } + + const ok = writeEngramSectionToPath(globalPath, section); + if (!ok) { + console.error(chalk.red("\n ❌ Write failed. Global MEMORY.md write failed or exceeded size cap.")); + process.exit(1); + } + + console.log(chalk.green(`\n ✅ Synced ${gods.length} god nodes${mistakeList.length > 0 ? ` and ${mistakeList.length} landmines` : ""} to ${globalPath}`)); + console.log(chalk.dim(`\n Next: Anthropic's Auto-Dream will consolidate this alongside its prose entries.\n`)); + return; + } - if (opts.dryRun) { - console.log(chalk.cyan("\n Section to write (dry-run):\n")); - console.log( - section - .split("\n") - .map((l) => " " + l) - .join("\n") - ); - console.log(chalk.dim("\n (dry-run — no changes written)")); - return; - } + if (scope === "entity") { + const godsRows = store.getGodNodes(500); + const entityGods = godsRows.filter((g) => (g.node.metadata as Record).memoryScope === "entity"); + + if (entityGods.length === 0) { + console.log(chalk.dim(" No entity-scoped memories found.")); + return; + } + + const outDir = join(homedir(), ".engramx", "entities"); + mkdirSync(outDir, { recursive: true }); + + let written = 0; + for (const g of entityGods.slice(0, 20)) { + const section = buildEngramSection({ + projectName: g.node.label, + branch: null, + stats: { + nodes: graphStats.nodes, + edges: graphStats.edges, + extractedPct: graphStats.extractedPct, + }, + godNodes: [{ label: g.node.label, kind: g.node.kind, sourceFile: g.node.sourceFile }], + landmines: [], + lastMined: graphStats.lastMined, + }); + + const filename = `${slugify(g.node.label)}.MEMORY.md`; + const outPath = join(outDir, filename); + + console.log(chalk.dim(` Target: ${outPath}`)); + + if (opts.dryRun) { + console.log(chalk.cyan("\n Section to write (dry-run):\n")); + console.log(section.split("\n").map((l) => " " + l).join("\n")); + continue; + } + + const ok = writeEngramSectionToPath(outPath, section); + if (ok) written++; + } + + if (!opts.dryRun) { + console.log(chalk.green(`\n ✅ Synced ${written} entity memories to ${outDir}`)); + console.log(chalk.dim(`\n Next: Anthropic's Auto-Dream will consolidate these alongside prose entries.\n`)); + } + return; + } - const ok = writeEngramSectionToMemoryMd(projectRoot, section); - if (!ok) { - console.error( - chalk.red( - "\n ❌ Write failed. MEMORY.md may be too large, or the engram section exceeded its size cap." - ) - ); + console.error(chalk.red(`Unknown scope: ${scope}`)); process.exit(1); + } finally { + store.close(); } - - console.log( - chalk.green( - `\n ✅ Synced ${gods.length} god nodes${mistakeList.length > 0 ? ` and ${mistakeList.length} landmines` : ""} to MEMORY.md` - ) - ); - console.log( - chalk.dim( - `\n Next: Anthropic's Auto-Dream will consolidate this alongside its prose entries.\n` - ) - ); } ); diff --git a/src/core.ts b/src/core.ts index ec3448f..54bafd2 100644 --- a/src/core.ts +++ b/src/core.ts @@ -13,20 +13,68 @@ import { mineGitHistory } from "./miners/git-miner.js"; import { mineSessionHistory, learnFromSession } from "./miners/session-miner.js"; import { mineSkills } from "./miners/skills-miner.js"; import type { GraphStats } from "./graph/schema.js"; +import { recordSession } from "./intelligence/token-tracker.js"; const ENGRAM_DIR = ".engram"; const DB_FILE = "graph.db"; const LOCK_FILE = "init.lock"; const DEFAULT_SKILLS_DIR = join(homedir(), ".claude", "skills"); -export function getDbPath(projectRoot: string): string { - return join(projectRoot, ENGRAM_DIR, DB_FILE); +// Global DB config: single database for all projects +const GLOBAL_DB_DIR = process.env.ENGRAM_GLOBAL_DB_DIR || join(homedir(), ".engramx"); +const GLOBAL_DB_FILE = process.env.ENGRAM_GLOBAL_DB_FILE || "memory.db"; + +export function getGlobalDbPath(): string { + return process.env.ENGRAM_GLOBAL_DB_PATH || join(GLOBAL_DB_DIR, GLOBAL_DB_FILE); +} + +export function getDbPath(_projectRoot: string): string { + // Backwards-compatible alias: always use the single global DB. + return getGlobalDbPath(); } export async function getStore(projectRoot: string): Promise { + // GraphStore is now a global DB; callers should pass projectRoot to + // project-aware methods when needed. return GraphStore.open(getDbPath(projectRoot)); } +/** + * Helper to encode a project-specific stat key stored in the global stats table. + * Use a stable base64-encoding of the projectRoot so keys are filesystem-safe. + */ +export function projectStatKey(projectRoot: string, key: string): string { + const id = Buffer.from(projectRoot).toString("base64"); + return `project:${id}:${key}`; +} + +/** + * Read the current git branch for a project. Lightweight (no shell) — reads + * .git/HEAD and returns branch name or 'detached' or null. + */ +export function readGitBranch(projectRoot: string): string | null { + try { + let current = resolve(projectRoot); + for (let depth = 0; depth < 10; depth++) { + const headPath = join(current, ".git", "HEAD"); + if (existsSync(headPath)) { + const content = readFileSync(headPath, "utf-8").trim(); + const refMatch = content.match(/^ref:\s+refs\/heads\/(.+)$/); + if (refMatch) return refMatch[1]; + if (/^[0-9a-f]{7,40}$/i.test(content)) return "detached"; + return null; + } + const parent = dirname(current); + if (parent === current) return null; + current = parent; + } + return null; + } catch { + return null; + } +} + + export interface InitResult { nodes: number; edges: number; @@ -72,7 +120,7 @@ export async function init( } catch (err: unknown) { if ((err as NodeJS.ErrnoException).code === "EEXIST") { throw new Error( - `engram: another init is running on ${root} (lock: ${lockPath}). ` + + `engramx: another init is running on ${root} (lock: ${lockPath}). ` + `If no other process is active, delete the lock file manually.` ); } @@ -85,7 +133,7 @@ export async function init( if (options.incremental) { const store = await getStore(root); try { - const mtimeJson = store.getStat("file_mtimes"); + const mtimeJson = store.getStat(projectStatKey(root, "file_mtimes")); if (mtimeJson) { previousMtimes = new Map(JSON.parse(mtimeJson)); } @@ -138,18 +186,20 @@ export async function init( const clearedFiles = new Set(); for (const node of allNodes) { if (node.sourceFile && !clearedFiles.has(node.sourceFile)) { - store.removeNodesForFile(node.sourceFile); + store.removeNodesForFile(node.sourceFile, root); clearedFiles.add(node.sourceFile); } } } else { store.clearAll(); } - store.bulkUpsert(allNodes, allEdges); - store.setStat("last_mined", String(Date.now())); - store.setStat("project_root", root); - // Persist mtimes for next incremental run - store.setStat("file_mtimes", JSON.stringify([...mtimes.entries()])); + const branch = readGitBranch(root); + // Bulk upsert with project scoping so the global DB can host multiple projects. + store.bulkUpsert(allNodes, allEdges, root, branch ?? undefined, "project"); + store.setStat(projectStatKey(root, "last_mined"), String(Date.now())); + store.setStat(projectStatKey(root, "project_root"), root); + // Persist mtimes for next incremental run (project-scoped) + store.setStat(projectStatKey(root, "file_mtimes"), JSON.stringify([...mtimes.entries()])); } finally { store.close(); } @@ -178,9 +228,61 @@ export async function query( question: string, options: { mode?: "bfs" | "dfs"; depth?: number; tokenBudget?: number } = {} ): Promise<{ text: string; estimatedTokens: number; nodesFound: number }> { + const root = resolve(projectRoot); const store = await getStore(projectRoot); try { - const result = queryGraph(store, question, options); + const result = queryGraph(store, question, { ...options, projectRoot: root }); + + // Instrument: record session metrics using full-corpus baseline. + // Baseline heuristic: naiveTokens = ceil(totalCharsAcrossProject / 4) + try { + // Collect unique source files for this project and sum their lengths. + const allNodes = store.getAllNodes(root); + const seenFiles = new Set(); + for (const n of allNodes) { + if (n.sourceFile) seenFiles.add(n.sourceFile); + } + + let totalChars = 0; + for (const f of seenFiles) { + try { + const fullPath = join(root, f); + if (existsSync(fullPath)) { + totalChars += readFileSync(fullPath, "utf-8").length; + } + } catch { + // ignore read errors + } + } + + const naiveTokens = Math.max(1, Math.ceil(totalChars / 4)); + const graphTokens = Math.max(1, Math.round(result.estimatedTokens || 0)); + + // Best-effort: record session stats into the store under project scope + try { + recordSession(store, naiveTokens, graphTokens, root); + } catch { + // non-fatal + } + } catch { + // non-fatal + } + + // Aggressive auto: ingest the query result into memory in the background + try { + void import("./intercept/auto-memory.js").then((m) => { + try { + // Use a shortened question as a relPath hint for dedupe keys + const hint = `query:${question.slice(0, 200)}`; + return m.performAutoLearnForContent(projectRoot, result.text, hint, `auto:query`); + } catch { + return undefined as unknown as Promise; + } + }).catch(() => undefined as unknown as Promise); + } catch { + /* swallow */ + } + return { text: result.text, estimatedTokens: result.estimatedTokens, nodesFound: result.nodes.length }; } finally { store.close(); @@ -194,7 +296,7 @@ export async function path( ): Promise<{ text: string; hops: number }> { const store = await getStore(projectRoot); try { - const result = shortestPath(store, source, target); + const result = shortestPath(store, source, target, undefined, projectRoot); return { text: result.text, hops: result.edges.length }; } finally { store.close(); @@ -207,7 +309,7 @@ export async function godNodes( ): Promise> { const store = await getStore(projectRoot); try { - return store.getGodNodes(topN).map((g) => ({ + return store.getGodNodes(topN, projectRoot).map((g) => ({ label: g.node.label, kind: g.node.kind, degree: g.degree, sourceFile: g.node.sourceFile, })); } finally { @@ -218,7 +320,7 @@ export async function godNodes( export async function stats(projectRoot: string): Promise { const store = await getStore(projectRoot); try { - return store.getStats(); + return store.getStats(projectRoot); } finally { store.close(); } @@ -351,7 +453,7 @@ export async function getFileContext( const store = await getStore(root); try { - const summary = renderFileStructure(store, relPath); + const summary = renderFileStructure(store, relPath, undefined, root); if (summary.codeNodeCount === 0) { // No code declarations → not worth a summary even if there's a // file metadata node. Treat as passthrough. @@ -430,7 +532,7 @@ export async function computeKeywordIDF( const store = await getStore(root); try { - const allNodes = store.getAllNodes(); + const allNodes = store.getAllNodes(projectRoot); const total = allNodes.length; if (total === 0) return []; @@ -467,20 +569,211 @@ export async function computeKeywordIDF( } } +import { generateConclusionNodes } from "./miners/conclusions-miner.js"; +import { extractLinkCandidates } from "./miners/linking-helpers.js"; + export async function learn( projectRoot: string, text: string, - sourceLabel = "manual" + sourceLabel = "manual", + memoryScope: string = "project" ): Promise<{ nodesAdded: number }> { - const { nodes, edges } = learnFromSession(text, sourceLabel); - if (nodes.length === 0 && edges.length === 0) return { nodesAdded: 0 }; + // Primary session mining (decisions/mistakes/patterns) + const sessionResult = learnFromSession(text, sourceLabel); + const conclusionResult = generateConclusionNodes(text, sourceLabel); + + const combinedNodes = [...sessionResult.nodes, ...conclusionResult.nodes]; + const combinedEdges = [...sessionResult.edges, ...conclusionResult.edges]; + + if (combinedNodes.length === 0 && combinedEdges.length === 0) return { nodesAdded: 0 }; + const store = await getStore(projectRoot); try { - store.bulkUpsert(nodes, edges); + // Bulk upsert nodes + edges (project-scoped) + store.bulkUpsert(combinedNodes, combinedEdges, projectRoot, undefined, memoryScope); + + // Ensure the project is discoverable even when graph content was created + // by a manual `learn` call (no full init). Write a namespaced project_root + // stat entry so the dashboard's project list includes this project. + try { + store.setStat(projectStatKey(projectRoot, "project_root"), projectRoot); + } catch { + // best-effort — non-fatal if stats write fails + } + + // Post-insert: create linking edges from conclusion nodes to existing + // graph nodes by simple keyword overlap. This helps surface relations + // between learned conclusions/fragments and code entities/files. + const now = Date.now(); + const allNodes = store.getAllNodes(projectRoot); + + const edgesToAdd = [] as typeof combinedEdges; + const seen = new Set(); + + for (const c of conclusionResult.nodes) { + // only consider conclusion/pattern nodes we created (metadata marker) + if (!c.metadata || (c.metadata as Record).miner !== "conclusion") continue; + + // Gather candidates from both the node label and the full session text + const scanText = `${c.label}\n${text}`; + const { keywords, filePaths, commands } = extractLinkCandidates(scanText); + + // Use IDF filtering to drop overly-common graph terms + let goodTokens: string[] = []; + try { + const idf = await computeKeywordIDF(projectRoot, keywords.slice(0, 80)); + goodTokens = idf.filter((r) => r.idf > 0).slice(0, 12).map((r) => r.keyword.toLowerCase()); + } catch { + goodTokens = keywords.slice(0, 12).map((k) => k.toLowerCase()); + } + + // 1) Keyword-based linking (similar_to) + for (const tok of goodTokens) { + for (const n of allNodes) { + if (n.id === c.id) continue; + if (n.label.toLowerCase().includes(tok)) { + const key = `${c.id}|${n.id}|similar_to`; + if (seen.has(key)) continue; + seen.add(key); + edgesToAdd.push({ + source: c.id, + target: n.id, + relation: "similar_to", + confidence: "INFERRED", + confidenceScore: 0.6, + sourceFile: sourceLabel, + sourceLocation: null, + lastVerified: now, + metadata: { auto: true, matchedToken: tok }, + }); + } else if (n.metadata && JSON.stringify(n.metadata).toLowerCase().includes(tok)) { + const key = `${c.id}|${n.id}|similar_to`; + if (seen.has(key)) continue; + seen.add(key); + edgesToAdd.push({ + source: c.id, + target: n.id, + relation: "similar_to", + confidence: "INFERRED", + confidenceScore: 0.55, + sourceFile: sourceLabel, + sourceLocation: null, + lastVerified: now, + metadata: { auto: true, matchedToken: tok, metaMatch: true }, + }); + } + } + } + + // 2) File path linking (depends_on) + for (const fp of filePaths) { + try { + let candidate = fp.replace(/^\.\//, ""); + candidate = candidate.replace(/^[A-Z]:\\/i, ""); + + const fileNodes = store.getNodesByFile(candidate, 500, projectRoot); + if (fileNodes.length > 0) { + for (const fn of fileNodes) { + const key = `${c.id}|${fn.id}|depends_on`; + if (seen.has(key)) continue; + seen.add(key); + edgesToAdd.push({ + source: c.id, + target: fn.id, + relation: "depends_on", + confidence: "INFERRED", + confidenceScore: 0.85, + sourceFile: sourceLabel, + sourceLocation: null, + lastVerified: now, + metadata: { auto: true, detectedPath: fp }, + }); + } + continue; + } + + // Fallback: match basename against node labels / sourceFile endings + const base = (candidate.split(/[\\/]/).pop() || candidate).toLowerCase(); + for (const n of allNodes) { + if (!n.sourceFile && !n.label) continue; + const sf = (n.sourceFile || "").toLowerCase(); + if (sf.endsWith(base) || (n.label || "").toLowerCase().includes(base)) { + const key = `${c.id}|${n.id}|depends_on`; + if (seen.has(key)) continue; + seen.add(key); + edgesToAdd.push({ + source: c.id, + target: n.id, + relation: "depends_on", + confidence: "INFERRED", + confidenceScore: 0.75, + sourceFile: sourceLabel, + sourceLocation: null, + lastVerified: now, + metadata: { auto: true, detectedPath: fp, fallback: true }, + }); + } + } + } catch { + // non-fatal + } + } + + // 3) Command mentions (mentions) + for (const cmd of commands) { + const lower = cmd.toLowerCase(); + for (const n of allNodes) { + if (n.id === c.id) continue; + if ((n.label || "").toLowerCase().includes(lower) || JSON.stringify(n.metadata || {}).toLowerCase().includes(lower)) { + const key = `${c.id}|${n.id}|mentions`; + if (seen.has(key)) continue; + seen.add(key); + edgesToAdd.push({ + source: c.id, + target: n.id, + relation: "mentions", + confidence: "INFERRED", + confidenceScore: 0.55, + sourceFile: sourceLabel, + sourceLocation: null, + lastVerified: now, + metadata: { auto: true, matchedCommand: cmd }, + }); + } + } + } + } + + if (edgesToAdd.length > 0) { + // Upsert linking edges (no new nodes) + store.bulkUpsert([], edgesToAdd, projectRoot, undefined, memoryScope); + } } finally { store.close(); } - return { nodesAdded: nodes.length }; + + // If this project has never been mined (no last_mined stat), trigger a + // best-effort incremental init in the background so file-level nodes + // (AST-extracted) become available for Read interception and the + // dashboard's Files tab. This is fire-and-forget and must not block + // the calling thread. + (async () => { + try { + const s = await getStore(projectRoot); + try { + const lm = s.getStat(projectStatKey(projectRoot, "last_mined")); + if (!lm || Number(lm) === 0) { + await init(projectRoot, { incremental: true }); + } + } finally { + s.close(); + } + } catch { + // swallow background init failures — learning succeeded regardless + } + })(); + + return { nodesAdded: combinedNodes.length }; } export interface MistakeEntry { @@ -511,7 +804,7 @@ export async function mistakes( ): Promise { const store = await getStore(projectRoot); try { - let items = store.getAllNodes().filter((n) => n.kind === "mistake"); + let items = store.getAllNodes(projectRoot).filter((n) => n.kind === "mistake"); if (options.sourceFile !== undefined) { const target = options.sourceFile; diff --git a/src/db/migrate.ts b/src/db/migrate.ts index 5b87546..b33b101 100644 --- a/src/db/migrate.ts +++ b/src/db/migrate.ts @@ -15,7 +15,7 @@ export interface MigrationResult { } /** Current schema version — bump this when adding new migrations. */ -export const CURRENT_SCHEMA_VERSION = 8; +export const CURRENT_SCHEMA_VERSION = 10; export interface RollbackResult { readonly fromVersion: number; @@ -184,6 +184,114 @@ CREATE INDEX IF NOT EXISTS idx_query_cache_file ON query_cache(file_path);`, WHERE kind = 'mistake' AND valid_until IS NOT NULL; `); }, + + // v3.1.0: project scoping — add project_root/project_branch/memory_scope + // columns so a single global DB can host multiple projects' data and + // memory types (project/global/entity). Backfill is optional — new + // rows will populate these columns automatically. + 9: (db: ExecDb) => { + addColumnIfMissing(db, "nodes", "project_root", "project_root TEXT NOT NULL DEFAULT ''"); + addColumnIfMissing(db, "nodes", "project_branch", "project_branch TEXT"); + addColumnIfMissing(db, "nodes", "memory_scope", "memory_scope TEXT"); + + addColumnIfMissing(db, "edges", "project_root", "project_root TEXT NOT NULL DEFAULT ''"); + + // Add project_root to provider_cache so caches can be scoped per project + addColumnIfMissing(db, "provider_cache", "project_root", "project_root TEXT NOT NULL DEFAULT ''"); + + // Index project_root on nodes/edges/provider_cache for efficient per-project queries + try { db.exec("CREATE INDEX IF NOT EXISTS idx_nodes_project_root ON nodes(project_root)"); } catch {} + try { db.exec("CREATE INDEX IF NOT EXISTS idx_edges_project_root ON edges(project_root)"); } catch {} + try { db.exec("CREATE INDEX IF NOT EXISTS idx_provider_cache_project_root ON provider_cache(project_root)"); } catch {} + }, + + // v3.2.0: canonical ids for deterministic dedupe + migration to merge existing duplicates + 10: (db: ExecDb) => { + addColumnIfMissing(db, "nodes", "canonical_id", "canonical_id TEXT"); + try { db.exec("CREATE INDEX IF NOT EXISTS idx_nodes_canonical_project ON nodes(canonical_id, project_root)"); } catch {} + + // Compute canonical ids for existing nodes and backfill + try { + // Import computeCanonicalId dynamically so migrations are self-contained + const { computeCanonicalId } = require("../graph/canonical.js"); + const res = db.exec("SELECT id, label, kind, memory_scope, project_root FROM nodes"); + const rows = (res[0] && res[0].values) ? res[0].values : []; + for (const r of rows) { + const id = String(r[0] ?? ""); + const label = String(r[1] ?? ""); + const kind = String(r[2] ?? ""); + const memoryScope = r[3] ?? "project"; + const projectRoot = r[4] ?? ""; + const canonical = computeCanonicalId(label, kind, memoryScope, projectRoot); + try { + // Use parameterized run when available + (db as unknown as RunDb).run("UPDATE nodes SET canonical_id = ? WHERE id = ?", [canonical, id]); + } catch { + try { db.exec(`UPDATE nodes SET canonical_id = '${String(canonical).replace(/'/g, "''")}' WHERE id = '${String(id).replace(/'/g, "''")}'`); } catch {} + } + } + + // Find canonical groups with >1 member and merge duplicates + const groups = db.exec("SELECT canonical_id, project_root FROM nodes WHERE canonical_id IS NOT NULL AND canonical_id <> '' GROUP BY canonical_id, project_root HAVING COUNT(*) > 1"); + const gvals = (groups[0] && groups[0].values) ? groups[0].values : []; + for (const gv of gvals) { + const canonical = String(gv[0]); + const proj = String(gv[1] ?? ""); + const selSql = `SELECT id FROM nodes WHERE canonical_id = '${String(canonical).replace(/'/g, "''")}' AND project_root = '${String(proj).replace(/'/g, "''")}' ORDER BY confidence_score DESC, last_verified DESC`; + const sel = db.exec(selSql); + const nrows = (sel[0] && sel[0].values) ? sel[0].values.map((v) => String(v[0])) : []; + if (nrows.length <= 1) continue; + const primary = nrows[0]; + const duplicates = nrows.slice(1); + + for (const dup of duplicates) { + try { + // Re-insert source edges with primary as source (INSERT OR IGNORE) + const srcEdges = db.exec(`SELECT source, target, relation, confidence, confidence_score, source_file, source_location, last_verified, metadata, project_root FROM edges WHERE source = '${String(dup).replace(/'/g, "''")}'`); + const srcVals = (srcEdges[0] && srcEdges[0].values) ? srcEdges[0].values : []; + for (const e of srcVals) { + const [_s, target, relation, confidence, confidence_score, source_file, source_location, last_verified, metadata, edge_proj] = e; + try { + (db as unknown as RunDb).run( + `INSERT OR IGNORE INTO edges (source, target, relation, confidence, confidence_score, source_file, source_location, last_verified, metadata, project_root) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + [primary, target, relation, confidence, confidence_score, source_file, source_location, last_verified, metadata, edge_proj] + ); + } catch { /* best-effort */ } + } + + // Re-insert target edges with primary as target + const tgtEdges = db.exec(`SELECT source, target, relation, confidence, confidence_score, source_file, source_location, last_verified, metadata, project_root FROM edges WHERE target = '${String(dup).replace(/'/g, "''")}'`); + const tgtVals = (tgtEdges[0] && tgtEdges[0].values) ? tgtEdges[0].values : []; + for (const e of tgtVals) { + const [source, _t, relation, confidence, confidence_score, source_file, source_location, last_verified, metadata, edge_proj] = e; + try { + (db as unknown as RunDb).run( + `INSERT OR IGNORE INTO edges (source, target, relation, confidence, confidence_score, source_file, source_location, last_verified, metadata, project_root) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + [source, primary, relation, confidence, confidence_score, source_file, source_location, last_verified, metadata, edge_proj] + ); + } catch { /* best-effort */ } + } + + // Remove old edges referencing the duplicate + try { + (db as unknown as RunDb).run("DELETE FROM edges WHERE source = ? OR target = ?", [dup, dup]); + } catch { db.exec(`DELETE FROM edges WHERE source = '${String(dup).replace(/'/g, "''")}' OR target = '${String(dup).replace(/'/g, "''")}'`); } + + // Finally remove the duplicate node + try { + (db as unknown as RunDb).run("DELETE FROM nodes WHERE id = ?", [dup]); + } catch { db.exec(`DELETE FROM nodes WHERE id = '${String(dup).replace(/'/g, "''")}'`); } + } catch { + // continue on error for best-effort merging + } + } + } + } catch (e) { + // Best-effort migration: if any of the merge steps fail, we don't + // want to abort the whole migration run. The canonical_id column + // still exists and the system will continue to operate. + } + }, }; type ExecDb = { exec: (sql: string) => Array<{ values: unknown[][] }> }; diff --git a/src/doctor/report.ts b/src/doctor/report.ts index 41ec017..6fc2f35 100644 --- a/src/doctor/report.ts +++ b/src/doctor/report.ts @@ -115,7 +115,7 @@ function componentToCheck(c: ComponentHealth): DoctorCheck { lsp: "LSP is best-effort — install a language server (typescript-language-server, pyright, rust-analyzer).", ast: - "Tree-sitter grammars missing. Reinstall engram: `engram update` or `npm install -g engramx@latest`.", + "Tree-sitter grammars missing. Reinstall engramx: `engram update` or `npm install -g engramx@latest`.", }; return { name: c.name, diff --git a/src/graph/canonical.ts b/src/graph/canonical.ts new file mode 100644 index 0000000..53a0a53 --- /dev/null +++ b/src/graph/canonical.ts @@ -0,0 +1,37 @@ +import { createHash } from "node:crypto"; + +// Deterministic canonical id generation for nodes. +// Inputs: label, kind, memoryScope, projectRoot (optional). We normalize +// the label (NFKC, collapse whitespace, remove punctuation) and hash a +// versioned namespace so we can change algorithm later. + +function normalizeLabel(s: string): string { + if (!s) return ""; + try { + // Unicode normalization to NFKC + let t = s.normalize("NFKC"); + // Replace non-letter/number/space with nothing (remove punctuation) + t = t.replace(/[^\p{L}\p{N}\s]/gu, " "); + // Collapse whitespace and trim + t = t.replace(/\s+/g, " ").trim(); + return t.toLowerCase(); + } catch { + return s.replace(/\s+/g, " ").trim().toLowerCase(); + } +} + +export function computeCanonicalId( + label: string, + kind: string | undefined, + memoryScope: string | undefined, + projectRoot?: string | null +): string { + const version = "v1"; // bump on algorithm changes + const norm = normalizeLabel(label || ""); + const scope = memoryScope || "project"; + const kindPart = kind || ""; + const rootPart = projectRoot ? String(projectRoot) : ""; + const base = `${version}|${scope}|${kindPart}|${norm}|${rootPart}`; + const hash = createHash("sha256").update(base).digest("hex").slice(0, 24); + return `c_${hash}`; +} diff --git a/src/graph/query.ts b/src/graph/query.ts index b071a9a..317b0fb 100644 --- a/src/graph/query.ts +++ b/src/graph/query.ts @@ -50,7 +50,8 @@ interface TraversalResult { function scoreNodes( store: GraphStore, - terms: string[] + terms: string[], + projectRoot?: string ): Array<{ score: number; node: GraphNode }> { // Use SQL-level filtering via searchNodes instead of materializing // the entire graph into JS. Each term seeds from the index; we then @@ -58,7 +59,7 @@ function scoreNodes( const seen = new Set(); const seedNodes: GraphNode[] = []; for (const t of terms) { - for (const node of store.searchNodes(t, 200)) { + for (const node of store.searchNodes(t, 200, projectRoot)) { if (!seen.has(node.id)) { seen.add(node.id); seedNodes.push(node); @@ -108,15 +109,15 @@ function scoreNodes( export function queryGraph( store: GraphStore, question: string, - options: { mode?: "bfs" | "dfs"; depth?: number; tokenBudget?: number } = {} + options: { mode?: "bfs" | "dfs"; depth?: number; tokenBudget?: number; projectRoot?: string } = {} ): TraversalResult { - const { mode = "bfs", depth = 3, tokenBudget = 2000 } = options; + const { mode = "bfs", depth = 3, tokenBudget = 2000, projectRoot } = options; const terms = question .toLowerCase() .split(/\s+/) .filter((t) => t.length > 2); - const scored = scoreNodes(store, terms); + const scored = scoreNodes(store, terms, projectRoot); const startNodes = scored.slice(0, 3).map((s) => s.node); if (startNodes.length === 0) { @@ -150,7 +151,7 @@ export function queryGraph( for (let d = 0; d < depth; d++) { const nextFrontier = new Set(); for (const nid of frontier) { - const neighbors = store.getNeighbors(nid); + const neighbors = store.getNeighbors(nid, undefined, projectRoot); for (const { node, edge } of neighbors) { if (shouldSkipEdgeFrom(nid, edge)) continue; if (!visited.has(node.id)) { @@ -169,7 +170,7 @@ export function queryGraph( while (stack.length > 0) { const { id, d } = stack.pop()!; if (d > depth) continue; - const neighbors = store.getNeighbors(id); + const neighbors = store.getNeighbors(id, undefined, projectRoot); for (const { node, edge } of neighbors) { if (shouldSkipEdgeFrom(id, edge)) continue; if (!visited.has(node.id)) { @@ -199,13 +200,14 @@ export function shortestPath( store: GraphStore, sourceTerm: string, targetTerm: string, - maxHops = 8 + maxHops = 8, + projectRoot?: string ): TraversalResult { const sourceTerms = sourceTerm.toLowerCase().split(/\s+/).filter((t) => t.length > 2); const targetTerms = targetTerm.toLowerCase().split(/\s+/).filter((t) => t.length > 2); - const sourceScored = scoreNodes(store, sourceTerms); - const targetScored = scoreNodes(store, targetTerms); + const sourceScored = scoreNodes(store, sourceTerms, projectRoot); + const targetScored = scoreNodes(store, targetTerms, projectRoot); if (sourceScored.length === 0 || targetScored.length === 0) { return { @@ -235,7 +237,7 @@ export function shortestPath( const node = store.getNode(path[i]); if (node) pathNodes.push(node); if (i < path.length - 1) { - const neighbors = store.getNeighbors(path[i]); + const neighbors = store.getNeighbors(path[i], undefined, projectRoot); const edge = neighbors.find((n) => n.node.id === path[i + 1])?.edge; if (edge) pathEdges.push(edge); } @@ -251,7 +253,7 @@ export function shortestPath( if (path.length > maxHops) continue; - const neighbors = store.getNeighbors(current); + const neighbors = store.getNeighbors(current, undefined, projectRoot); for (const { node } of neighbors) { if (!seen.has(node.id)) { seen.add(node.id); @@ -440,12 +442,13 @@ export interface FileStructureResult { export function renderFileStructure( store: GraphStore, relativeFilePath: string, - tokenBudget = 600 + tokenBudget = 600, + projectRoot?: string ): FileStructureResult { // Use targeted SQL queries instead of full table scans. On 50k-node // projects, getAllNodes()/getAllEdges() materialize the entire graph // into JS arrays and time out silently. - const allFileNodes = store.getNodesByFile(relativeFilePath); + const allFileNodes = store.getNodesByFile(relativeFilePath, 500, projectRoot); const fileNodes = allFileNodes.filter((n) => !isHiddenKeyword(n)); if (fileNodes.length === 0) { @@ -471,7 +474,7 @@ export function renderFileStructure( // Fetch only edges that touch this file's nodes (indexed query). const fileNodeIds = new Set(fileNodes.map((n) => n.id)); - const fileEdges = store.getEdgesForNodes([...fileNodeIds]); + const fileEdges = store.getEdgesForNodes([...fileNodeIds], projectRoot); // Degree map: how many edges touch each node in this file. Used to sort // nodes within each kind group so the most-connected (= most important) @@ -520,7 +523,7 @@ export function renderFileStructure( ]; const lines: string[] = []; - lines.push(`[engram] Structural summary for ${relativeFilePath}`); + lines.push(`[engramx] Structural summary for ${relativeFilePath}`); lines.push( `Nodes: ${fileNodes.length} | avg extraction confidence: ${avgConfidence.toFixed(2)}` ); diff --git a/src/graph/store.ts b/src/graph/store.ts index 0efad5d..6a78ea8 100644 --- a/src/graph/store.ts +++ b/src/graph/store.ts @@ -99,15 +99,87 @@ export class GraphStore { runMigrations(this.db, this.dbPath); } + // Non-blocking save: export the DB synchronously (cheap) then + // write the bytes to disk asynchronously so file I/O doesn't block + // the main event loop. Coalesce rapid successive saves into a single + // final write to avoid thrashing the filesystem. + private _savePending = false as boolean; + private _saveQueuedBuffer: Buffer | null = null; + save(): void { - const data = this.db.export(); - writeFileSync(this.dbPath, Buffer.from(data)); + let data: Uint8Array; + try { + data = this.db.export(); + } catch (e) { + // If export fails for any reason, fall back to a best-effort + // synchronous write attempt (rare). Swallow errors to avoid + // crashing callers. + try { + writeFileSync(this.dbPath, Buffer.from([])); + } catch { + /* swallow */ + } + return; + } + + const buffer = Buffer.from(data); + const tmpPath = this.dbPath + ".tmp"; + + // If a save is already in progress, stash the latest buffer and + // return. The in-progress write will detect _saveQueuedBuffer and + // write it afterwards. This coalesces rapid updates. + if (this._savePending) { + this._saveQueuedBuffer = buffer; + return; + } + + this._savePending = true; + const writeOnce = async (): Promise => { + try { + // Async write temp -> rename for atomicity + await import("node:fs/promises").then((fs) => fs.writeFile(tmpPath, buffer)); + await import("node:fs/promises").then((fs) => fs.rename(tmpPath, this.dbPath)); + + // If another save was queued while we were writing, write it now + if (this._saveQueuedBuffer) { + const nextBuf = this._saveQueuedBuffer; + this._saveQueuedBuffer = null; + try { + await import("node:fs/promises").then((fs) => fs.writeFile(tmpPath, nextBuf!)); + await import("node:fs/promises").then((fs) => fs.rename(tmpPath, this.dbPath)); + } catch (e) { + // Best-effort fallback to synchronous write + try { + writeFileSync(this.dbPath, nextBuf!); + } catch { + /* swallow */ + } + } + } + } catch (e) { + // Best-effort fallback: synchronous write + try { + writeFileSync(this.dbPath, buffer); + } catch { + /* swallow */ + } + } finally { + this._savePending = false; + } + }; + + // Fire-and-forget + void writeOnce(); } - upsertNode(node: GraphNode): void { + upsertNode(node: GraphNode, defaults?: { projectRoot?: string; projectBranch?: string; memoryScope?: string }): void { + const projectRoot = (node as any).projectRoot ?? defaults?.projectRoot ?? ""; + const projectBranch = (node as any).projectBranch ?? defaults?.projectBranch ?? null; + const memoryScope = (node as any).memoryScope ?? defaults?.memoryScope ?? null; + this.db.run( - `INSERT OR REPLACE INTO nodes (id, label, kind, source_file, source_location, confidence, confidence_score, last_verified, query_count, metadata, valid_until, invalidated_by_commit) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + `INSERT OR REPLACE INTO nodes (id, label, kind, source_file, source_location, confidence, confidence_score, last_verified, query_count, metadata, valid_until, invalidated_by_commit, project_root, project_branch, memory_scope) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, [ node.id, node.label, @@ -121,14 +193,134 @@ export class GraphStore { JSON.stringify(node.metadata), node.validUntil ?? null, node.invalidatedByCommit ?? null, + projectRoot, + projectBranch, + memoryScope, ] ); } - upsertEdge(edge: GraphEdge): void { + /** + * Merge-aware single node upsert. If a node supplies a canonical_id (or + * if a canonical id can be derived from the node.id prefix `c_`), the + * method will try to find an existing node with that canonical_id (scoped + * to project_root) and merge metadata/fields rather than blindly + * replacing the row. + * + * Returns the final node id written to the DB (existing or newly-created). + */ + mergeUpsertNode(node: GraphNode, defaults?: { projectRoot?: string; projectBranch?: string; memoryScope?: string }): string { + const projectRoot = defaults?.projectRoot ?? ""; + const projectBranch = defaults?.projectBranch ?? null; + const memoryScope = defaults?.memoryScope ?? null; + + const canonical = (node as any).canonicalId ?? node.id?.startsWith("c_") ? (node as any).canonicalId ?? node.id : null; + + if (!canonical) { + // No canonical id — fall back to simple upsert using provided id. + this.upsertNode(node, { projectRoot, projectBranch, memoryScope }); + return node.id; + } + + // Try to find existing node by canonical_id (project-scoped) + const sql = projectRoot + ? "SELECT * FROM nodes WHERE canonical_id = ? AND project_root = ? LIMIT 1" + : "SELECT * FROM nodes WHERE canonical_id = ? LIMIT 1"; + const stmt = this.db.prepare(sql); + if (projectRoot) stmt.bind([canonical, projectRoot]); + else stmt.bind([canonical]); + + if (stmt.step()) { + const row = stmt.getAsObject(); + stmt.free(); + const existing = this.rowToNode(row); + + // Merge metadata (shallow merge; arrays concatenated dedup) + const existingMeta = existing.metadata ?? {}; + const incomingMeta = node.metadata ?? {}; + const mergedMeta: Record = { ...existingMeta }; + for (const k of Object.keys(incomingMeta)) { + const v = (incomingMeta as Record)[k]; + const ev = (existingMeta as Record)[k]; + if (Array.isArray(ev) && Array.isArray(v)) { + mergedMeta[k] = Array.from(new Set([...ev, ...v])); + } else if (ev === undefined || ev === null) { + mergedMeta[k] = v; + } else { + // prefer existing value for opaque fields + mergedMeta[k] = ev; + } + } + + const mergedLabel = (existing.label && existing.label.length >= (node.label || "").length) ? existing.label : (node.label || existing.label); + const mergedKind = existing.kind || node.kind; + const mergedSourceFile = existing.sourceFile || node.sourceFile || ""; + const mergedSourceLocation = existing.sourceLocation || node.sourceLocation || null; + const mergedConfidence = existing.confidence || node.confidence; + const mergedConfidenceScore = Math.max(existing.confidenceScore || 0, node.confidenceScore || 0); + const mergedLastVerified = Math.max(existing.lastVerified || 0, node.lastVerified || 0); + const mergedQueryCount = Math.max(existing.queryCount || 0, node.queryCount || 0); + + const updateSql = `UPDATE nodes SET label = ?, kind = ?, source_file = ?, source_location = ?, confidence = ?, confidence_score = ?, last_verified = ?, query_count = ?, metadata = ?, valid_until = ?, invalidated_by_commit = ?, project_root = ?, project_branch = ?, memory_scope = ?, canonical_id = ? WHERE id = ?`; + + let metadataStr = "{}"; + try { metadataStr = JSON.stringify(mergedMeta); } catch { metadataStr = "{}"; } + + this.db.run(updateSql, [ + mergedLabel, + mergedKind, + mergedSourceFile, + mergedSourceLocation, + mergedConfidence, + mergedConfidenceScore, + mergedLastVerified, + mergedQueryCount, + metadataStr, + node.validUntil ?? null, + node.invalidatedByCommit ?? null, + projectRoot, + projectBranch, + memoryScope, + canonical, + existing.id, + ]); + + return existing.id; + } + + // No existing — insert a new node using the canonical id as the primary id + const idToUse = canonical; + this.db.run( + `INSERT OR REPLACE INTO nodes (id, label, kind, source_file, source_location, confidence, confidence_score, last_verified, query_count, metadata, valid_until, invalidated_by_commit, project_root, project_branch, memory_scope, canonical_id) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + [ + idToUse, + node.label, + node.kind, + node.sourceFile, + node.sourceLocation, + node.confidence, + node.confidenceScore, + node.lastVerified, + node.queryCount, + JSON.stringify(node.metadata), + node.validUntil ?? null, + node.invalidatedByCommit ?? null, + projectRoot, + projectBranch, + memoryScope, + canonical, + ] + ); + + return idToUse; + } + + upsertEdge(edge: GraphEdge, defaults?: { projectRoot?: string }): void { + const projectRoot = (edge as any).projectRoot ?? defaults?.projectRoot ?? ""; this.db.run( - `INSERT OR REPLACE INTO edges (source, target, relation, confidence, confidence_score, source_file, source_location, last_verified, metadata) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`, + `INSERT OR REPLACE INTO edges (source, target, relation, confidence, confidence_score, source_file, source_location, last_verified, metadata, project_root) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, [ edge.source, edge.target, @@ -139,20 +331,27 @@ export class GraphStore { edge.sourceLocation, edge.lastVerified, JSON.stringify(edge.metadata), + projectRoot, ] ); } /** * Remove all nodes and edges associated with a specific source file. + * If projectRoot is provided, only clears nodes/edges for that project. * Used by the file watcher for incremental re-indexing — old nodes for * a changed file are cleared before re-extracting. */ - deleteBySourceFile(sourceFile: string): void { + deleteBySourceFile(sourceFile: string, projectRoot?: string): void { this.db.run("BEGIN TRANSACTION"); try { - this.db.run("DELETE FROM edges WHERE source_file = ?", [sourceFile]); - this.db.run("DELETE FROM nodes WHERE source_file = ?", [sourceFile]); + if (projectRoot) { + this.db.run("DELETE FROM edges WHERE source_file = ? AND project_root = ?", [sourceFile, projectRoot]); + this.db.run("DELETE FROM nodes WHERE source_file = ? AND project_root = ?", [sourceFile, projectRoot]); + } else { + this.db.run("DELETE FROM edges WHERE source_file = ?", [sourceFile]); + this.db.run("DELETE FROM nodes WHERE source_file = ?", [sourceFile]); + } this.db.run("COMMIT"); } catch (e) { this.db.run("ROLLBACK"); @@ -160,11 +359,13 @@ export class GraphStore { } } - countBySourceFile(sourceFile: string): number { - const stmt = this.db.prepare( - "SELECT COUNT(*) AS n FROM nodes WHERE source_file = ?" - ); - stmt.bind([sourceFile]); + countBySourceFile(sourceFile: string, projectRoot?: string): number { + const sql = projectRoot + ? "SELECT COUNT(*) AS n FROM nodes WHERE source_file = ? AND project_root = ?" + : "SELECT COUNT(*) AS n FROM nodes WHERE source_file = ?"; + const stmt = this.db.prepare(sql); + if (projectRoot) stmt.bind([sourceFile, projectRoot]); + else stmt.bind([sourceFile]); let count = 0; if (stmt.step()) { const row = stmt.getAsObject() as { n: number }; @@ -174,10 +375,35 @@ export class GraphStore { return count; } - bulkUpsert(nodes: GraphNode[], edges: GraphEdge[]): void { + /** + * Merge-aware bulk upsert. Nodes that provide a canonical_id will be + * merged into any existing node with the same canonical_id (project-scoped). + * Returns after inserting/updating nodes and edges; edges referencing + * replaced node ids are remapped to the merged node id. + */ + bulkUpsert(nodes: GraphNode[], edges: GraphEdge[], projectRoot?: string, projectBranch?: string, memoryScope?: string): void { this.db.run("BEGIN TRANSACTION"); - for (const node of nodes) this.upsertNode(node); - for (const edge of edges) this.upsertEdge(edge); + const idMap = new Map(); + + for (const node of nodes) { + try { + const finalId = this.mergeUpsertNode(node, { projectRoot, projectBranch, memoryScope }); + if (finalId && finalId !== node.id) idMap.set(node.id, finalId); + } catch (e) { + // Best-effort: swallow per-node errors to avoid failing the whole bulk + } + } + + for (const edge of edges) { + // Remap source/target ids if nodes were canonicalized/merged + const s = idMap.get(edge.source) ?? edge.source; + const t = idMap.get(edge.target) ?? edge.target; + const edgeCopy = { ...edge, source: s, target: t }; + try { + this.upsertEdge(edgeCopy, { projectRoot }); + } catch {} + } + this.db.run("COMMIT"); this.save(); } @@ -194,14 +420,16 @@ export class GraphStore { return null; } - searchNodes(query: string, limit = 20): GraphNode[] { + searchNodes(query: string, limit = 20, projectRoot?: string): GraphNode[] { const escaped = query.replace(/%/g, "\\%").replace(/_/g, "\\_"); const pattern = `%${escaped}%`; const results: GraphNode[] = []; - const stmt = this.db.prepare( - "SELECT * FROM nodes WHERE label LIKE ? ESCAPE '\\' OR id LIKE ? ESCAPE '\\' ORDER BY query_count DESC LIMIT ?" - ); - stmt.bind([pattern, pattern, limit]); + const sql = projectRoot + ? "SELECT * FROM nodes WHERE (label LIKE ? ESCAPE '\\' OR id LIKE ? ESCAPE '\\') AND project_root = ? ORDER BY query_count DESC LIMIT ?" + : "SELECT * FROM nodes WHERE label LIKE ? ESCAPE '\\' OR id LIKE ? ESCAPE '\\' ORDER BY query_count DESC LIMIT ?"; + const stmt = this.db.prepare(sql); + if (projectRoot) stmt.bind([pattern, pattern, projectRoot, limit]); + else stmt.bind([pattern, pattern, limit]); while (stmt.step()) { results.push(this.rowToNode(stmt.getAsObject())); } @@ -211,14 +439,23 @@ export class GraphStore { getNeighbors( nodeId: string, - relationFilter?: EdgeRelation + relationFilter?: EdgeRelation, + projectRoot?: string ): Array<{ node: GraphNode; edge: GraphEdge }> { const sql = relationFilter - ? "SELECT * FROM edges WHERE (source = ? OR target = ?) AND relation = ?" - : "SELECT * FROM edges WHERE source = ? OR target = ?"; + ? projectRoot + ? "SELECT * FROM edges WHERE (source = ? OR target = ?) AND relation = ? AND project_root = ?" + : "SELECT * FROM edges WHERE (source = ? OR target = ?) AND relation = ?" + : projectRoot + ? "SELECT * FROM edges WHERE (source = ? OR target = ?) AND project_root = ?" + : "SELECT * FROM edges WHERE source = ? OR target = ?"; const params = relationFilter - ? [nodeId, nodeId, relationFilter] - : [nodeId, nodeId]; + ? projectRoot + ? [nodeId, nodeId, relationFilter, projectRoot] + : [nodeId, nodeId, relationFilter] + : projectRoot + ? [nodeId, nodeId, projectRoot] + : [nodeId, nodeId]; const stmt = this.db.prepare(sql); stmt.bind(params); @@ -233,23 +470,31 @@ export class GraphStore { return results; } - getGodNodes(topN = 10): Array<{ node: GraphNode; degree: number }> { + getGodNodes(topN = 10, projectRoot?: string): Array<{ node: GraphNode; degree: number }> { const results: Array<{ node: GraphNode; degree: number }> = []; // Exclude structural plumbing (file/import/module) AND concept nodes. // The `concept` kind is used by the skills-miner for both skills and // keyword nodes — a keyword like "landing page" may have hundreds of // triggered_by edges but isn't a "core abstraction" of the codebase. // Users want real code entities + decisions/patterns/mistakes here. - const stmt = this.db.prepare( - `SELECT n.*, COUNT(*) as degree - FROM nodes n - JOIN edges e ON e.source = n.id OR e.target = n.id - WHERE n.kind NOT IN ('file', 'import', 'module', 'concept') - GROUP BY n.id - ORDER BY degree DESC - LIMIT ?` - ); - stmt.bind([topN]); + const sql = projectRoot + ? `SELECT n.*, COUNT(*) as degree + FROM nodes n + JOIN edges e ON e.source = n.id OR e.target = n.id + WHERE n.kind NOT IN ('file', 'import', 'module', 'concept') AND n.project_root = ? + GROUP BY n.id + ORDER BY degree DESC + LIMIT ?` + : `SELECT n.*, COUNT(*) as degree + FROM nodes n + JOIN edges e ON e.source = n.id OR e.target = n.id + WHERE n.kind NOT IN ('file', 'import', 'module', 'concept') + GROUP BY n.id + ORDER BY degree DESC + LIMIT ?`; + const stmt = this.db.prepare(sql); + if (projectRoot) stmt.bind([projectRoot, topN]); + else stmt.bind([topN]); while (stmt.step()) { const row = stmt.getAsObject(); results.push({ @@ -261,12 +506,14 @@ export class GraphStore { return results; } - getNodesByFile(sourceFile: string, limit = 500): GraphNode[] { + getNodesByFile(sourceFile: string, limit = 500, projectRoot?: string): GraphNode[] { const results: GraphNode[] = []; - const stmt = this.db.prepare( - "SELECT * FROM nodes WHERE source_file = ? LIMIT ?" - ); - stmt.bind([sourceFile, limit]); + const sql = projectRoot + ? "SELECT * FROM nodes WHERE source_file = ? AND project_root = ? LIMIT ?" + : "SELECT * FROM nodes WHERE source_file = ? LIMIT ?"; + const stmt = this.db.prepare(sql); + if (projectRoot) stmt.bind([sourceFile, projectRoot, limit]); + else stmt.bind([sourceFile, limit]); while (stmt.step()) { results.push(this.rowToNode(stmt.getAsObject())); } @@ -274,7 +521,7 @@ export class GraphStore { return results; } - getEdgesForNodes(nodeIds: string[]): GraphEdge[] { + getEdgesForNodes(nodeIds: string[], projectRoot?: string): GraphEdge[] { if (nodeIds.length === 0) return []; // Chunk to stay under SQLite's SQLITE_LIMIT_VARIABLE_NUMBER (999). // Each chunk binds chunk.length * 2 params (source IN + target IN). @@ -284,9 +531,11 @@ export class GraphStore { for (let i = 0; i < nodeIds.length; i += CHUNK) { const chunk = nodeIds.slice(i, i + CHUNK); const placeholders = chunk.map(() => "?").join(","); - const sql = `SELECT * FROM edges WHERE source IN (${placeholders}) OR target IN (${placeholders})`; + let sql = `SELECT * FROM edges WHERE source IN (${placeholders}) OR target IN (${placeholders})`; + if (projectRoot) sql = `SELECT * FROM edges WHERE (source IN (${placeholders}) OR target IN (${placeholders})) AND project_root = ?`; const stmt = this.db.prepare(sql); - stmt.bind([...chunk, ...chunk]); + if (projectRoot) stmt.bind([...chunk, ...chunk, projectRoot]); + else stmt.bind([...chunk, ...chunk]); while (stmt.step()) { const edge = this.rowToEdge(stmt.getAsObject()); const key = `${edge.source}|${edge.target}|${edge.relation}`; @@ -300,9 +549,11 @@ export class GraphStore { return results; } - getAllNodes(): GraphNode[] { + getAllNodes(projectRoot?: string): GraphNode[] { const results: GraphNode[] = []; - const stmt = this.db.prepare("SELECT * FROM nodes"); + const sql = projectRoot ? "SELECT * FROM nodes WHERE project_root = ?" : "SELECT * FROM nodes"; + const stmt = this.db.prepare(sql); + if (projectRoot) stmt.bind([projectRoot]); while (stmt.step()) { results.push(this.rowToNode(stmt.getAsObject())); } @@ -327,13 +578,22 @@ export class GraphStore { ); } - getStats(): GraphStats { - const nodeCount = (this.db.exec("SELECT COUNT(*) FROM nodes")[0]?.values[0]?.[0] as number) ?? 0; - const edgeCount = (this.db.exec("SELECT COUNT(*) FROM edges")[0]?.values[0]?.[0] as number) ?? 0; + getStats(projectRoot?: string): GraphStats { + // Helper to encode project stat keys + const encodeProjectKey = (p: string, k: string) => `project:${Buffer.from(p).toString("base64")}:${k}`; + + const nodeCount = projectRoot + ? (this.db.exec("SELECT COUNT(*) FROM nodes WHERE project_root = ?", [projectRoot])[0]?.values[0]?.[0] as number) ?? 0 + : (this.db.exec("SELECT COUNT(*) FROM nodes")[0]?.values[0]?.[0] as number) ?? 0; + const edgeCount = projectRoot + ? (this.db.exec("SELECT COUNT(*) FROM edges WHERE project_root = ?", [projectRoot])[0]?.values[0]?.[0] as number) ?? 0 + : (this.db.exec("SELECT COUNT(*) FROM edges")[0]?.values[0]?.[0] as number) ?? 0; + + const confSql = projectRoot + ? "SELECT confidence, COUNT(*) as c FROM edges WHERE project_root = ? GROUP BY confidence" + : "SELECT confidence, COUNT(*) as c FROM edges GROUP BY confidence"; + const confRows = projectRoot ? this.db.exec(confSql, [projectRoot]) : this.db.exec(confSql); - const confRows = this.db.exec( - "SELECT confidence, COUNT(*) as c FROM edges GROUP BY confidence" - ); const total = edgeCount || 1; const confMap: Record = {}; if (confRows[0]) { @@ -342,12 +602,12 @@ export class GraphStore { } } - const savedRow = this.db.exec( - "SELECT value FROM stats WHERE key = 'tokens_saved'" - ); - const lastMinedRow = this.db.exec( - "SELECT value FROM stats WHERE key = 'last_mined'" - ); + // Stats table: for project-scoped values we use namespaced keys + const savedKey = projectRoot ? encodeProjectKey(projectRoot, "tokens_saved") : "tokens_saved"; + const lastMinedKey = projectRoot ? encodeProjectKey(projectRoot, "last_mined") : "last_mined"; + + const savedRow = this.db.exec("SELECT value FROM stats WHERE key = ?", [savedKey]); + const lastMinedRow = this.db.exec("SELECT value FROM stats WHERE key = ?", [lastMinedKey]); return { nodes: nodeCount, @@ -409,14 +669,17 @@ export class GraphStore { * Get all cached provider results for a file. Returns only non-stale * entries (cached_at + ttl > now). */ - getCachedContext(filePath: string): CachedContext[] { + getCachedContext(filePath: string, projectRoot?: string): CachedContext[] { const now = Date.now(); const results: CachedContext[] = []; - const stmt = this.db.prepare( - `SELECT * FROM provider_cache - WHERE file_path = ? AND (cached_at + ttl * 1000) > ?` - ); - stmt.bind([filePath, now]); + const sql = projectRoot + ? `SELECT * FROM provider_cache + WHERE file_path = ? AND project_root = ? AND (cached_at + ttl * 1000) > ?` + : `SELECT * FROM provider_cache + WHERE file_path = ? AND (cached_at + ttl * 1000) > ?`; + const stmt = this.db.prepare(sql); + if (projectRoot) stmt.bind([filePath, projectRoot, now]); + else stmt.bind([filePath, now]); while (stmt.step()) { results.push(this.rowToCachedContext(stmt.getAsObject())); } @@ -430,14 +693,18 @@ export class GraphStore { */ getCachedContextForProvider( provider: string, - filePath: string + filePath: string, + projectRoot?: string ): CachedContext | null { const now = Date.now(); - const stmt = this.db.prepare( - `SELECT * FROM provider_cache - WHERE provider = ? AND file_path = ? AND (cached_at + ttl * 1000) > ?` - ); - stmt.bind([provider, filePath, now]); + const sql = projectRoot + ? `SELECT * FROM provider_cache + WHERE provider = ? AND file_path = ? AND project_root = ? AND (cached_at + ttl * 1000) > ?` + : `SELECT * FROM provider_cache + WHERE provider = ? AND file_path = ? AND (cached_at + ttl * 1000) > ?`; + const stmt = this.db.prepare(sql); + if (projectRoot) stmt.bind([provider, filePath, projectRoot, now]); + else stmt.bind([provider, filePath, now]); if (stmt.step()) { const row = stmt.getAsObject(); stmt.free(); @@ -455,14 +722,24 @@ export class GraphStore { filePath: string, content: string, ttl: number, - queryUsed = "" + queryUsed = "", + projectRoot?: string ): void { - this.db.run( - `INSERT OR REPLACE INTO provider_cache - (provider, file_path, content, query_used, cached_at, ttl) - VALUES (?, ?, ?, ?, ?, ?)`, - [provider, filePath, content, queryUsed, Date.now(), ttl] - ); + if (projectRoot) { + this.db.run( + `INSERT OR REPLACE INTO provider_cache + (provider, file_path, content, query_used, cached_at, ttl, project_root) + VALUES (?, ?, ?, ?, ?, ?, ?)`, + [provider, filePath, content, queryUsed, Date.now(), ttl, projectRoot] + ); + } else { + this.db.run( + `INSERT OR REPLACE INTO provider_cache + (provider, file_path, content, query_used, cached_at, ttl, project_root) + VALUES (?, ?, ?, ?, ?, ?, '')`, + [provider, filePath, content, queryUsed, Date.now(), ttl] + ); + } } /** @@ -473,18 +750,28 @@ export class GraphStore { provider: string, entries: ReadonlyArray<{ filePath: string; content: string }>, ttl: number, - queryUsed = "" + queryUsed = "", + projectRoot?: string ): void { if (entries.length === 0) return; this.db.run("BEGIN TRANSACTION"); try { for (const entry of entries) { - this.db.run( - `INSERT OR REPLACE INTO provider_cache - (provider, file_path, content, query_used, cached_at, ttl) - VALUES (?, ?, ?, ?, ?, ?)`, - [provider, entry.filePath, entry.content, queryUsed, Date.now(), ttl] - ); + if (projectRoot) { + this.db.run( + `INSERT OR REPLACE INTO provider_cache + (provider, file_path, content, query_used, cached_at, ttl, project_root) + VALUES (?, ?, ?, ?, ?, ?, ?)`, + [provider, entry.filePath, entry.content, queryUsed, Date.now(), ttl, projectRoot] + ); + } else { + this.db.run( + `INSERT OR REPLACE INTO provider_cache + (provider, file_path, content, query_used, cached_at, ttl, project_root) + VALUES (?, ?, ?, ?, ?, ?, '')`, + [provider, entry.filePath, entry.content, queryUsed, Date.now(), ttl] + ); + } } this.db.run("COMMIT"); this.save(); @@ -497,12 +784,19 @@ export class GraphStore { /** * Remove all stale cache entries. Called at SessionStart before warmup. */ - pruneStaleCache(): number { + pruneStaleCache(projectRoot?: string): number { const now = Date.now(); - this.db.run( - "DELETE FROM provider_cache WHERE (cached_at + ttl * 1000) <= ?", - [now] - ); + if (projectRoot) { + this.db.run( + "DELETE FROM provider_cache WHERE project_root = ? AND (cached_at + ttl * 1000) <= ?", + [projectRoot, now] + ); + } else { + this.db.run( + "DELETE FROM provider_cache WHERE (cached_at + ttl * 1000) <= ?", + [now] + ); + } const result = this.db.exec("SELECT changes()"); return (result[0]?.values[0]?.[0] as number) ?? 0; } @@ -511,24 +805,32 @@ export class GraphStore { * Remove all cache entries for a provider. Used when a provider is * disabled or its configuration changes. */ - clearProviderCache(provider: string): void { - this.db.run("DELETE FROM provider_cache WHERE provider = ?", [provider]); + clearProviderCache(provider: string, projectRoot?: string): void { + if (projectRoot) this.db.run("DELETE FROM provider_cache WHERE provider = ? AND project_root = ?", [provider, projectRoot]); + else this.db.run("DELETE FROM provider_cache WHERE provider = ?", [provider]); } /** * Get count of cached entries per provider. */ - getCacheStats(): Array<{ provider: string; count: number; stale: number }> { + getCacheStats(projectRoot?: string): Array<{ provider: string; count: number; stale: number }> { const now = Date.now(); const results: Array<{ provider: string; count: number; stale: number }> = []; const stmt = this.db.prepare( - `SELECT provider, + projectRoot + ? `SELECT provider, + COUNT(*) as total, + SUM(CASE WHEN (cached_at + ttl * 1000) <= ? THEN 1 ELSE 0 END) as stale + FROM provider_cache WHERE project_root = ? + GROUP BY provider` + : `SELECT provider, COUNT(*) as total, SUM(CASE WHEN (cached_at + ttl * 1000) <= ? THEN 1 ELSE 0 END) as stale - FROM provider_cache - GROUP BY provider` + FROM provider_cache + GROUP BY provider` ); - stmt.bind([now]); + if (projectRoot) stmt.bind([now, projectRoot]); + else stmt.bind([now]); while (stmt.step()) { const row = stmt.getAsObject(); results.push({ @@ -581,6 +883,12 @@ export class GraphStore { private rowToNode(row: Record): GraphNode { const validUntilRaw = row.valid_until; const invalidatedByRaw = row.invalidated_by_commit; + const meta = JSON.parse((row.metadata as string) || "{}"); + // Project scoping fields exposed in metadata for convenience. + (meta as Record).projectRoot = (row.project_root as string) ?? ""; + (meta as Record).projectBranch = (row.project_branch as string) ?? null; + (meta as Record).memoryScope = (row.memory_scope as string) ?? null; + return { id: row.id as string, label: row.label as string, @@ -591,7 +899,7 @@ export class GraphStore { confidenceScore: (row.confidence_score as number) ?? 1.0, lastVerified: (row.last_verified as number) ?? 0, queryCount: (row.query_count as number) ?? 0, - metadata: JSON.parse((row.metadata as string) || "{}"), + metadata: meta, validUntil: validUntilRaw === null || validUntilRaw === undefined ? undefined @@ -604,6 +912,9 @@ export class GraphStore { } private rowToEdge(row: Record): GraphEdge { + const meta = JSON.parse((row.metadata as string) || "{}"); + (meta as Record).projectRoot = (row.project_root as string) ?? ""; + return { source: row.source as string, target: row.target as string, @@ -613,7 +924,7 @@ export class GraphStore { sourceFile: (row.source_file as string) ?? "", sourceLocation: (row.source_location as string) ?? null, lastVerified: (row.last_verified as number) ?? 0, - metadata: JSON.parse((row.metadata as string) || "{}"), + metadata: meta, }; } } diff --git a/src/hooks.ts b/src/hooks.ts index 43eb968..9578845 100644 --- a/src/hooks.ts +++ b/src/hooks.ts @@ -40,7 +40,7 @@ if [ -z "$ENGRAM_BIN" ]; then fi if [ -d ".engram" ] && [ -f "$ENGRAM_BIN" ]; then - echo "[engram] Branch switched — rebuilding graph..." + echo "[engramx] Branch switched — rebuilding graph..." node "$ENGRAM_BIN" init . --quiet 2>/dev/null & fi ${HOOK_END} diff --git a/src/integrations/pi.ts b/src/integrations/pi.ts new file mode 100644 index 0000000..378e28f --- /dev/null +++ b/src/integrations/pi.ts @@ -0,0 +1,102 @@ +/** + * PI integration helpers — minimal client to call the local engram HTTP + * server. Designed for simple integrations from external tooling (like + * pi). This is intentionally tiny and dependency-free: it uses the + * platform fetch API (Node 20+) and reads the local token file at + * ~/.engram/http-server.token when a token is not supplied. + * + * Usage: + * import { postHook, postLearn, query } from "../integrations/pi.js"; + * await postHook({ hook_event_name: 'SessionStart', cwd: '/path', source: 'startup' }); + */ + +import { readFileSync, existsSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; + +// Node 20+ provides a global `fetch`, but TypeScript's lib config +// may not include DOM types in all environments. Declare the symbol +// loosely here so this helper compiles under the repo's strict tsconfig. +declare const fetch: any; + +const DEFAULT_PORT = 7337; +const TOKEN_MIN_LEN = 32; + +function tokenPath(): string { + return join(homedir(), ".engram", "http-server.token"); +} + +function readTokenFromFile(): string | null { + const p = tokenPath(); + if (!existsSync(p)) return null; + try { + const s = readFileSync(p, "utf-8").trim(); + return s.length >= TOKEN_MIN_LEN ? s : null; + } catch { + return null; + } +} + +async function fetchWithAuth( + path: string, + opts: { method?: string; body?: unknown; port?: number; token?: string } = {} +): Promise { + const port = opts.port ?? DEFAULT_PORT; + const token = opts.token ?? readTokenFromFile(); + if (!token) throw new Error("No engram HTTP server token found; start the server or pass a token."); + + const url = `http://127.0.0.1:${port}${path}`; + + const headers: Record = { + Authorization: `Bearer ${token}`, + }; + let body: string | undefined; + if (opts.body !== undefined) { + headers["Content-Type"] = "application/json"; + body = JSON.stringify(opts.body); + } + + const res = await fetch(url, { method: opts.method ?? "GET", headers, body }); + if (res.status === 204) return null; + if (res.status >= 400) { + const txt = await res.text(); + throw new Error(`engram HTTP ${res.status}: ${txt}`); + } + return await res.json(); +} + +export async function postHook( + payload: unknown, + port?: number, + token?: string +): Promise { + return await fetchWithAuth("/hook", { method: "POST", body: payload, port, token }); +} + +export async function postLearn( + content: string, + file?: string, + port?: number, + token?: string +): Promise { + return await fetchWithAuth("/learn", { method: "POST", body: { content, file }, port, token }); +} + +export async function query( + q: string, + port?: number, + token?: string +): Promise { + const portToUse = port ?? DEFAULT_PORT; + const tokenToUse = token ?? readTokenFromFile(); + if (!tokenToUse) throw new Error("No engram HTTP server token found; start the server or pass a token."); + const url = `http://127.0.0.1:${portToUse}/query?q=${encodeURIComponent(q)}`; + const res = await fetch(url, { headers: { Authorization: `Bearer ${tokenToUse}` } }); + if (res.status >= 400) { + const txt = await res.text(); + throw new Error(`engram HTTP ${res.status}: ${txt}`); + } + return await res.json(); +} + +export default { postHook, postLearn, query }; diff --git a/src/intelligence/token-tracker.ts b/src/intelligence/token-tracker.ts index 139d7b0..e30315e 100644 --- a/src/intelligence/token-tracker.ts +++ b/src/intelligence/token-tracker.ts @@ -3,6 +3,8 @@ * The viral screenshot generator. Hard numbers, not marketing claims. */ import type { GraphStore } from "../graph/store.js"; +import { projectStatKey } from "../core.js"; +import { resolve as resolvePath } from "node:path"; export interface SessionTokens { naiveTokens: number; @@ -22,29 +24,65 @@ export interface CumulativeStats { const COST_PER_MILLION_TOKENS = 3.0; +/** + * Record a single session's token numbers. If projectRoot is provided, + * stats are stored namespaced to that project so dashboards per-project + * show the correct totals. + */ export function recordSession( store: GraphStore, naiveTokens: number, - graphTokens: number + graphTokens: number, + projectRoot?: string ): SessionTokens { const saved = Math.max(0, naiveTokens - graphTokens); const savedPct = naiveTokens > 0 ? Math.round((saved / naiveTokens) * 1000) / 10 : 0; - const prev = getCumulativeStats(store); - store.setStat("total_sessions", String(prev.totalSessions + 1)); - store.setStat("total_naive_tokens", String(prev.totalNaiveTokens + naiveTokens)); - store.setStat("total_graph_tokens", String(prev.totalGraphTokens + graphTokens)); - store.setStat("total_tokens_saved", String(prev.totalSaved + saved)); + const prev = getCumulativeStats(store, projectRoot); + const root = projectRoot ? resolvePath(projectRoot) : undefined; + const sessKey = root ? projectStatKey(root, "total_sessions") : "total_sessions"; + const naiveKey = root ? projectStatKey(root, "total_naive_tokens") : "total_naive_tokens"; + const graphKey = root ? projectStatKey(root, "total_graph_tokens") : "total_graph_tokens"; + const savedKey = root ? projectStatKey(root, "total_tokens_saved") : "total_tokens_saved"; + + store.setStat(sessKey, String(prev.totalSessions + 1)); + store.setStat(naiveKey, String(prev.totalNaiveTokens + naiveTokens)); + store.setStat(graphKey, String(prev.totalGraphTokens + graphTokens)); + store.setStat(savedKey, String(prev.totalSaved + saved)); + + // Append to a session log for time-series visualization. Keep the + // last 1000 entries to avoid unbounded growth. + try { + const logKey = root ? projectStatKey(root, "session_log") : "session_log"; + const existing = store.getStat(logKey); + let arr: any[] = []; + if (existing) { + try { arr = JSON.parse(existing); if (!Array.isArray(arr)) arr = []; } catch { arr = []; } + } + arr.push({ ts: Date.now(), naiveTokens, graphTokens, saved, savedPct }); + if (arr.length > 1000) arr = arr.slice(-1000); + store.setStat(logKey, JSON.stringify(arr)); + } catch { + // best-effort only + } return { naiveTokens, graphTokens, saved, savedPct }; } -export function getCumulativeStats(store: GraphStore): CumulativeStats { - const totalSessions = store.getStatNum("total_sessions"); - const totalNaiveTokens = store.getStatNum("total_naive_tokens"); - const totalGraphTokens = store.getStatNum("total_graph_tokens"); - const totalSaved = store.getStatNum("total_tokens_saved"); +/** + * Read cumulative token stats. If projectRoot is provided, favors + * project-scoped keys (project::) and falls back to the + * global keys when absent. + */ +export function getCumulativeStats(store: GraphStore, projectRoot?: string): CumulativeStats { + const root = projectRoot ? resolvePath(projectRoot) : undefined; + const makeKey = (k: string) => (root ? projectStatKey(root, k) : k); + + const totalSessions = store.getStatNum(makeKey("total_sessions")); + const totalNaiveTokens = store.getStatNum(makeKey("total_naive_tokens")); + const totalGraphTokens = store.getStatNum(makeKey("total_graph_tokens")); + const totalSaved = store.getStatNum(makeKey("total_tokens_saved")); // avgReduction is a percentage (0-100). E.g., 88.4 means 88.4% fewer tokens // consumed when engram intercepts vs the naive full-read baseline. const avgReduction = totalNaiveTokens > 0 diff --git a/src/intercept/auto-memory.ts b/src/intercept/auto-memory.ts new file mode 100644 index 0000000..4d2a540 --- /dev/null +++ b/src/intercept/auto-memory.ts @@ -0,0 +1,173 @@ +import { existsSync, readFileSync } from "node:fs"; +import { basename, relative, extname } from "node:path"; +import { createHash } from "node:crypto"; +import { getStore, learn, projectStatKey } from "../core.js"; +import { readConfig } from "../tuner/config.js"; +import { isContentUnsafeForIntercept } from "./context.js"; + +// Conservative caps to avoid storing huge blobs +const MAX_CONTENT_CHARS = 16_000; +const MIN_CONTENT_CHARS = 20; + +function truncateContent(s: string): string { + if (!s) return ""; + return s.length > MAX_CONTENT_CHARS ? s.slice(0, MAX_CONTENT_CHARS) : s; +} + +function sha1Hex(s: string): string { + return createHash("sha1").update(s).digest("hex"); +} + +function isLikelyTextFile(filePath: string): boolean { + const ext = extname(filePath || "").toLowerCase(); + return ext === ".md" || ext === ".markdown" || ext === ".txt" || ext === ".rst" || ext === ".mdown" || ext === ".adoc" || ext === ".html" || ext === ".htm"; +} + +/** + * Perform auto-learn for a piece of textual content into the engram store. + * Dedupe by hashing the content per-scope+relPath so identical content + * isn't re-inserted repeatedly. Uses project's global DB via getStore. + * + * Non-throwing: any internal error is swallowed so callers can fire-and-forget. + */ +export async function performAutoLearnForContent( + projectRoot: string, + content: string, + relPath?: string, + sourceLabel?: string +): Promise { + try { + if (!projectRoot || !content || typeof content !== "string") return; + const cfg = readConfig(projectRoot); + + // Aggressive default: if config doesn't mention autoMemory fields, + // treat as enabled across all scopes. For safety, tests can override + // readConfig to set different defaults. + const autoEnabled = (cfg as any).autoMemoryEnabled !== undefined ? (cfg as any).autoMemoryEnabled : true; + if (!autoEnabled) return; + + const scopes: string[] = (cfg as any).autoMemoryScopes ?? ["project", "global", "entity"]; + + // Trim + sanity + const trimmed = truncateContent(content).trim(); + if (trimmed.length < MIN_CONTENT_CHARS) return; + + // If a relPath is given and the file looks unsafe (binary / secrets), skip + if (relPath && isContentUnsafeForIntercept(relPath)) return; + + // Simple entity-detection: first Markdown/H1 heading or first line + const firstLine = trimmed.split(/\r?\n/)[0] || ""; + const entityMatch = firstLine.match(/^#{1,3}\s+(.{2,200})/) ?? firstLine.match(/^(.{2,100})$/); + const entityName = entityMatch ? (entityMatch[1] || entityMatch[0]).trim() : null; + + // For each configured scope, dedupe by stat key and call learn + for (const scope of scopes) { + // entity scope: require an entity name to avoid noise + if (scope === "entity" && !entityName) continue; + + // Build a stat key to track last-ingested hash for this relPath+scope + const keySuffix = `auto_mem_hash:${scope}:${encodeURIComponent(relPath ?? "session")}`; + const statKey = projectStatKey(projectRoot, keySuffix); + + const hash = sha1Hex(trimmed + "|" + scope); + + // Check existing hash + let store; + try { + store = await getStore(projectRoot); + const existing = store.getStat(statKey); + if (existing === hash) { + // Nothing new to learn for this scope + continue; + } + } finally { + if (store) store.close(); + } + + // Build a sensible source label + const src = sourceLabel ?? `auto:${relPath ?? "session"}`; + + // For entity scope, craft a focused payload that emphasizes the entity + const payload = scope === "entity" && entityName + ? `entity: ${entityName}\n\n${trimmed}` + : trimmed; + + try { + await learn(projectRoot, payload, src, scope); + } catch { + // swallow learn errors + } + + // After successful learn attempt, record the new hash + try { + const s = await getStore(projectRoot); + try { + s.setStat(statKey, hash); + } finally { + s.close(); + } + } catch { + // swallow + } + } + } catch { + // swallow any unexpected error; this module must be silent + } +} + +/** + * Convenience wrapper used at SessionStart: learns the session brief + * across configured scopes. Fire-and-forget by callers. + */ +export async function onSessionStart(projectRoot: string, fullText: string): Promise { + try { + await performAutoLearnForContent(projectRoot, fullText, "session-start", "session-start"); + } catch { + // swallow + } +} + +/** + * Convenience wrapper for PostToolUse Read/Edit/Write handling. + * If the tool provided textual output, prefer that; otherwise attempt + * to read the file from disk and ingest. + */ +export async function onPostToolFile( + projectRoot: string, + cwd: string, + filePath: string | undefined, + toolResponse: unknown +): Promise { + try { + if (!filePath) return; + // Prefer toolResponse string if available + let maybeText: string | null = null; + if (typeof toolResponse === "string") maybeText = toolResponse; + else if (toolResponse && typeof toolResponse === "object") { + const t = toolResponse as Record; + if (typeof t.output === "string") maybeText = t.output; + else if (typeof t.stdout === "string") maybeText = t.stdout; + else if (typeof t.content === "string") maybeText = t.content; + } + + // If we don't have textual toolResponse, read file from disk + if (!maybeText) { + try { + const resolved = filePath.startsWith("/") ? filePath : require("node:path").resolve(cwd, filePath); + if (existsSync(resolved) && !isContentUnsafeForIntercept(resolved)) { + maybeText = readFileSync(resolved, "utf8"); + // use project-relative path for relPath + filePath = relative(projectRoot, resolved).replaceAll("\\", "/"); + } + } catch { + maybeText = null; + } + } + + if (maybeText && typeof maybeText === "string") { + await performAutoLearnForContent(projectRoot, maybeText, filePath, `auto:posttool`); + } + } catch { + // swallow + } +} diff --git a/src/intercept/dispatch.ts b/src/intercept/dispatch.ts index f8ad281..e51f128 100644 --- a/src/intercept/dispatch.ts +++ b/src/intercept/dispatch.ts @@ -52,7 +52,11 @@ import { handleCwdChanged, type CwdChangedHookPayload, } from "./handlers/cwd-changed.js"; +import { handleAssistantMessage, type AssistantMessagePayload } from "./handlers/assistant-message.js"; import { findProjectRoot, isValidCwd } from "./context.js"; +import { resolve as resolvePath } from "node:path"; +import { getStore } from "../core.js"; +import { recordSession } from "../intelligence/token-tracker.js"; import { logHookEvent } from "../intelligence/hook-log.js"; import { composeCostFields } from "../cost/instrument.js"; @@ -147,6 +151,11 @@ export async function dispatchHook( handleCwdChanged(payload as unknown as CwdChangedHookPayload) ); + case "AssistantMessage": + return runHandler(() => + handleAssistantMessage(payload as unknown as AssistantMessagePayload) + ); + default: return PASSTHROUGH; } @@ -208,18 +217,47 @@ async function dispatchPreToolUse( const projectRoot = findProjectRoot(cwd); if (projectRoot) { const decision = extractPreToolDecision(result); - const filePath = - typeof handlerPayload.tool_input?.file_path === "string" - ? handlerPayload.tool_input.file_path - : undefined; - const cost = composeCostFields(tool, filePath, result); + const rawFilePath = typeof handlerPayload.tool_input?.file_path === "string" + ? handlerPayload.tool_input.file_path + : undefined; + // Resolve a candidate absolute path for cost estimation. Many hook + // payloads provide project-relative paths; statSync requires an + // absolute path so we resolve relative to the hook's cwd. + const absForCost = (rawFilePath && typeof rawFilePath === "string") + ? (rawFilePath.startsWith("/") ? rawFilePath : resolvePath(cwd, rawFilePath)) + : undefined; + const cost = composeCostFields(tool, absForCost, result); logHookEvent(projectRoot, { event: "PreToolUse", tool, - path: filePath, + path: rawFilePath, decision, ...cost, }); + + // Record session-level token stats (best-effort, async). We use + // wouldHaveRead as the naive baseline and injected as the graph + // tokens consumed. Fire-and-forget so we don't block hook handling. + try { + (async () => { + try { + const naive = Number((cost && (cost as any).wouldHaveRead) || 0); + const graph = Number((cost && (cost as any).injected) || 0); + if (naive > 0 || graph > 0) { + const s = await getStore(projectRoot); + try { + recordSession(s, naive, graph, projectRoot); + } finally { + s.close(); + } + } + } catch { + /* swallow */ + } + })(); + } catch { + /* swallow */ + } } } } catch { diff --git a/src/intercept/handlers/assistant-message.ts b/src/intercept/handlers/assistant-message.ts new file mode 100644 index 0000000..e42d9fa --- /dev/null +++ b/src/intercept/handlers/assistant-message.ts @@ -0,0 +1,74 @@ +import { findProjectRoot, isValidCwd } from "../context.js"; +import { isHookDisabled, PASSTHROUGH, type HandlerResult } from "../safety.js"; +import { learn } from "../../core.js"; +import { logHookEvent } from "../../intelligence/hook-log.js"; + +export interface AssistantMessagePayload { + readonly hook_event_name: "AssistantMessage" | string; + readonly cwd: string; + readonly content?: string; + readonly summary?: string; + readonly memoryScope?: string; // project | global | entity + readonly sourceLabel?: string; +} + +/** + * Handle an AssistantMessage hook payload. This is an opt-in hook that + * external clients (the agent host) can call to tell engram that the + * assistant produced content that should be persisted as memory. + * + * Behaviour: + * - Validates cwd and project root + * - If a textual `content` or `summary` is provided, calls core.learn() + * asynchronously to persist conclusions/fragments into the graph. + * - Returns PASSTHROUGH (observer-only). Any internal error is swallowed. + */ +export async function handleAssistantMessage( + payload: AssistantMessagePayload +): Promise { + if (payload.hook_event_name !== "AssistantMessage") return PASSTHROUGH; + + try { + const cwd = payload.cwd; + if (!isValidCwd(cwd)) return PASSTHROUGH; + + const projectRoot = findProjectRoot(cwd); + if (projectRoot === null) return PASSTHROUGH; + + if (isHookDisabled(projectRoot)) return PASSTHROUGH; + + const raw = typeof payload.content === "string" && payload.content.trim().length > 0 + ? payload.content + : typeof payload.summary === "string" && payload.summary.trim().length > 0 + ? payload.summary + : null; + + if (!raw) return PASSTHROUGH; + + // Short-circuit tiny fragments — require at least a short sentence. + const trimmed = raw.trim(); + if (trimmed.length < 20) return PASSTHROUGH; + + const scope = (typeof payload.memoryScope === "string" && payload.memoryScope) ? payload.memoryScope : "project"; + const src = payload.sourceLabel ?? "assistant"; + + // Fire-and-forget: persist the assistant's content as a learned memory. + try { + void learn(projectRoot, trimmed, src, scope).catch(() => {}); + } catch { + /* swallow */ + } + + try { + // Dashboard-friendly lightweight event log. Do NOT include the + // learned content in the log to avoid accidental leakage. + logHookEvent(projectRoot, { event: "Learn", tool: "Assistant", path: null, tokensSaved: 0 }); + } catch { + // best-effort + } + } catch { + // swallow + } + + return PASSTHROUGH; +} diff --git a/src/intercept/handlers/cwd-changed.ts b/src/intercept/handlers/cwd-changed.ts index 398ab4a..2e519c4 100644 --- a/src/intercept/handlers/cwd-changed.ts +++ b/src/intercept/handlers/cwd-changed.ts @@ -65,7 +65,7 @@ export async function handleCwdChanged( const projectName = basename(resolve(projectRoot)); const lines: string[] = []; lines.push( - `[engram] Project switched to ${projectName} (${graphStats.nodes} nodes, ${graphStats.edges} edges)` + `[engramx] Project switched to ${projectName} (${graphStats.nodes} nodes, ${graphStats.edges} edges)` ); if (gods.length > 0) { lines.push("Core entities:"); diff --git a/src/intercept/handlers/mistake-guard.ts b/src/intercept/handlers/mistake-guard.ts index 265d0bd..745f964 100644 --- a/src/intercept/handlers/mistake-guard.ts +++ b/src/intercept/handlers/mistake-guard.ts @@ -8,7 +8,7 @@ * - `2` → strict: tool is denied with the warning as reason * * Only fires for PreToolUse events on Edit / Write / Bash. Read events - * already surface mistakes via the engram:mistakes context provider — + * already surface mistakes via the engramx:mistakes context provider — * duplicating the warning at tool-call time would be noise. * * Matching algorithm: @@ -107,10 +107,10 @@ export async function findMatchingMistakesAsync( // the miner could have stored either shape depending on how the // miner was invoked. Dedupe by node id. const candidates = [ - ...store.getNodesByFile(normalized), + ...store.getNodesByFile(normalized, 500, projectRoot), ...(normalized === target.filePath ? [] - : store.getNodesByFile(target.filePath)), + : store.getNodesByFile(target.filePath, 500, projectRoot)), ]; const seenIds = new Set(); for (const m of candidates) { @@ -129,7 +129,7 @@ export async function findMatchingMistakesAsync( // filtered to mistake-kind nodes. Bounded by project size; this // only runs when ENGRAM_MISTAKE_GUARD is explicitly enabled. const allMistakes = store - .getAllNodes() + .getAllNodes(projectRoot) .filter((n) => n.kind === "mistake") .filter((n) => n.validUntil === undefined || n.validUntil > now); diff --git a/src/intercept/handlers/post-tool.ts b/src/intercept/handlers/post-tool.ts index 418294b..896cc16 100644 --- a/src/intercept/handlers/post-tool.ts +++ b/src/intercept/handlers/post-tool.ts @@ -19,6 +19,7 @@ import { isHookDisabled, PASSTHROUGH, type HandlerResult } from "../safety.js"; import { logHookEvent } from "../../intelligence/hook-log.js"; import { handleBashPostTool, type FileOp } from "./bash-postool.js"; import { syncFile } from "../../watcher.js"; +import { onPostToolFile } from "../auto-memory.js"; export interface PostToolHookPayload { readonly hook_event_name: "PostToolUse" | string; @@ -129,6 +130,19 @@ export async function handlePostTool( /* silent */ }); } + + // Aggressive auto: ingest textual tool outputs / files into memory. + // Fire-and-forget — must not block the hook response. + try { + if ( + (toolName === "Read" || toolName === "Edit" || toolName === "Write") && + typeof filePath === "string" + ) { + void onPostToolFile(projectRoot, cwd, filePath, payload.tool_response).catch(() => {}); + } + } catch { + /* swallow */ + } } catch { // Observer errors are never surfaced. } diff --git a/src/intercept/handlers/pre-compact.ts b/src/intercept/handlers/pre-compact.ts index 06250a1..67399f2 100644 --- a/src/intercept/handlers/pre-compact.ts +++ b/src/intercept/handlers/pre-compact.ts @@ -50,7 +50,7 @@ function formatCompactBrief(args: { const lines: string[] = []; lines.push( - `[engram] Compaction survival — ${args.projectName} (${args.nodeCount} nodes, ${args.edgeCount} edges)` + `[engramx] Compaction survival — ${args.projectName} (${args.nodeCount} nodes, ${args.edgeCount} edges)` ); if (args.godNodes.length > 0) { diff --git a/src/intercept/handlers/read.ts b/src/intercept/handlers/read.ts index 07828fd..ce2f2f1 100644 --- a/src/intercept/handlers/read.ts +++ b/src/intercept/handlers/read.ts @@ -134,8 +134,8 @@ export async function handleRead( // Skip the structure provider — we already have the summary. Only // resolve the enrichment providers. const enrichmentProviders = [ - "engram:mistakes", - "engram:git", + "engramx:mistakes", + "engramx:git", "mempalace", "context7", "obsidian", @@ -168,8 +168,8 @@ async function buildNodeContext( ): Promise { const store = await getStore(projectRoot); try { - const nodes = store.getNodesByFile(relPath); - const edges = store.getEdgesForNodes(nodes.map((n) => n.id)); + const nodes = store.getNodesByFile(relPath, 500, projectRoot); + const edges = store.getEdgesForNodes(nodes.map((n) => n.id), projectRoot); // Extract import package names from import edges const imports = edges @@ -188,7 +188,7 @@ async function buildNodeContext( `tests/${baseName.split("/").pop()}`, ]; const hasTests = testPatterns.some((pattern) => - store.searchNodes(pattern, 1).length > 0 + store.searchNodes(pattern, 1, projectRoot).length > 0 ); // Get churn rate from git metadata if available diff --git a/src/intercept/handlers/session-start.ts b/src/intercept/handlers/session-start.ts index 3248b51..592aeb0 100644 --- a/src/intercept/handlers/session-start.ts +++ b/src/intercept/handlers/session-start.ts @@ -29,6 +29,7 @@ import { findProjectRoot, isValidCwd } from "../context.js"; import { isHookDisabled, PASSTHROUGH, type HandlerResult } from "../safety.js"; import { buildSessionContextResponse } from "../formatter.js"; import { warmAllProviders } from "../../providers/resolver.js"; +import { onSessionStart } from "../auto-memory.js"; export interface SessionStartHookPayload { readonly hook_event_name: "SessionStart" | string; @@ -108,7 +109,7 @@ function formatBrief(args: { ? describeAgo(Date.now() - args.stats.lastMined) : "unknown"; const branchStr = args.branch ? ` (branch: ${args.branch})` : ""; - lines.push(`[engram] Project brief for ${args.projectName}${branchStr}`); + lines.push(`[engramx] Project brief for ${args.projectName}${branchStr}`); lines.push( `Graph: ${args.stats.nodes} nodes, ${args.stats.edges} edges, ${args.stats.extractedPct}% extracted. Last mined: ${minedAgo}.` ); @@ -231,9 +232,11 @@ export async function handleSessionStart( ): Promise { if (payload.hook_event_name !== "SessionStart") return PASSTHROUGH; - // Skip resumed sessions — they already have prior context. + // Include resumed sessions too — inject the brief even on resume so + // session context is always available to the agent. onSessionStart is + // idempotent / dedupes, so repeated injections are safe. const source = payload.source ?? "startup"; - if (source === "resume") return PASSTHROUGH; + // fall through for 'resume' — inject on resume as well. // cwd must be a real absolute directory. Anything else causes // findProjectRoot to walk from the ambient process cwd, which could @@ -310,6 +313,14 @@ export async function handleSessionStart( // resolution with per-provider timeouts and graceful degradation. }); + // Aggressive auto: learn the session brief into memory across scopes. + // Fire-and-forget — must never block session start. + try { + void onSessionStart(projectRoot, fullText).catch(() => {}); + } catch { + /* swallow */ + } + return buildSessionContextResponse("SessionStart", fullText); } catch { // Any composition error → passthrough. Sessions must never fail diff --git a/src/intercept/handlers/user-prompt.ts b/src/intercept/handlers/user-prompt.ts index 6107a14..491f3f4 100644 --- a/src/intercept/handlers/user-prompt.ts +++ b/src/intercept/handlers/user-prompt.ts @@ -21,7 +21,7 @@ * any other persistent storage. Only metadata about the injection * decision (yes/no) may be logged. */ -import { query, computeKeywordIDF } from "../../core.js"; +import { query, computeKeywordIDF, learn } from "../../core.js"; import { findProjectRoot, isValidCwd } from "../context.js"; import { isHookDisabled, PASSTHROUGH, type HandlerResult } from "../safety.js"; import { buildSessionContextResponse } from "../formatter.js"; @@ -163,6 +163,19 @@ export async function handleUserPromptSubmit( if (isHookDisabled(projectRoot)) return PASSTHROUGH; + // Explicit memory requests: if the user explicitly asks the agent to + // remember/save/store this prompt, persist it to the project's memory. + // Fire-and-forget so we don't delay the hook response. We bypass the + // autoMemoryEnabled flag for explicit user requests. + try { + const explicitRemember = /^\s*(?:remember|save|store|dont forget|don't forget|note)\b/i; + if (explicitRemember.test(prompt)) { + void learn(projectRoot, prompt, "user:remember", "project").catch(() => {}); + } + } catch { + /* swallow */ + } + // v0.3.1: TF-IDF common-term filter. // // The gate logic has two layers: @@ -230,7 +243,7 @@ export async function handleUserPromptSubmit( // Format the injection. Include a short header so Claude knows this // is engram-provided context and not part of the user's message. - const header = `[engram] Pre-query context for this message (matched ${result.nodesFound} graph nodes):`; + const header = `[engramx] Pre-query context for this message (matched ${result.nodesFound} graph nodes):`; const text = `${header}\n\n${result.text}`; return buildSessionContextResponse("UserPromptSubmit", text); diff --git a/src/intercept/memory-md.ts b/src/intercept/memory-md.ts index 2f54db1..c07f3df 100644 --- a/src/intercept/memory-md.ts +++ b/src/intercept/memory-md.ts @@ -38,8 +38,8 @@ import { import { join } from "node:path"; /** The marker block engram owns in MEMORY.md. */ -const ENGRAM_MARKER_START = ""; -const ENGRAM_MARKER_END = ""; +const ENGRAM_MARKER_START = ""; +const ENGRAM_MARKER_END = ""; /** Sanity cap on MEMORY.md file size. */ const MAX_MEMORY_FILE_BYTES = 1_000_000; @@ -155,18 +155,19 @@ export function upsertEngramSection( * Never throws. Returns true on successful write, false on any kind * of failure (file too large, write error, etc.). */ -export function writeEngramSectionToMemoryMd( - projectRoot: string, - engramSection: string -): boolean { - if (!projectRoot || typeof projectRoot !== "string") return false; +export function writeEngramSectionToPath(memoryPath: string, engramSection: string): boolean { + // Allow global opt-out of automatic MEMORY.md writes. Set + // ENGRAM_DISABLE_AUTO_MEMORY_MD=1 in environments where writes must be + // suppressed. This keeps the code present for manual operations or for + // later re-enabling without removing the implementation. + if (process.env.ENGRAM_DISABLE_AUTO_MEMORY_MD === "1") return false; + + if (!memoryPath || typeof memoryPath !== "string") return false; if (engramSection.length > MAX_ENGRAM_SECTION_BYTES) { // Refuse to write a section that would blow through the budget return false; } - const memoryPath = join(projectRoot, "MEMORY.md"); - try { // Read existing content if any let existing = ""; @@ -191,4 +192,11 @@ export function writeEngramSectionToMemoryMd( } } +export function writeEngramSectionToMemoryMd(projectRoot: string, engramSection: string): boolean { + if (process.env.ENGRAM_DISABLE_AUTO_MEMORY_MD === "1") return false; + if (!projectRoot || typeof projectRoot !== "string") return false; + const memoryPath = join(projectRoot, "MEMORY.md"); + return writeEngramSectionToPath(memoryPath, engramSection); +} + export { ENGRAM_MARKER_START, ENGRAM_MARKER_END, MAX_ENGRAM_SECTION_BYTES }; diff --git a/src/miners/conclusions-miner.ts b/src/miners/conclusions-miner.ts new file mode 100644 index 0000000..47d7906 --- /dev/null +++ b/src/miners/conclusions-miner.ts @@ -0,0 +1,90 @@ +import type { GraphNode, GraphEdge } from "../graph/schema.js"; + +import { computeCanonicalId } from "../graph/canonical.js"; + +function makeId(...parts: string[]): string { + return parts + .filter(Boolean) + .join("_") + .replace(/[^a-zA-Z0-9]+/g, "_") + .replace(/^_|_$/g, "") + .toLowerCase() + .slice(0, 60); +} + +function firstSentence(s: string): string { + if (!s) return ""; + const m = s.trim().match(/([\s\S]{1,200}?[\.\!?])(\s|$)/); + if (m) return m[1].trim(); + const line = s.split(/\r?\n/)[0] || s; + return line.trim().slice(0, 200); +} + +export function generateConclusionNodes(text: string, sourceLabel = "session-summary") { + const nodes: GraphNode[] = []; + const edges: GraphEdge[] = []; + if (!text || typeof text !== "string") return { nodes, edges }; + + const now = Date.now(); + // Build a high-level conclusion node (pattern) using the first sentence. + const conclLabel = (firstSentence(text) || "Conclusion from session"); + // Use a canonical id for conclusions so repeated summaries map to the same + // conceptual node across sessions. + const conclId = computeCanonicalId(`Conclusion: ${conclLabel}`, "pattern", "project", null); + const conclusionNode: GraphNode = { + id: conclId, + label: `Conclusion: ${conclLabel}`, + kind: "pattern", + sourceFile: sourceLabel, + sourceLocation: null, + confidence: "INFERRED", + confidenceScore: 0.75, + lastVerified: now, + queryCount: 0, + metadata: { miner: "conclusion", sourceLabel }, + }; + nodes.push(conclusionNode); + + // Split into fragments (sentences), cap at 20 fragments. + const fragments = text + .split(/[\n\.\!\?]+/) // naive split on sentence terminators/newlines + .map((f) => f.trim()) + .filter((f) => f.length >= 20) + .slice(0, 20); + + for (let i = 0; i < fragments.length; i++) { + const frag = fragments[i]; + // Fragments are smaller and semantically transient; still use a + // canonical id so identical fragments across summaries dedupe. + const fragId = computeCanonicalId(frag.slice(0, 200), "concept", "project", null); + const node: GraphNode = { + id: fragId, + label: frag.length > 300 ? frag.slice(0, 297) + "..." : frag, + kind: "concept", + sourceFile: sourceLabel, + sourceLocation: null, + confidence: "INFERRED", + confidenceScore: 0.6, + lastVerified: now, + queryCount: 0, + metadata: { miner: "conclusion-fragment", index: i, sourceLabel }, + }; + nodes.push(node); + + // Link fragment -> conclusion as rationale_for + const edge: GraphEdge = { + source: node.id, + target: conclusionNode.id, + relation: "rationale_for", + confidence: "INFERRED", + confidenceScore: 0.6, + sourceFile: sourceLabel, + sourceLocation: null, + lastVerified: now, + metadata: { auto: true }, + }; + edges.push(edge); + } + + return { nodes, edges }; +} diff --git a/src/miners/git-miner.ts b/src/miners/git-miner.ts index ddd2d93..4a422be 100644 --- a/src/miners/git-miner.ts +++ b/src/miners/git-miner.ts @@ -22,7 +22,7 @@ function runGit(args: string[], cwd: string): string { } catch (err: unknown) { const msg = err instanceof Error ? err.message : String(err); if (msg.includes("TIMEOUT") || msg.includes("timed out")) { - console.error(`[engram] git command timed out: git ${args.join(" ")}`); + console.error(`[engramx] git command timed out: git ${args.join(" ")}`); } return ""; } diff --git a/src/miners/linking-helpers.ts b/src/miners/linking-helpers.ts new file mode 100644 index 0000000..1be22da --- /dev/null +++ b/src/miners/linking-helpers.ts @@ -0,0 +1,104 @@ +/** + * Heuristic helpers for extracting linking candidates from free text. + * Used by core.learn to create inferred edges between summary/conclusion + * fragments and existing graph nodes (files, identifiers, commands). + */ +import { basename } from "node:path"; + +function splitCamelCaseToken(tok: string): string[] { + // Split camelCase and PascalCase: fooBar -> [foo, Bar] + const s = tok.replace(/([a-z0-9])([A-Z])/g, "$1 $2"); + // Replace non-word separators with spaces then split + return s.split(/[^A-Za-z0-9]+/).filter(Boolean).map((t) => t.toLowerCase()); +} + +export function extractKeywords(text: string, minLength = 4, maxTokens = 60): string[] { + if (!text) return []; + const seen = new Set(); + + // 1) inline code/backtick spans are high-value — include them + for (const m of text.matchAll(/`([^`]+)`/g)) { + const t = String(m[1]).trim(); + for (const part of t.split(/[^A-Za-z0-9_]/).filter(Boolean)) { + if (part.length >= 2) { + for (const p of splitCamelCaseToken(part)) { + if (p.length >= minLength) seen.add(p); + } + } + } + } + + // 2) identifiers / words from plain text + for (const m of text.matchAll(/\b[A-Za-z_][A-Za-z0-9_]{2,}\b/g)) { + const tok = String(m[0]); + for (const p of splitCamelCaseToken(tok)) { + if (p.length >= minLength) seen.add(p); + } + if (seen.size >= maxTokens) break; + } + + // 3) bigrams of nouns (naive): two adjacent words of alpha chars + if (seen.size < maxTokens) { + for (const m of text.matchAll(/\b([A-Za-z]{3,})\s+([A-Za-z]{3,})\b/g)) { + const bigram = (m[1] + " " + m[2]).toLowerCase(); + if (bigram.length >= minLength && bigram.split(/\s+/).every((s) => s.length >= 3)) seen.add(bigram); + if (seen.size >= maxTokens) break; + } + } + + return Array.from(seen).slice(0, maxTokens); +} + +export function extractFilePaths(text: string): string[] { + if (!text) return []; + const out: string[] = []; + // Match explicit paths like src/foo/bar.ts or ./module/index.js or README.md + const pathRe = /(?:`([^`]+)`)|((?:\.\/?|\/?)[\w\-\.\/]+\.[a-z0-9]{1,6})/gi; + for (const m of text.matchAll(pathRe)) { + const candidate = (m[1] || m[2] || "").trim(); + if (!candidate) continue; + // Strip surrounding ./ prefixes + let c = candidate.replace(/^\.\//, ""); + // Normalize windows drive like C:/foo -> foo (best-effort) + c = c.replace(/^[A-Z]:\\/i, ""); + if (c.length > 0) out.push(c); + } + // Dedupe while preserving order + return Array.from(new Set(out)); +} + +export function extractCommands(text: string): string[] { + if (!text) return []; + const cmds = new Set(); + // Prefer backticked commands e.g. `git commit` or `npx tsx ...` + for (const m of text.matchAll(/`([^`]{2,200})`/g)) { + const t = String(m[1]).trim(); + if (t.length > 0) cmds.add(t); + } + + // Look for common command names followed by args, up to punctuation + const common = ["git", "npm", "npx", "yarn", "pnpm", "node", "engram", "pi", "make"]; + const re = new RegExp(`\\b(${common.join("|")})\\b[^\\n\\,;\.]{0,80}`, "gi"); + for (const m of text.matchAll(re)) { + const t = String(m[0]).trim(); + if (t.length > 0) cmds.add(t); + } + + return Array.from(cmds); +} + +export function basenameFromPath(p: string): string | null { + try { + return basename(p); + } catch { + return null; + } +} + +export function extractLinkCandidates(text: string) { + return { + keywords: extractKeywords(text), + filePaths: extractFilePaths(text), + commands: extractCommands(text), + }; +} diff --git a/src/miners/pdf-miner.ts b/src/miners/pdf-miner.ts new file mode 100644 index 0000000..b8973d0 --- /dev/null +++ b/src/miners/pdf-miner.ts @@ -0,0 +1,64 @@ +import { readFileSync } from "node:fs"; +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; + +const execFileP = promisify(execFile as any) as (file: string, args: string[]) => Promise<{ stdout: string; stderr: string }>; + +/** + * Try to extract text from a PDF. + * - Prefer `pdf-parse` if available via node_modules (async import). + * - Fallback to `pdftotext` binary if present (`pdftotext file -`). + * - Returns extracted text or null on failure. + */ +export async function extractTextFromPdf(filePath: string): Promise { + // 1) Try dynamic import of pdf-parse (npm module) + try { + // pdf-parse is a CommonJS module; dynamic import may return a namespace + // where the default export is the function we want. + // eslint-disable-next-line @typescript-eslint/no-var-requires + const m = await import("pdf-parse"); + const pdfParse = (m && (m.default ?? m)) as any; + if (typeof pdfParse === "function") { + const data = readFileSync(filePath); + try { + const r = await pdfParse(data as any); + if (r && typeof r.text === "string") return r.text; + } catch { + // fallthrough to other methods + } + } + } catch { + // dynamic import failed — try pdftotext + } + + // 2) Try pdftotext CLI + try { + const { stdout } = await execFileP("pdftotext", [filePath, "-"]); + if (stdout && stdout.length > 0) return stdout; + } catch { + // can't run pdftotext or it failed + } + + return null; +} + +/** + * Wrapper for extracting text from a file. For PDF paths, calls extractTextFromPdf. + * For other files, reads UTF-8 text. Returns null if extraction failed. + */ +export async function extractTextFromFile(filePath: string): Promise { + const ext = (filePath.split(".").pop() || "").toLowerCase(); + try { + if (ext === "pdf") { + return await extractTextFromPdf(filePath); + } + // Plain text fallback + try { + return readFileSync(filePath, "utf-8"); + } catch { + return null; + } + } catch { + return null; + } +} diff --git a/src/miners/session-miner.ts b/src/miners/session-miner.ts index 72b8fc2..10ca698 100644 --- a/src/miners/session-miner.ts +++ b/src/miners/session-miner.ts @@ -28,6 +28,8 @@ const PATTERN_PATTERNS = [ /(?:we use|our approach|the way we|standard is)\s+(.{10,60})/gi, ]; +import { computeCanonicalId } from "../graph/canonical.js"; + function makeId(...parts: string[]): string { return parts .filter(Boolean) @@ -47,8 +49,11 @@ function mineText(text: string, sourceFile: string): SessionMineResult { const normalized = label.trim().toLowerCase(); if (seenLabels.has(normalized) || normalized.length < 5) return; seenLabels.add(normalized); + // Compute canonical id for learned session-level concepts so duplicates + // across sessions and summaries map to the same entity. + const canonical = computeCanonicalId(label.trim(), kind, "project", null); nodes.push({ - id: makeId("session", kind, normalized), + id: canonical, label: label.trim(), kind, sourceFile, diff --git a/src/providers/ast.ts b/src/providers/ast.ts index c130020..1edc537 100644 --- a/src/providers/ast.ts +++ b/src/providers/ast.ts @@ -1,10 +1,10 @@ /** - * engram:ast provider — real AST-based symbol extraction via web-tree-sitter. + * engramx:ast provider — real AST-based symbol extraction via web-tree-sitter. * * Confidence 1.0 (exact, not estimated). Tier 1. 200ms timeout. * * Falls back gracefully to null on any error so the resolver can continue - * with the regex-based engram:structure provider. + * with the regex-based engramx:structure provider. */ import { readFileSync } from "node:fs"; import type { Node } from "web-tree-sitter"; @@ -161,7 +161,7 @@ function formatSymbols(symbols: Symbol[], tokenBudget: number): string { // ─── Provider ──────────────────────────────────────────────────────────────── export const astProvider: ContextProvider = { - name: "engram:ast", + name: "engramx:ast", label: "AST STRUCTURE", tier: 1, tokenBudget: 300, @@ -185,7 +185,7 @@ export const astProvider: ContextProvider = { if (symbols.length === 0) return null; return { - provider: "engram:ast", + provider: "engramx:ast", content: formatSymbols(symbols, this.tokenBudget), confidence: 1.0, cached: false, diff --git a/src/providers/engram-git.ts b/src/providers/engram-git.ts index a27bf4f..6c8be3a 100644 --- a/src/providers/engram-git.ts +++ b/src/providers/engram-git.ts @@ -1,5 +1,5 @@ /** - * engram:git provider — surfaces recent changes, churn rate, and last + * engramx:git provider — surfaces recent changes, churn rate, and last * author for a file from git history. * * Tier 1: internal, available when in a git repo (<100ms). @@ -10,7 +10,7 @@ import { execFileSync } from "node:child_process"; import type { ContextProvider, NodeContext, ProviderResult } from "./types.js"; export const gitProvider: ContextProvider = { - name: "engram:git", + name: "engramx:git", label: "CHANGES", tier: 1, tokenBudget: 50, @@ -58,7 +58,7 @@ export const gitProvider: ContextProvider = { ]; return { - provider: "engram:git", + provider: "engramx:git", content: parts.join("\n"), confidence: 0.9, cached: false, diff --git a/src/providers/engram-mistakes.ts b/src/providers/engram-mistakes.ts index c1363c6..0fccade 100644 --- a/src/providers/engram-mistakes.ts +++ b/src/providers/engram-mistakes.ts @@ -1,5 +1,5 @@ /** - * engram:mistakes provider — surfaces known issues and past failures + * engramx:mistakes provider — surfaces known issues and past failures * from the mistake memory system. * * Tier 1: internal, always available, no cache needed (<10ms). @@ -8,7 +8,7 @@ import { getStore } from "../core.js"; import type { ContextProvider, NodeContext, ProviderResult } from "./types.js"; export const mistakesProvider: ContextProvider = { - name: "engram:mistakes", + name: "engramx:mistakes", label: "KNOWN ISSUES", tier: 1, tokenBudget: 50, @@ -23,7 +23,7 @@ export const mistakesProvider: ContextProvider = { try { const now = Date.now(); const allMistakes = store - .getNodesByFile(filePath) + .getNodesByFile(filePath, 500, context.projectRoot) .filter((n) => n.kind === "mistake") // v3.0 bi-temporal: hide mistakes whose source code has been // refactored away (`validUntil` set by the git miner when it @@ -39,7 +39,7 @@ export const mistakesProvider: ContextProvider = { .join("\n"); return { - provider: "engram:mistakes", + provider: "engramx:mistakes", content: lines, confidence: 0.95, cached: false, diff --git a/src/providers/engram-structure.ts b/src/providers/engram-structure.ts index 675e7b0..73f62df 100644 --- a/src/providers/engram-structure.ts +++ b/src/providers/engram-structure.ts @@ -1,5 +1,5 @@ /** - * engram:structure provider — serves the structural summary from the + * engramx:structure provider — serves the structural summary from the * local graph. This is the existing renderFileStructure behavior * wrapped in the ContextProvider interface. * @@ -10,7 +10,7 @@ import { getStore } from "../core.js"; import type { ContextProvider, NodeContext, ProviderResult } from "./types.js"; export const structureProvider: ContextProvider = { - name: "engram:structure", + name: "engramx:structure", label: "STRUCTURE", tier: 1, tokenBudget: 250, @@ -27,7 +27,7 @@ export const structureProvider: ContextProvider = { if (!result || result.nodeCount === 0) return null; return { - provider: "engram:structure", + provider: "engramx:structure", content: result.text, confidence: result.avgConfidence, cached: false, diff --git a/src/providers/lsp.ts b/src/providers/lsp.ts index 50d50c9..7dc4e59 100644 --- a/src/providers/lsp.ts +++ b/src/providers/lsp.ts @@ -38,7 +38,7 @@ async function getConnection(): Promise { } export const lspProvider: ContextProvider = { - name: "engram:lsp", + name: "engramx:lsp", label: "LSP CONTEXT", tier: 1, tokenBudget: 100, @@ -68,7 +68,7 @@ export const lspProvider: ContextProvider = { : content; return { - provider: "engram:lsp", + provider: "engramx:lsp", content: truncated, confidence: 0.95, cached: false, diff --git a/src/providers/resolver.ts b/src/providers/resolver.ts index 5f070b9..7e7ec9a 100644 --- a/src/providers/resolver.ts +++ b/src/providers/resolver.ts @@ -65,7 +65,7 @@ async function getMcpProviders(): Promise { // One-line stderr warning per bad entry — don't crash, don't // noop-swallow. Users need to know their config didn't take. process.stderr.write( - `[engram] mcp-providers.json entry ${f.index}: ${f.reason}\n` + `[engramx] mcp-providers.json entry ${f.index}: ${f.reason}\n` ); } } @@ -171,11 +171,11 @@ export async function resolveRichPacket( if (results.length === 0) return null; - // When engram:ast succeeds (confidence 1.0), drop the lower-confidence - // engram:structure result to avoid duplicate structural content. - const hasAst = results.some((r) => r.provider === "engram:ast"); + // When engramx:ast succeeds (confidence 1.0), drop the lower-confidence + // engramx:structure result to avoid duplicate structural content. + const hasAst = results.some((r) => r.provider === "engramx:ast"); const deduped = hasAst - ? results.filter((r) => r.provider !== "engram:structure") + ? results.filter((r) => r.provider !== "engramx:structure") : results; // v3.0 — per-provider budget backstop. Providers are supposed to @@ -185,7 +185,7 @@ export async function resolveRichPacket( const budgetedResults = enforcePerProviderBudget(deduped, allProviders); // v3.0 — mistakes-boost reranking. Results that mention a label from - // the engram:mistakes provider get their confidence boosted (capped + // the engramx:mistakes provider get their confidence boosted (capped // at 1.0) so they sort up within their priority tier. This surfaces // structural context that touches known-broken areas ahead of other // structural context of equal priority. @@ -232,10 +232,10 @@ export async function resolveRichPacket( .map((r) => r.provider); // When called as enrichment (structure excluded), use a lighter header - const isEnrichment = enabledProviders && !enabledProviders.includes("engram:structure"); + const isEnrichment = enabledProviders && !enabledProviders.includes("engramx:structure"); const header = isEnrichment - ? `[engram] Additional context (${providerNames.length} providers, ~${totalTokens} tokens)` - : `[engram] Rich context for ${filePath} (${providerNames.length} providers, ~${totalTokens} tokens)`; + ? `[engramx] Additional context (${providerNames.length} providers, ~${totalTokens} tokens)` + : `[engramx] Rich context for ${filePath} (${providerNames.length} providers, ~${totalTokens} tokens)`; const text = `${header}\n\n${sections.join("\n\n")}`; return { @@ -371,7 +371,8 @@ export async function warmAllProviders( store.warmCache( result.provider, [...result.entries], - result.provider === "context7" ? 4 * 3600 : 3600 + result.provider === "context7" ? 4 * 3600 : 3600, + projectRoot ); store.save(); } finally { @@ -429,7 +430,7 @@ export function enforcePerProviderBudget( } /** - * Extract mistake labels from an engram:mistakes provider result. The + * Extract mistake labels from an engramx:mistakes provider result. The * provider formats mistakes as ` !