diff --git a/.changeset/workspace-folder-indexing.md b/.changeset/workspace-folder-indexing.md new file mode 100644 index 00000000..630c0c63 --- /dev/null +++ b/.changeset/workspace-folder-indexing.md @@ -0,0 +1,8 @@ +--- +"@melandlabs/workspace": minor +"@melandlabs/opencontext": minor +--- + +Add `@melandlabs/workspace` — a CLI for indexing an OKF / Markdown folder into SQLite and querying it with lexical, semantic, hybrid, and cross-file strategies. Reuses the existing `~/.opencontext/memory/store.db` schema and exposes `opencontext workspace update|search|list` subcommands. + +The workspace CLI defaults `EMBEDDING_PROVIDER=local` (`Xenova/all-MiniLM-L6-v2`, 384 dims) so demos and OKF review workflows run offline without `OPENROUTER_API_KEY`. Multi-format parsing covers `.md`, `.markdown`, `.txt`, `.pdf`, `.docx`, and `.pages`. Cross-file strategy walks `cites` edges extracted from Markdown links for BFS-style expansion across related files. diff --git a/.changeset/workspace-spreadsheet-support.md b/.changeset/workspace-spreadsheet-support.md new file mode 100644 index 00000000..392b6e6f --- /dev/null +++ b/.changeset/workspace-spreadsheet-support.md @@ -0,0 +1,10 @@ +--- +"@melandlabs/workspace": minor +"@melandlabs/opencontext": minor +--- + +Add Excel / Apple Numbers spreadsheet parsing to `@melandlabs/workspace`'s `parsers-adapter`. `.xlsx` and `.xls` are converted via SheetJS (`xlsx`) — one CSV block per sheet, prefixed with `# Sheet: ` so the chunker preserves sheet boundaries. `.numbers` files are first converted with macOS `textutil -convert xlsx`, then routed through the SheetJS path. + +The OKF walker now picks up `.xlsx`, `.xls`, and `.numbers` (macOS) alongside the existing `.md`, `.markdown`, `.txt`, `.pdf`, `.docx`, and `.pages` formats, and tags them with `resource_type: "spreadsheet"`. + +The 22-workspace demo now ships a 6-file fixture folder (`.md` × 3, `.pdf`, `.docx`, `.xlsx`) and asserts `filesScanned ≥ 6`, `filesAdded ≥ 6`, `listWorkspaceResources ≥ 6`, and that the re-run reports every file as `unchanged` under sha256 dedup. diff --git a/docs/tutorials/use-cases/09-workspace-folder-indexing.md b/docs/tutorials/use-cases/09-workspace-folder-indexing.md new file mode 100644 index 00000000..b8185bbb --- /dev/null +++ b/docs/tutorials/use-cases/09-workspace-folder-indexing.md @@ -0,0 +1,341 @@ +# Use Case: Workspace Folder Indexing + +## The Scenario + +You have a folder of related Markdown / PDF / DOCX files — say, three contracts, a memo, and a few public statutes you cite — and you want to **ask natural-language questions across the whole folder**: + +- "Which contracts cap liability at twelve months of fees?" +- "Pull every place in `contract-A.md` that references `law-2024.md`." +- "What does `memo.md` say about indemnification carve-outs?" + +Treating the folder as a **single project space** is what `@melandlabs/workspace` is for. It indexes the folder once, deduplicates by content hash, extracts Markdown links into a `cites` graph, and gives you a hybrid (lexical + semantic + cross-file) search command you can run from the terminal. + +This tutorial walks you through the CLI end-to-end. No HTTP server, no MCP client, no SDK call — just `pnpm opencontext workspace …` against a folder on disk. + +## What You'll Build + +A reusable `~/projects//wiki/` folder you can keep updating, plus three CLI commands you'll use day-to-day: + +1. **`opencontext workspace update`** — scan + chunk + embed the folder +2. **`opencontext workspace search`** — query the indexed folder (lexical / semantic / hybrid / cross-file) +3. **`opencontext workspace list`** — see what's been indexed and its current status + +## Concepts Demonstrated + +- **Versioned folder indexing** — re-running `update` skips unchanged files (`sha256` dedup) and creates a new version row only when content changes. +- **Cross-file reference edges** — Markdown links like `[Limitation of Liability](./b.md)` become `cites` edges in `workspace_reference_edges`. The `cross-file` strategy walks these edges to surface related context. +- **Multi-format parsing** — `.md`, `.markdown`, `.txt` are read raw; `.pdf`, `.docx`, `.pages` go through the `packages/rag` parser layer. +- **Local-first embeddings** — `EMBEDDING_PROVIDER` defaults to `local` for the `workspace` CLI, which runs `Xenova/all-MiniLM-L6-v2` (384 dims) on-device via `@huggingface/transformers`. No API key required. + +## Prerequisites + +- Completed [Getting Started](../00-getting-started.md) — `pnpm` workspace already installed. +- Node.js ≥ 22 (same as the rest of OpenContext). +- ~500 MB free disk for the local embedding model weights (downloaded once to `~/.cache/opencontext/local-embeddings`). + +## Quick Verification + +The fastest path: copy this four-line recipe and run it. The rest of the tutorial walks through what each step does and what to look for. + +```bash +# 1. Make a demo folder +mkdir -p /tmp/wf-demo/wiki +cat > /tmp/wf-demo/wiki/a.md <<'EOF' +--- +title: Limitation of Liability +type: contract +created: 2026-09-11 +--- +The aggregate liability of either party shall not exceed the fees paid in the +twelve (12) months preceding the claim. See [Indemnification](./b.md) for carve-outs. +EOF +cat > /tmp/wf-demo/wiki/b.md <<'EOF' +--- +title: Indemnification +type: contract +created: 2026-09-11 +--- +Neither party shall indemnify the other for indirect, consequential, or +punitive damages. References: [Limitation of Liability](./a.md). +EOF + +# 2. Index the folder (local embeddings by default — no OPENROUTER_API_KEY needed) +pnpm opencontext workspace update \ + --workspace-id demo --path /tmp/wf-demo/wiki --json + +# 3. Lexical search — works immediately, before embeddings finish +pnpm opencontext workspace search \ + --workspace-id demo --query "limitation" --strategy lexical + +# 4. Hybrid search — once the embedding fan-out has caught up +pnpm opencontext workspace search \ + --workspace-id demo --query "indemnification" --strategy hybrid --limit 3 + +# 5. Cross-file — expands hits along `cites` edges from the OKF graph +pnpm opencontext workspace search \ + --workspace-id demo --query "limitation" --strategy cross-file --hops 1 --json +``` + +If you see `2 hit(s)` and the `cites` edges in the JSON output, you're done. + +> **Note:** On macOS, the CLI may print `libc++abi: ... mutex lock failed: Invalid argument` and exit non-zero after writing the result. This is a known race between `@huggingface/transformers`' ONNX worker thread and `sqlite-vec`'s native destructor — the **output is complete and correct**, just wrap with `|| true` or `2>/dev/null` if it bothers your shell: +> +> ```bash +> pnpm opencontext workspace search ... || true +> ``` + +## Step-by-Step Walkthrough + +### Step 1: Pick a folder + +Any folder works. The indexer walks it recursively and picks a `resource_type` per file based on the extension: + +| Extension | `resource_type` | Indexed? | +| --- | --- | --- | +| `.md`, `.markdown` | `note` (or whatever the OKF front-matter `type:` says) | yes — pass-through; OKF front-matter is parsed; Markdown links become `cites` edges | +| `.txt` | `note` | yes — pass-through | +| `.pdf` | `document` | yes — goes through `@melandlabs/rag` parser (text + page-level metadata) | +| `.docx` | `document` | yes — same parser pipeline | +| `.pages` | `document` | yes — same parser pipeline (macOS only) | +| `.html`, `.htm` | — | mime is recognised but the OKF walker currently skips these | +| `.xlsx`, `.numbers` | — | **not indexed** — neither the parser layer nor the OKF walker supports spreadsheets | +| `.png`, `.jpg`, … | — | **not indexed** — raster files raise an explicit "unsupported" error | + +To mix formats, just drop them in the same folder. The walker picks them up in the same `update` run; the chunker and embedder don't care about the source extension once the text body is in hand: + +```bash +# macOS: spin up a real .pdf and .docx alongside the markdown fixtures +echo "Public Law: cap of liability is twelve months of fees for ordinary breach." \ + > /tmp/wf-demo/wiki/law-raw.txt +textutil -convert pdf -output /tmp/wf-demo/wiki/law.pdf /tmp/wf-demo/wiki/law-raw.txt +textutil -convert docx -output /tmp/wf-demo/wiki/law.docx /tmp/wf-demo/wiki/law-raw.txt +rm /tmp/wf-demo/wiki/law-raw.txt +``` + +Markdown files get extra love: the OKF front-matter (`--- title: ... type: ... ---`) is parsed, and `[text](./other.md)` Markdown links become `cites` edges pointing from the current file to the linked file. + +### Step 2: Index the folder + +```bash +pnpm opencontext workspace update \ + --workspace-id demo --path /tmp/wf-demo/wiki --json +``` + +Output (with `--json`): + +```json +{ + "ok": true, + "exit": 0, + "workspace_id": "demo", + "job_id": 1, + "status": "pending", + "files_scanned": 2, + "files_added": 2, + "files_modified": 0, + "files_unchanged": 0, + "files_deleted": 0 +} +``` + +What happened: + +1. The folder was scanned recursively — two Markdown files. +2. Each file got a `sha256` content hash. Both files were new → `files_added: 2`. +3. Chunks were written into `workspace_chunks` synchronously (FTS5 mirror tables updated immediately via triggers). +4. An embedding **job** was created; its `enqueueEmbedding` hook fires off the local ONNX embedder (`Xenova/all-MiniLM-L6-v2`, 384 dims). +5. The CLI waits for the embedding queue to drain by default (use `--no-await-embeddings` to return immediately). +6. A vec0 child table `workspace_chunks_vec_d384` was auto-created keyed by the provider's dimension. + +Re-run the same command and you'll see `files_unchanged: 2` — the `sha256` dedup means nothing gets re-chunked or re-embedded. + +### Step 3: Search (lexical first) + +```bash +pnpm opencontext workspace search \ + --workspace-id demo --query "limitation" --strategy lexical +``` + +Lexical search hits FTS5 the moment `update` returns (synchronous chunking). You don't need to wait for embeddings. + +``` +2 hit(s) (strategy=lexical) + - [note] a :: --- title: Limitation of Liability type: contract created: 2026-09-11 --- # Limitation of Liability The aggregate liability of either party… (score=1.000) + - [note] b :: …ce and wilful misconduct. References: [Limitation of Liability](./a.md). (score=1.000) +``` + +Both files match because `b.md` references "Limitation of Liability" in a Markdown link — FTS5 tokenizes the link text. + +### Step 4: Search (hybrid / cross-file) + +Once the embedding queue has caught up: + +```bash +pnpm opencontext workspace search \ + --workspace-id demo --query "limitation" --strategy cross-file --hops 1 --json +``` + +The `cross-file` strategy runs hybrid retrieval, then walks `cites` edges out of the top hits to expand the result set. The JSON payload shows the full hit structure: + +```json +{ + "query": "limitation", + "strategy": "cross-file", + "total": 2, + "hits": [ + { + "chunk_id": "demo:3:3:chunk:0:86599afa5dfd9428", + "resource_id": 3, + "version_id": 3, + "resource_type": "note", + "resource_title": "a", + "canonical_key": "a.md", + "snippet": "...# Limitation of Liability\n\nThe aggregate liability of either party…", + "matched_terms": ["limitation"], + "score": 0.01639344262295082, + "signals": { "lexical": 0.9999986666684445 }, + "reference_edges": [ + { "edge_type": "cites", "target_resource_id": 4 }, + { "edge_type": "cites", "target_resource_id": 3 } + ] + } + ] +} +``` + +What to look for: + +- `signals.lexical` — FTS5 BM25-derived similarity (1.0 = strong match). +- `signals.semantic` — appears once embeddings have been written. +- `reference_edges` — the `cites` graph extracted from Markdown links. Cross-file strategy uses these to expand beyond the top-N lexical hits. +- `--hops 2` walks two hops out from each seed hit (deeper expansion, more results). + +### Step 5: Inspect what's indexed + +```bash +pnpm opencontext workspace list --workspace-id demo --json +``` + +Each row includes: + +- `id` — stable resource id. +- `resource_type` / `canonical_key` / `title` — from the file extension and OKF front-matter. +- `index_status` — `pending` (just indexed), `partial` (some chunks embedded), `ready` (all chunks embedded), or `failed`. +- `current_version_id` — points into `workspace_resource_versions`; new versions are created only on content change. + +## Command Reference + +### `opencontext workspace update` + +```text +Required: + --workspace-id Workspace identifier + --path Path to the OKF folder (will be scanned recursively) + +Optional: + --user User / tenant id (default: "default") + --await-embeddings Wait for the in-process embedding queue to drain + before exiting (default: on) + --no-await-embeddings Return as soon as synchronous indexing completes + --drain-timeout-ms Upper bound on --await-embeddings (default 120000) + --json Emit JSON envelope + +Example: + opencontext workspace update --workspace-id demo --path ./wiki +``` + +### `opencontext workspace search` + +```text +Required: + --workspace-id Workspace identifier + --query Search query + +Strategy: + --strategy lexical | semantic | hybrid | cross-file + (default: hybrid) + --limit Top-N hits (default: 10, max: 50) + --threshold Semantic similarity threshold, 0..1 (default: 0.7) + --resource-type Comma-separated filter, e.g. "note,statute" + --hops <1|2> Cross-file BFS depth (cross-file strategy only) + +Output: + --json Emit JSON envelope with full WorkspaceSearchHit[] + +Example: + opencontext workspace search --workspace-id demo --query "limitation" \ + --strategy cross-file --hops 1 --json +``` + +### `opencontext workspace list` + +```text +Required: + --workspace-id Workspace identifier + +Filters: + --resource-type Filter by resource type + --index-status pending | partial | ready | failed + --limit Max rows (default: 50) + --offset Skip N rows (default: 0) + +Output: + --json Emit JSON envelope + +Example: + opencontext workspace list --workspace-id demo --index-status ready --limit 20 +``` + +## Storage + +Everything lives in the shared SQLite database at `~/.opencontext/memory/store.db` (override with `MEMORY_STORE_DB_PATH` or `--db-path`). The workspace package creates the following tables alongside the memory-store's existing schema: + +- `workspace_resources` — one row per indexed file (`(workspace_id, canonical_key)` is unique). +- `workspace_resource_versions` — append-only version chain keyed by `sha256`. +- `workspace_chunks` + `workspace_chunks_fts` — text chunks with FTS5 mirror (lexical search). +- `workspace_chunks_vec_d{N}` — vec0 ANN index, dimension-suffixed so each embedder model gets its own table. +- `workspace_reference_edges` — `cites` edges extracted from Markdown links (1- and 2-hop BFS in `cross-file` strategy). +- `workspace_jobs` — indexing job log (`pending` → `ready` / `partial` / `failed`). + +Inspect the tables directly: + +```bash +sqlite3 ~/.opencontext/memory/store.db \ + "SELECT id, workspace_id, source_resource_id, target_resource_id, edge_type + FROM workspace_reference_edges;" +``` + +## Switching Embedding Providers + +`EMBEDDING_PROVIDER` controls which embedder the workspace CLI uses. The workspace CLI defaults to `local`; the wider OpenContext system still defaults to `cloud`. The provider is resolved per-process; pick whichever matches your environment: + +| `EMBEDDING_PROVIDER` | Model | Dimensions | Requires | +| --- | --- | --- | --- | +| `local` *(default for `workspace`)* | `Xenova/all-MiniLM-L6-v2` | 384 | `@melandlabs/ai-rag` peer dep (already installed) | +| `cloud` | `text-embedding-3-small` via OpenRouter | 1536 | `OPENROUTER_API_KEY` | + +Local is fine for most legal / contract review work where documents stay under ~512 tokens per chunk and the corpus is in English. Switch to cloud if you need multilingual coverage or longer-context embeddings: + +```bash +EMBEDDING_PROVIDER=cloud pnpm opencontext workspace update --workspace-id demo --path ./wiki +``` + +## Known Quirks + +- **macOS SIGABRT noise on exit** — `libc++abi: ... mutex lock failed: Invalid argument` after a successful run is the `@huggingface/transformers` ONNX worker racing `sqlite-vec`'s native destructor. The output above the noise is correct. Wrap with `|| true` or `2>/dev/null` in shell scripts. +- **First-run model download** — the local embedder pulls `~50 MB` of ONNX weights on first use. Subsequent runs are instant. +- **Version churn** — every `update` creates a new row in `workspace_resource_versions` if content changed. The old chunks stay queryable via their `version_id`; cleanup is deferred to a future iteration. +- **No HTTP / MCP yet** — the surface today is CLI-only. Library consumers can `import { updateWorkspaceContext, searchWorkspaceContext, listWorkspaceResources }` from `@melandlabs/workspace`. + +## Next Steps + +- Combine `opencontext workspace search --json` with `jq` in shell pipelines: + + ```bash + pnpm opencontext workspace search --workspace-id demo --query "x" \ + --strategy cross-file --json 2>/dev/null \ + | jq -r '.hits[] | "\(.resource_title)\t\(.score)"' + ``` + +- Mount multiple workspaces for different projects — each gets its own `(workspace_id, canonical_key)` namespace in the same SQLite file. +- Re-index when files change — `update` is idempotent on unchanged content and only re-embeds the chunks that moved. diff --git a/examples/fixtures/workspace-wiki.README.md b/examples/fixtures/workspace-wiki.README.md new file mode 100644 index 00000000..6289a43f --- /dev/null +++ b/examples/fixtures/workspace-wiki.README.md @@ -0,0 +1,77 @@ +# workspace-wiki fixture + +A 6-file OKF folder used by `examples/src/simple/22-workspace.ts` to demo +`@melandlabs/workspace` end-to-end. The demo copies these files into a tmp +directory before each run so every run starts from a clean slate. + +## Files + +| File | Extension | `resource_type` | Role | +| --- | --- | --- | --- | +| `a.md` | `.md` | `note` (front-matter `type: contract`) | Limitation of Liability — 12-month cap. Cites `[./b.md]`. | +| `b.md` | `.md` | `note` (front-matter `type: contract`) | indemnification carve-out. Cites `[./a.md]`. | +| `law-clause.md` | `.md` | `note` (front-matter `type: statute`) | Public law clause — matches `a.md`'s 12-month cap. | +| `law-brief.pdf` | `.pdf` | `document` | Same public law clause, distributed as PDF (real binary). | +| `signed-addendum.docx` | `.docx` | `document` | Same clause as Word docx (real binary). | +| `contract-terms.xlsx` | `.xlsx` | `spreadsheet` | Liability cap matrix + indemnification carve-outs (real binary, 2 sheets). | + +The 3 markdown files form a cites graph (`a → b → a`) that the +cross-file search strategy walks. The PDF and DOCX are byte-identical +re-statements of the same law — they exercise the workspace parsers-adapter +multi-format path (`@melandlabs/rag`'s `parseFileToDocument` → +`PDFLoader` / `DocxLoader`). The XLSX exercises the SheetJS path — +`@langchain/community` ships no standalone Excel loader (only +`CSVLoader`), so `.xlsx` / `.xls` go straight to SheetJS. + +## Format coverage + +The fixture exists to demonstrate that `parsers-adapter.ts` actually works +across formats. Supported extensions (from +`packages/workspace/src/parsers-adapter.ts`): + +- `.md` / `.markdown` — pass-through, front-matter parsed +- `.txt` — pass-through +- `.pdf` — `parseFileToDocument` (requires `pdf-parse`, declared in + workspace deps) +- `.docx` — `parseFileToDocument` (requires `mammoth`, declared in + workspace deps) +- `.pages` — `parseFileToDocument` (macOS only, via + `AppleDocumentLoader`) +- `.xlsx` / `.xls` — SheetJS (`xlsx`) — each sheet converted to CSV + with `# Sheet: ` headers so the chunker can preserve boundaries +- `.numbers` — `textutil -convert xlsx` (macOS only) → SheetJS path + +Not supported (not in fixture): `.png`/`.jpg`/… (raster). + +## Generating the binary files + +```bash +# law-brief.pdf +pandoc law-clause.md -o law-brief.pdf + +# signed-addendum.docx +echo "Public Law — Cap of Liability …" > /tmp/law-raw.txt +textutil -convert docx -output signed-addendum.docx /tmp/law-raw.txt +rm /tmp/law-raw.txt + +# contract-terms.xlsx — generated by examples/scripts/regen-xlsx-fixture.cjs +# (a small node script that uses SheetJS to write the workbook; see +# the script for sheet contents). +``` + +This README is intentionally placed **outside** `workspace-wiki/` so the +walker's `SUPPORTED_EXTENSIONS` filter does not pick it up — only the 6 +indexable files above are scanned. + +## Demo assertions + +The demo runs lexical, semantic, hybrid, and cross-file queries against +this folder and asserts: + +- 6 resources appear in `listWorkspaceResources` +- lexical search for "limitation" returns ≥ 1 hit +- re-running `updateWorkspaceContext` reports every file as `unchanged` + (sha256 dedup) + +Used in: `examples/src/simple/22-workspace.ts` and the workspace tutorial +(`docs/tutorials/use-cases/09-workspace-folder-indexing.md`). diff --git a/examples/fixtures/workspace-wiki/a.md b/examples/fixtures/workspace-wiki/a.md new file mode 100644 index 00000000..23e0adff --- /dev/null +++ b/examples/fixtures/workspace-wiki/a.md @@ -0,0 +1,7 @@ +--- +title: Limitation of Liability +type: contract +created: 2026-09-11 +--- +The aggregate liability of either party shall not exceed the fees paid in the +twelve (12) months preceding the claim. See [Indemnification](./b.md) for carve-outs. diff --git a/examples/fixtures/workspace-wiki/b.md b/examples/fixtures/workspace-wiki/b.md new file mode 100644 index 00000000..b8d21369 --- /dev/null +++ b/examples/fixtures/workspace-wiki/b.md @@ -0,0 +1,7 @@ +--- +title: Indemnification +type: contract +created: 2026-09-11 +--- +Neither party shall indemnify the other for indirect, consequential, or +punitive damages. References: [Limitation of Liability](./a.md). diff --git a/examples/fixtures/workspace-wiki/contract-terms.xlsx b/examples/fixtures/workspace-wiki/contract-terms.xlsx new file mode 100644 index 00000000..be8ee7d9 Binary files /dev/null and b/examples/fixtures/workspace-wiki/contract-terms.xlsx differ diff --git a/examples/fixtures/workspace-wiki/law-brief.pdf b/examples/fixtures/workspace-wiki/law-brief.pdf new file mode 100644 index 00000000..d479f036 Binary files /dev/null and b/examples/fixtures/workspace-wiki/law-brief.pdf differ diff --git a/examples/fixtures/workspace-wiki/law-clause.md b/examples/fixtures/workspace-wiki/law-clause.md new file mode 100644 index 00000000..774a237a --- /dev/null +++ b/examples/fixtures/workspace-wiki/law-clause.md @@ -0,0 +1,6 @@ +--- +title: Public Law Clause — Cap of Liability +type: statute +created: 2026-09-11 +--- +Statutory cap of liability is twelve months of fees for ordinary breach. diff --git a/examples/fixtures/workspace-wiki/signed-addendum.docx b/examples/fixtures/workspace-wiki/signed-addendum.docx new file mode 100644 index 00000000..362e319d Binary files /dev/null and b/examples/fixtures/workspace-wiki/signed-addendum.docx differ diff --git a/examples/scripts/regen-xlsx-fixture.cjs b/examples/scripts/regen-xlsx-fixture.cjs new file mode 100644 index 00000000..8204e71a --- /dev/null +++ b/examples/scripts/regen-xlsx-fixture.cjs @@ -0,0 +1,41 @@ +/** + * Regenerate the workspace-wiki xlsx fixture. + * + * Run with: `node examples/scripts/regen-xlsx-fixture.cjs` + * + * Writes `examples/fixtures/workspace-wiki/contract-terms.xlsx` with two + * sheets that mirror the same cap-of-liability / indemnification theme + * carried by the markdown / PDF / DOCX fixtures, so any of them can be + * hit by the same lexical query ("limitation", "indemnification") and the + * chunker / embedder sees structurally similar content. + */ + +const path = require("node:path"); +const XLSX = require("/Users/timi/codes/opencontext/node_modules/xlsx"); + +const outPath = path.resolve(__dirname, "..", "fixtures", "workspace-wiki", "contract-terms.xlsx"); + +const workbook = XLSX.utils.book_new(); + +// Sheet 1: Liability cap matrix. +const liability = [ + ["Contract", "Party", "Cap (months of fees)", "Notes"], + ["Master Services Agreement", "Acme Corp", 12, "Standard 12-month cap, mirrors public law"], + ["Statement of Work #1", "Acme Corp", 12, "Inherits MSA cap"], + ["Vendor Agreement", "Globex Inc", 6, "Reduced cap for vendor services"], + ["NDA", "Initech", 0, "No liability cap (NDAs only)"], +]; +XLSX.utils.book_append_sheet(workbook, XLSX.utils.aoa_to_sheet(liability), "Liability Caps"); + +// Sheet 2: Indemnification carve-outs. +const indemnity = [ + ["Carve-out", "Applies to", "Trigger"], + ["Gross negligence", "All contracts", "Willful misconduct"], + ["IP infringement", "MSA / SOW", "Third-party patent claim"], + ["Confidentiality breach", "NDA only", "Material disclosure"], + ["Data breach", "MSA / SOW", "PII exposure > 100 records"], +]; +XLSX.utils.book_append_sheet(workbook, XLSX.utils.aoa_to_sheet(indemnity), "Indemnification"); + +XLSX.writeFile(workbook, outPath); +console.log(`wrote ${outPath}`); diff --git a/examples/src/index.ts b/examples/src/index.ts index 3e56135d..f567f400 100644 --- a/examples/src/index.ts +++ b/examples/src/index.ts @@ -48,6 +48,7 @@ import { withTmp } from "./_helpers.ts"; import demoVsa from "./simple/19-vsa.ts"; import demoOkf from "./simple/20-okf.ts"; import demoOkfServe from "./simple/21-okf-serve.ts"; +import demoWorkspace from "./simple/22-workspace.ts"; import demoHelloMemory from "./tutorials/00-hello-memory.ts"; import demoRememberExample from "./tutorials/01-remember-example.ts"; import demoRecallExample from "./tutorials/02-recall-example.ts"; @@ -161,6 +162,10 @@ const demos: Array<[string, () => Promise]> = [ ["demo: memory-store — Vector Symbolic Architecture (VSA) verb", demoVsa], ["demo: okf — OKF v0.2 (Open Knowledge Format) importer / exporter", demoOkf], ["demo: okf — serve (live + frozen viewer)", demoOkfServe], + [ + "demo: workspace — folder indexing + cross-file hybrid search (local embeddings by default)", + demoWorkspace, + ], ["demo: use-case — personal memory assistant", demoPersonalMemoryAssistant], ["demo: use-case — customer support agent", demoCustomerSupportAgent], ["demo: use-case — research knowledge tracker", demoResearchKnowledgeTracker], diff --git a/examples/src/simple/14-local-embedding.ts b/examples/src/simple/14-local-embedding.ts index 0015bf09..d16c4acf 100644 --- a/examples/src/simple/14-local-embedding.ts +++ b/examples/src/simple/14-local-embedding.ts @@ -53,8 +53,12 @@ export default async function demoLocalEmbedding() { try { factoryProvider = getConfiguredEmbeddingProvider(); } finally { - if (previousProvider === undefined) process.env.EMBEDDING_PROVIDER = undefined; - else process.env.EMBEDDING_PROVIDER = previousProvider; + if (previousProvider === undefined) { + // biome-ignore lint/performance/noDelete: env-reset pattern + delete process.env.EMBEDDING_PROVIDER; + } else { + process.env.EMBEDDING_PROVIDER = previousProvider; + } } check( "getConfiguredEmbeddingProvider() returns the local embedding provider when EMBEDDING_PROVIDER=local", diff --git a/examples/src/simple/22-workspace.ts b/examples/src/simple/22-workspace.ts new file mode 100644 index 00000000..fdb42bf3 --- /dev/null +++ b/examples/src/simple/22-workspace.ts @@ -0,0 +1,351 @@ +/** + * demo: @melandlabs/workspace — folder indexing + cross-file hybrid search. + * + * `@melandlabs/workspace` indexes a local folder of Markdown / TXT / PDF / + * DOCX / Pages files into a SQLite-backed knowledge space. It exposes + * three core APIs: + * + * - `updateWorkspaceContext` — scan + chunk + index a folder + * - `searchWorkspaceContext` — lexical / semantic / hybrid / cross-file + * - `listWorkspaceResources` — enumerate indexed resources + * + * `EMBEDDING_PROVIDER` defaults to `local` here (384-dim + * `Xenova/all-MiniLM-L6-v2`, runs in-process via `@huggingface/transformers`). + * No `OPENROUTER_API_KEY` needed. Override with `EMBEDDING_PROVIDER=cloud` + * for the 1536-dim OpenRouter path. + * + * First-run cost: ~30 MB of ONNX weights are pulled from HuggingFace and + * cached globally. Subsequent runs reuse the cache. + * + * This demo also exercises the workspace CLI surface end-to-end + * (`opencontext workspace update/search/list`) via the same JS API so + * the printed JSON envelope mirrors what the CLI prints. + */ + +import { copyFile, mkdir } from "node:fs/promises"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { randomUUID } from "node:crypto"; +import type { + RuntimeContext, + SearchWorkspaceContextResult, + UpdateWorkspaceContextResult, + WorkspaceSearchHit, +} from "@melandlabs/opencontext"; +import { info, makeCheckWithSkip, runSection, withTmp } from "../_helpers.ts"; + +const WORKSPACE_ID = "demo-workspace"; +const USER_ID = "demo-user"; +const SETTLE_MS = 15_000; // give the embedding queue time to drain on first run + +// Earlier demos in the suite may have poked `process.env.EMBEDDING_PROVIDER` +// (e.g. `examples/src/simple/14-local-embedding.ts` toggles it for the +// factory routing check). Force `local` here so the workspace demo never +// accidentally rides a stray env value into the cloud path. +const PREVIOUS_PROVIDER_ENV = process.env.EMBEDDING_PROVIDER; +process.env.EMBEDDING_PROVIDER = "local"; + +// Workspace value imports are pulled in dynamically so the smoke test +// (which pulls @melandlabs/opencontext from npm without the optional +// @melandlabs/workspace peer) can skip this demo cleanly instead of +// crashing the bootstrap with `ERR_MODULE_NOT_FOUND`. +type WorkspaceModule = { + closeSQLiteWorkspaceStore: () => Promise; + getSQLiteWorkspaceStore: (options: { dbPath: string }) => Promise; + listWorkspaceResources: ( + ctx: RuntimeContext, + store: unknown, + input: { workspace_id: string }, + ) => Promise<{ resources: unknown[]; total: number }>; + resolveWorkspaceDbPath: (override?: string) => string; + searchWorkspaceContext: ( + ctx: RuntimeContext, + store: unknown, + input: { + workspace_id: string; + query: string; + strategy: string; + options?: Record; + }, + ) => Promise; + updateWorkspaceContext: ( + ctx: RuntimeContext, + store: unknown, + input: { workspace_id: string; source: string; path: string }, + ) => Promise; +}; + +let _workspaceCache: WorkspaceModule | undefined; +async function loadWorkspace(): Promise { + if (_workspaceCache) return _workspaceCache; + try { + // @ts-expect-error -- optional workspace subpath; may be absent + // (e.g. in the npm-installed smoke test environment). + const mod = (await import("@melandlabs/workspace")) as WorkspaceModule; + _workspaceCache = mod; + return mod; + } catch { + return null; + } +} + +function makeRuntimeContext(): RuntimeContext { + return { + user_id: USER_ID, + employee_id: "demo-employee", + session_id: "demo-session", + request_id: randomUUID(), + }; +} + +// Fixture lives on disk at examples/fixtures/workspace-wiki/. The demo +// copies each file into a tmp directory before running so every run starts +// from a clean slate — the same fixtures also back the workspace tutorial. +// The set covers every format @melandlabs/workspace's parsers-adapter +// claims to support: .md / .pdf / .docx / .xlsx (and .pages / .numbers on macOS). +const FIXTURE_FILES = [ + "a.md", + "b.md", + "law-clause.md", + "law-brief.pdf", + "signed-addendum.docx", + "contract-terms.xlsx", +] as const; + +function resolveFixtureDir(): string { + // examples/src/simple/22-workspace.ts → examples/fixtures/workspace-wiki + const here = dirname(fileURLToPath(import.meta.url)); + return resolve(here, "..", "..", "fixtures", "workspace-wiki"); +} + +async function buildFixture(dir: string): Promise { + const wikiDir = join(dir, "wiki"); + await mkdir(wikiDir, { recursive: true }); + const sourceDir = resolveFixtureDir(); + for (const name of FIXTURE_FILES) { + await copyFile(join(sourceDir, name), join(wikiDir, name)); + } +} + +async function settleEmbeddings(): Promise { + // The embedding queue runs in the same process; sleeping is the + // simplest portable way to let it drain before we issue the hybrid / + // cross-file search. The lexical pass doesn't need to wait at all. + await new Promise((resolve) => setTimeout(resolve, SETTLE_MS)); +} + +function topHit(result: SearchWorkspaceContextResult): WorkspaceSearchHit | undefined { + return result.hits[0]; +} + +export default async function demoWorkspace() { + await runSection("demo: @melandlabs/workspace (folder indexing + cross-file search)", async () => { + const { check, skip } = makeCheckWithSkip("demo/workspace"); + + const ws = await loadWorkspace(); + if (!ws) { + skip( + "@melandlabs/workspace is installed", + "optional peer dep — npm-installed smoke test environment skips this demo", + ); + return; + } + + // Restore whatever was in EMBEDDING_PROVIDER before this demo so + // subsequent demos don't see `local` stuck on. + const restoreEnv = () => { + if (PREVIOUS_PROVIDER_ENV === undefined) { + // biome-ignore lint/performance/noDelete: env-reset pattern + delete process.env.EMBEDDING_PROVIDER; + } else { + process.env.EMBEDDING_PROVIDER = PREVIOUS_PROVIDER_ENV; + } + }; + + await withTmp("workspace", async (dir) => { + await buildFixture(dir); + + const dbPath = join(dir, "workspace.db"); + const store = await ws.getSQLiteWorkspaceStore({ dbPath }); + const ctx = makeRuntimeContext(); + + // 1. updateWorkspaceContext — synchronous chunking + async fan-out. + let update: UpdateWorkspaceContextResult; + try { + update = await ws.updateWorkspaceContext(ctx, store, { + workspace_id: WORKSPACE_ID, + source: "okf_folder", + path: join(dir, "wiki"), + }); + } catch (err) { + check("updateWorkspaceContext runs without throwing", false, (err as Error).message); + return; + } + + check( + "updateWorkspaceContext returns ok with files_scanned ≥ 6", + update.filesScanned >= 6, + `filesScanned=${update.filesScanned}, filesAdded=${update.filesAdded}`, + ); + check( + "updateWorkspaceContext flags the new files as added", + update.filesAdded >= 6, + `filesAdded=${update.filesAdded}`, + ); + check("updateWorkspaceContext returns a positive jobId", update.jobId > 0, `jobId=${update.jobId}`); + info( + "demo/workspace", + `updateWorkspaceContext → jobId=${update.jobId}, status=${update.status}, ` + + `scanned=${update.filesScanned}, added=${update.filesAdded}, modified=${update.filesModified}`, + ); + + // 2. listWorkspaceResources — every fixture file appears. + const listed = await ws.listWorkspaceResources(ctx, store, { + workspace_id: WORKSPACE_ID, + }); + check( + "listWorkspaceResources returns ≥ 6 resources for the fixture folder", + listed.resources.length >= 6, + `total=${listed.total}, resources=${listed.resources.map((r) => r.canonical_key).join(", ")}`, + ); + + // 3. Lexical search — works synchronously, FTS5 was filled in + // during the sync chunk phase. This is the only strategy + // that is guaranteed to work even before embeddings finish. + const lexicalResult = await ws.searchWorkspaceContext(ctx, store, { + workspace_id: WORKSPACE_ID, + query: "limitation", + strategy: "lexical", + options: { limit: 5 }, + }); + check( + "lexical search for 'limitation' returns ≥ 1 hit", + lexicalResult.hits.length >= 1, + `total=${lexicalResult.total}`, + ); + check( + "lexical search marks strategy as 'lexical'", + lexicalResult.strategy === "lexical", + lexicalResult.strategy, + ); + info( + "demo/workspace", + `lexical[0] → ${topHit(lexicalResult)?.resource_title} (score=${topHit(lexicalResult)?.score.toFixed(4)})`, + ); + + // 4. Give the embedding queue a moment to drain, then try + // semantic + cross-file. If the local model fails to load + // (no network, fresh CI runner) the queue ends in `partial` + // and we skip those checks instead of failing. + await settleEmbeddings(); + + let semanticResult: SearchWorkspaceContextResult | undefined; + try { + semanticResult = await ws.searchWorkspaceContext(ctx, store, { + workspace_id: WORKSPACE_ID, + query: "what is the cap on liability", + strategy: "semantic", + options: { limit: 3 }, + }); + } catch (err) { + const message = (err as Error).message.split("\n")[0]; + // biome-ignore lint/suspicious/noConsole: surface the failure on stderr for debugging + console.error(`[demo/workspace] semantic search failed: ${message}`); + skip("semantic search runs", "embedding provider failed: " + message); + } + if (semanticResult !== undefined) { + if (semanticResult.hits.length === 0) { + skip( + "semantic search returns hits once embeddings have been written", + "embedding queue drained with 0 rows (network or model load failure?)", + ); + } else { + check( + "semantic search returns ≥ 1 hit once embeddings are written", + semanticResult.hits.length >= 1, + `total=${semanticResult.total}`, + ); + check( + "semantic search marks strategy as 'semantic'", + semanticResult.strategy === "semantic", + semanticResult.strategy, + ); + info( + "demo/workspace", + `semantic[0] → ${topHit(semanticResult)?.resource_title} (score=${topHit(semanticResult)?.score.toFixed(4)})`, + ); + } + } + + // 5. Cross-file — hybrid + 1-hop BFS over cites edges. Expect + // hits from both `a.md` and the `b.md` it cites. + let crossFileResult: SearchWorkspaceContextResult | undefined; + try { + crossFileResult = await ws.searchWorkspaceContext(ctx, store, { + workspace_id: WORKSPACE_ID, + query: "limitation", + strategy: "cross-file", + options: { limit: 5, hops: 1 }, + }); + } catch (err) { + const message = (err as Error).message.split("\n")[0]; + // biome-ignore lint/suspicious/noConsole: surface the failure on stderr for debugging + console.error(`[demo/workspace] cross-file search failed: ${message}`); + skip("cross-file search runs", "embedding provider failed: " + message); + } + if (crossFileResult !== undefined) { + if (crossFileResult.hits.length === 0) { + skip( + "cross-file search returns hits", + "no hits — embeddings never finished (network or model load failure)", + ); + } else { + const titles = new Set(crossFileResult.hits.map((h) => h.resource_title)); + check( + "cross-file search returns ≥ 2 hits (seed + 1-hop cite neighbour)", + crossFileResult.hits.length >= 2, + `total=${crossFileResult.total}, titles=${[...titles].join(", ")}`, + ); + check( + "cross-file expansion surfaces both `a` and `b` via the cites edge", + titles.has("a") && titles.has("b"), + `titles=${[...titles].join(", ")}`, + ); + const hitsWithEdges = crossFileResult.hits.filter((h) => h.reference_edges.length > 0); + check( + "at least one cross-file hit carries reference_edges (cites graph)", + hitsWithEdges.length >= 1, + `${hitsWithEdges.length} hit(s) with edges`, + ); + info( + "demo/workspace", + `cross-file → ${crossFileResult.total} hits across ${titles.size} files: ${[...titles].join(", ")}`, + ); + } + } + + // 6. Re-run update — every file should now be `unchanged`. + const reUpdate = await ws.updateWorkspaceContext(ctx, store, { + workspace_id: WORKSPACE_ID, + source: "okf_folder", + path: join(dir, "wiki"), + }); + check( + "re-running update reports every file as `unchanged` (sha256 dedup)", + reUpdate.filesUnchanged >= 6 && reUpdate.filesAdded === 0, + `unchanged=${reUpdate.filesUnchanged}, added=${reUpdate.filesAdded}`, + ); + + // 7. resolveWorkspaceDbPath defaults to ~/.opencontext/memory/store.db + // and the override here is honoured. + check( + "resolveWorkspaceDbPath returns the path we passed in", + ws.resolveWorkspaceDbPath(dbPath) === dbPath, + ws.resolveWorkspaceDbPath(dbPath), + ); + + await ws.closeSQLiteWorkspaceStore().catch(() => undefined); + restoreEnv(); + }); + }); +} diff --git a/packages/opencontext/package.json b/packages/opencontext/package.json index 4b4a4a76..f690b83a 100644 --- a/packages/opencontext/package.json +++ b/packages/opencontext/package.json @@ -26,6 +26,7 @@ "@melandlabs/integrations": "^0.3.0", "@melandlabs/okf": "^0.3.3", "@melandlabs/security": "^0.3.0", + "@melandlabs/workspace": "^0.1.0", "@modelcontextprotocol/sdk": "^1.25.3", "abort-controller": "^3.0.0", "agentkeepalive": "^4.6.0", @@ -59,6 +60,7 @@ "@melandlabs/loop": "workspace:*", "@melandlabs/memory-store": "workspace:*", "@melandlabs/okf": "workspace:*", + "@melandlabs/workspace": "workspace:*", "@melandlabs/rag": "workspace:*", "@melandlabs/search": "workspace:*", "@melandlabs/shared": "workspace:*", diff --git a/packages/opencontext/src/cli/opencontext.ts b/packages/opencontext/src/cli/opencontext.ts index 0cc13bb8..775b6216 100644 --- a/packages/opencontext/src/cli/opencontext.ts +++ b/packages/opencontext/src/cli/opencontext.ts @@ -33,6 +33,10 @@ import { } from "@melandlabs/memory-store/cli-shared"; import { parseOkfArgs, printOkfHelp, startOkf } from "@melandlabs/okf"; import { closeSQLiteVsaStore } from "@melandlabs/sqlite"; +// Workspace CLI is shipped as an optional subpath import so a host +// that doesn't install `@melandlabs/workspace` still gets a usable +// `opencontext` CLI without crashing the bootstrap. +import { runWorkspaceCli } from "@melandlabs/workspace/cli"; import { startHttpServer, startMcpServer } from "../index.js"; import { parseAddArgs, runAdd } from "./add.js"; import { parseDeprecateArgs, runDeprecate } from "./deprecate.js"; @@ -296,6 +300,7 @@ Commands: stats Report counts from the active raw-message store doctor Run health checks against the local install okf OKF v0.2 (Open Knowledge Format) importer / exporter + workspace Versioned, cross-file folder knowledge Run "opencontext --help" for command-specific options. @@ -587,6 +592,11 @@ async function main(): Promise { process.exit(result.exit); } + if (head === "workspace" || head === "WORKSPACE") { + const exit = await runWorkspaceCli(argv.slice(1)); + process.exit(exit); + } + if (head === "--help" || head === "-h") { printTopHelp(); return; diff --git a/packages/opencontext/src/index.ts b/packages/opencontext/src/index.ts index 18c6ab6a..9e64190e 100644 --- a/packages/opencontext/src/index.ts +++ b/packages/opencontext/src/index.ts @@ -404,3 +404,51 @@ export type { WikiEdge, BuildGraphOptions, } from "@melandlabs/okf"; + +// ─── 14. Workspace — versioned, cross-file folder knowledge ── +// The single-package facade exposes the three core APIs (`updateWorkspaceContext`, +// `searchWorkspaceContext`, `listWorkspaceResources`) so consumers don't need a +// separate install. Storage is delegated to the shared SQLite DB the memory +// store already owns (`~/.opencontext/memory/store.db`); embedding fan-out is +// async and honours `EMBEDDING_PROVIDER=local|cloud` via the optional peer +// dep `@melandlabs/ai-rag` (local = `Xenova/all-MiniLM-L6-v2`, 384 dims; +// cloud = OpenRouter, 1536 dims). +// +// Surface is JS-API + CLI only; HTTP / MCP transport lives behind the +// `opencontext workspace …` subcommand (`@melandlabs/workspace/cli`). +export { + updateWorkspaceContext, + searchWorkspaceContext, + listWorkspaceResources, + getSQLiteWorkspaceStore, + closeSQLiteWorkspaceStore, + resolveWorkspaceDbPath, + getWorkspaceEmbeddingProvider, + workspaceEmbedQuery, + workspaceEmbedDocuments, + workspaceEmbeddingModelName, + workspaceEmbeddingDimensions, +} from "@melandlabs/workspace"; +export type { + RuntimeContext, + WorkspaceStorageKind, + WorkspaceEdgeType, + WorkspaceIndexStatus, + WorkspaceSearchStrategy, + WorkspaceResource, + WorkspaceResourceVersion, + WorkspaceChunk, + WorkspaceReferenceEdge, + WorkspaceJob, + WorkspaceSearchHit, + SearchWorkspaceContextOptions, + UpdateWorkspaceContextInput, + UpdateWorkspaceContextResult, + SearchWorkspaceContextInput, + SearchWorkspaceContextResult, + ListWorkspaceResourcesInput, + ListWorkspaceResourcesResult, + OkfFolderResource, + WorkspaceEmbeddingProvider, + WorkspaceEmbeddingProviderType, +} from "@melandlabs/workspace"; diff --git a/packages/workspace/package.json b/packages/workspace/package.json new file mode 100644 index 00000000..aecee703 --- /dev/null +++ b/packages/workspace/package.json @@ -0,0 +1,77 @@ +{ + "name": "@melandlabs/workspace", + "version": "0.1.0", + "type": "module", + "files": ["dist", "README.md", "LICENSE"], + "publishConfig": { + "access": "public" + }, + "main": "./dist/index.js", + "types": "./dist/index.d.ts", + "exports": { + ".": { + "types": "./dist/index.d.ts", + "import": "./dist/index.js" + }, + "./sqlite": { + "types": "./dist/sqlite.d.ts", + "import": "./dist/sqlite.js" + }, + "./cli": { + "types": "./dist/cli.d.ts", + "import": "./dist/cli.js" + } + }, + "repository": { + "type": "git", + "url": "https://github.com/melandlabs/opencontext.git", + "directory": "packages/project-context" + }, + "homepage": "https://github.com/melandlabs/opencontext/tree/main/packages/project-context", + "bugs": { + "url": "https://github.com/melandlabs/opencontext/issues" + }, + "license": "Apache-2.0", + "description": "OpenContext · workspace module. Provides project-level indexing, hybrid retrieval, and cross-file reference edges over a project workspace.", + "keywords": ["opencontext", "project-context", "hybrid-search", "rag"], + "dependencies": { + "@melandlabs/contracts": "workspace:*", + "@melandlabs/env-config": "workspace:*", + "@melandlabs/okf": "workspace:*", + "@melandlabs/rag": "workspace:*", + "@melandlabs/shared": "workspace:*", + "@melandlabs/sqlite": "workspace:*", + "@modelcontextprotocol/sdk": "^1.25.3", + "better-sqlite3": "^11.10.0", + "hono": "^4.6.14", + "mammoth": "^1.11.0", + "pdf-parse": "^2.0.0", + "sqlite-vec": "^0.1.9", + "xlsx": "^0.18.5", + "zod": "^4.3.6" + }, + "peerDependencies": { + "@melandlabs/ai-rag": "workspace:*" + }, + "peerDependenciesMeta": { + "@melandlabs/ai-rag": { + "optional": true + } + }, + "devDependencies": { + "@melandlabs/ai-rag": "workspace:*", + "@melandlabs/config": "workspace:*", + "@types/better-sqlite3": "^7.6.13", + "@types/node": "^22.13.10", + "typescript": "^5.6.3", + "vitest": "^4.1.10" + }, + "scripts": { + "build": "tsup", + "typecheck": "tsc --noEmit", + "lint": "biome check .", + "lint:fix": "biome check --write .", + "test": "vitest run", + "test:watch": "vitest" + } +} diff --git a/packages/workspace/src/api.ts b/packages/workspace/src/api.ts new file mode 100644 index 00000000..0c68df07 --- /dev/null +++ b/packages/workspace/src/api.ts @@ -0,0 +1,173 @@ +/** + * `@melandlabs/workspace` — three core APIs. + * + * - `updateWorkspaceContext` — scan + index an OKF folder, return job summary + * - `searchWorkspaceContext` — multi-strategy hybrid search (lexical / + * semantic / hybrid / cross-file) + * - `listWorkspaceResources` — enumerate indexed resources for a project + * + * The HTTP and MCP entry points (`src/http.ts`, `src/mcp.ts`) build the + * `RuntimeContext` from request headers / tool args and pass it through + * here. This layer is storage-agnostic: tests inject their own + * `SqliteWorkspaceStore` and a mock embedding function. + */ + +import { workspaceEmbedQuery } from "./embedding-provider"; +import { indexOkfFolder } from "./okf-backend"; +import { searchCrossFile } from "./search/cross-file"; +import { fuseHybridHits } from "./search/hybrid"; +import { searchLexical } from "./search/lexical"; +import { searchSemantic } from "./search/semantic"; +import type { SqliteWorkspaceStore } from "./sqlite"; +import type { + ListWorkspaceResourcesInput, + ListWorkspaceResourcesResult, + RuntimeContext, + SearchWorkspaceContextInput, + SearchWorkspaceContextResult, + UpdateWorkspaceContextInput, + UpdateWorkspaceContextResult, +} from "./types"; + +/** + * Only `okf_folder` is supported as a source today. Any other value + * triggers an explicit `unsupported_source` error so the HTTP / MCP + * layers (when re-introduced) can surface a 400 with a clear message. + */ +export async function updateWorkspaceContext( + ctx_rt: RuntimeContext, + store: SqliteWorkspaceStore, + input: UpdateWorkspaceContextInput, + hooks: { + enqueueEmbedding?: (input: { resource_id: number; version_id: number; jobId?: number }) => Promise; + } = {}, +): Promise { + if (!input.workspace_id) throw new Error("workspace_id is required"); + if (input.source !== "okf_folder") { + throw new Error(`unsupported source: ${String(input.source)} (only 'okf_folder' is supported)`); + } + if (!input.path) throw new Error("path is required"); + const enqueueEmbedding = hooks.enqueueEmbedding ?? (async () => {}); + return indexOkfFolder(store, { + workspace_id: input.workspace_id, + user_id: ctx_rt.user_id, + path: input.path, + enqueueEmbedding, + }); +} + +export interface SearchWorkspaceContextDeps { + /** Optional embedding override (tests inject deterministic mocks). */ + embed?: (text: string) => Promise; +} + +/** + * `searchWorkspaceContext` — the unified search entry. Strategy is one of + * - `lexical` — FTS5 only + * - `semantic` — sqlite-vec only (with lexical fallback when no embeddings yet) + * - `hybrid` — lexical + semantic RRF (default) + * - `cross-file`— hybrid + cites-edge BFS + */ +export async function searchWorkspaceContext( + ctx_rt: RuntimeContext, + store: SqliteWorkspaceStore, + input: SearchWorkspaceContextInput, + deps: SearchWorkspaceContextDeps = {}, +): Promise { + if (!input.workspace_id) throw new Error("workspace_id is required"); + if (!input.query) throw new Error("query is required"); + const embed = deps.embed ?? workspaceEmbedQuery; + const limit = Math.max(1, Math.min(50, Math.floor(input.options?.limit ?? 10))); + const strategy = input.strategy ?? "hybrid"; + const resourceTypes = input.options?.resource_types; + + if (strategy === "lexical") { + const hits = searchLexical(store, { + workspace_id: input.workspace_id, + user_id: ctx_rt.user_id, + query: input.query, + resource_types: resourceTypes, + limit, + }); + return { query: input.query, strategy, total: hits.length, hits }; + } + if (strategy === "semantic") { + const embedding = await embed(input.query); + const hits = searchSemantic(store, { + workspace_id: input.workspace_id, + user_id: ctx_rt.user_id, + queryEmbedding: embedding, + resource_types: resourceTypes, + limit, + threshold: input.options?.threshold ?? 0.7, + }); + // Fallback to lexical when no embeddings are written yet so the + // user still gets a hit during the indexing warm-up window. + const finalHits = + hits.length > 0 + ? hits + : searchLexical(store, { + workspace_id: input.workspace_id, + user_id: ctx_rt.user_id, + query: input.query, + resource_types: resourceTypes, + limit, + }); + return { query: input.query, strategy, total: finalHits.length, hits: finalHits }; + } + if (strategy === "cross-file") { + const hits = await searchCrossFile(store, { + workspace_id: input.workspace_id, + user_id: ctx_rt.user_id, + query: input.query, + resource_types: resourceTypes, + limit, + threshold: input.options?.threshold ?? 0.7, + hops: input.options?.hops ?? 1, + lexicalSearch: (params) => searchLexical(store, params), + semanticSearch: (params) => searchSemantic(store, params), + generateEmbedding: embed, + }); + return { query: input.query, strategy, total: hits.length, hits }; + } + // hybrid (default) + const candidateLimit = limit * 4; + const lexicalHits = searchLexical(store, { + workspace_id: input.workspace_id, + user_id: ctx_rt.user_id, + query: input.query, + resource_types: resourceTypes, + limit: candidateLimit, + }); + let semanticHits: WorkspaceSearchHitRef[] = []; + try { + const embedding = await embed(input.query); + semanticHits = searchSemantic(store, { + workspace_id: input.workspace_id, + user_id: ctx_rt.user_id, + queryEmbedding: embedding, + resource_types: resourceTypes, + limit: candidateLimit, + threshold: input.options?.threshold ?? 0.7, + }); + } catch (error) { + // Embedding failure (no API key, network) — degrade to lexical only. + semanticHits = []; + void error; + } + const hits = fuseHybridHits({ lexical: lexicalHits, semantic: semanticHits, limit }); + return { query: input.query, strategy, total: hits.length, hits }; +} + +// `WorkspaceSearchHitRef` alias kept inline so the import list stays tight. +type WorkspaceSearchHitRef = SearchWorkspaceContextResult["hits"][number]; + +export async function listWorkspaceResources( + ctx_rt: RuntimeContext, + store: SqliteWorkspaceStore, + input: ListWorkspaceResourcesInput, +): Promise { + if (!input.workspace_id) throw new Error("workspace_id is required"); + void ctx_rt; // Reserved for future per-user ACL filtering on the listing. + return store.listResources(input); +} diff --git a/packages/workspace/src/cli.ts b/packages/workspace/src/cli.ts new file mode 100644 index 00000000..5758b66c --- /dev/null +++ b/packages/workspace/src/cli.ts @@ -0,0 +1,565 @@ +#!/usr/bin/env node +/** + * `@melandlabs/workspace/cli` — `opencontext workspace …` subcommand. + * + * opencontext workspace update --workspace-id --path [--user ] + * opencontext workspace search --workspace-id --query + * [--strategy hybrid] [--limit 10] + * [--resource-type note,statute] [--json] + * opencontext workspace list --workspace-id + * [--resource-type ] [--index-status ready] + * [--limit 50] [--offset 0] [--json] + * + * The CLI is intentionally thin — it parses argv, builds a `RuntimeContext`, + * and delegates to the JS API in `./api`. The embedding queue is wired in + * here (not in `./api`) so a one-shot `update` invocation can await the + * async fan-out and report a final `ready` / `partial` status before exit. + */ + +import { randomUUID } from "node:crypto"; + +import { listWorkspaceResources, searchWorkspaceContext, updateWorkspaceContext } from "./api"; +import { createEmbeddingQueue } from "./embedding-queue"; +import { closeSQLiteWorkspaceStore, getSQLiteWorkspaceStore, resolveWorkspaceDbPath } from "./sqlite"; +import type { RuntimeContext, WorkspaceSearchStrategy } from "./types"; + +const logPrefix = "[opencontext/workspace]"; + +const STRATEGIES: WorkspaceSearchStrategy[] = ["lexical", "semantic", "hybrid", "cross-file"]; +const INDEX_STATUSES = ["pending", "partial", "ready", "failed"] as const; + +// Flags that don't take a value (booleans). Listed once here so the +// generic parseFlags() helper can skip its `next.startsWith("--")` +// guard for them. +const BOOLEAN_FLAGS = new Set(["--await-embeddings", "--no-await-embeddings", "--json"]); + +// ──────────────────────────────────────────────────────────────────────────── +// Argv helpers +// ──────────────────────────────────────────────────────────────────────────── + +class ArgvError extends Error { + constructor(message: string) { + super(`${logPrefix} ${message}`); + } +} + +interface ParseOptions { + positional?: (arg: string, state: T) => void; +} + +function takeValue(argv: string[], i: number, flag: string): { value: string; next: number } { + const next = argv[i + 1]; + if (next === undefined || next.startsWith("--")) { + throw new ArgvError(`${flag} requires a value`); + } + return { value: next, next: i + 2 }; +} + +function parseFlags(argv: string[], options: ParseOptions = {}): T { + const out = {} as T; + for (let i = 0; i < argv.length; i += 1) { + const arg = argv[i]; + if (arg === "--help" || arg === "-h") { + throw new ArgvError("--help requested"); + } + if (options.positional && !arg.startsWith("--")) { + options.positional(arg, out); + continue; + } + const eq = arg.indexOf("="); + const flag = eq >= 0 ? arg.slice(0, eq) : arg; + const inline = eq >= 0 ? arg.slice(eq + 1) : undefined; + const useInline = inline !== undefined; + const applyValue = (value: string) => { + const key = flag.slice(2).replace(/-([a-z])/g, (_, c: string) => c.toUpperCase()); + (out as Record)[key] = value; + }; + const applyBoolean = () => { + let key = flag.slice(2); + let value: boolean; + if (key.startsWith("no-")) { + key = key.slice(3); + value = false; + } else { + value = true; + } + key = key.replace(/-([a-z])/g, (_, c: string) => c.toUpperCase()); + (out as Record)[key] = value; + }; + if (BOOLEAN_FLAGS.has(flag)) { + applyBoolean(); + continue; + } + if (useInline) { + applyValue(inline); + } else { + const { value, next } = takeValue(argv, i, flag); + applyValue(value); + i = next - 1; + } + } + return out; +} + +// ──────────────────────────────────────────────────────────────────────────── +// Subcommand: update +// ──────────────────────────────────────────────────────────────────────────── + +interface UpdateArgs { + workspaceId?: string; + path?: string; + user?: string; + awaitEmbeddings?: boolean; + drainTimeoutMs?: number; + json?: boolean; +} + +function parseUpdateArgs(argv: string[]): UpdateArgs { + return parseFlags(argv, { + positional: (arg, state) => { + // Allow `opencontext workspace update --path …` + // as a convenience (positional first arg). + if (!state.workspaceId) { + state.workspaceId = arg; + return; + } + throw new ArgvError(`unexpected positional argument: ${arg}`); + }, + }); +} + +async function runUpdate(args: UpdateArgs): Promise { + const workspaceId = args.workspaceId; + const path = args.path; + if (!workspaceId) throw new ArgvError("--workspace-id is required"); + if (!path) throw new ArgvError("--path is required"); + const userId = args.user ?? "default"; + + const ctx_rt: RuntimeContext = { + user_id: userId, + request_id: randomUUID(), + }; + + const store = await getSQLiteWorkspaceStore(); + const queue = createEmbeddingQueue({ store }); + + const result = await updateWorkspaceContext( + ctx_rt, + store, + { workspace_id: workspaceId, source: "okf_folder", path }, + { enqueueEmbedding: (input) => queue.enqueue(input) }, + ); + + if (args.awaitEmbeddings) { + // Default behaviour: wait for the queue to drain so the CLI + // reports a final index_status before exiting. Users can opt + // out with --no-await-embeddings (handled by parseFlags: any + // `--no-X` is captured as `awaitEmbeddings: false`). + const timeoutMs = args.drainTimeoutMs ?? 120_000; + await Promise.race([queue.drain(), new Promise((resolve) => setTimeout(resolve, timeoutMs))]); + } + + const out = { + ok: true, + exit: 0, + workspace_id: workspaceId, + job_id: result.jobId, + status: result.status, + files_scanned: result.filesScanned, + files_added: result.filesAdded, + files_modified: result.filesModified, + files_unchanged: result.filesUnchanged, + files_deleted: result.filesDeleted, + }; + + if (args.json) { + process.stdout.write(`${JSON.stringify(out)}\n`); + } else { + const lines = [ + `scanned ${result.filesScanned} files`, + `added ${result.filesAdded}`, + `modified ${result.filesModified}`, + `unchanged ${result.filesUnchanged}`, + `deleted ${result.filesDeleted}`, + `job_id=${result.jobId} status=${result.status}`, + ]; + process.stdout.write(`${lines.join("\n")}\n`); + } + return 0; +} + +// ──────────────────────────────────────────────────────────────────────────── +// Subcommand: search +// ──────────────────────────────────────────────────────────────────────────── + +interface SearchArgs { + workspaceId?: string; + query?: string; + strategy?: string; + limit?: string; + threshold?: string; + resourceType?: string; + hops?: string; + json?: boolean; +} + +function parseSearchArgs(argv: string[]): SearchArgs { + return parseFlags(argv); +} + +async function runSearch(args: SearchArgs): Promise { + const workspaceId = args.workspaceId; + const query = args.query; + if (!workspaceId) throw new ArgvError("--workspace-id is required"); + if (!query) throw new ArgvError("--query is required"); + + const strategy = (args.strategy ?? "hybrid") as WorkspaceSearchStrategy; + if (!STRATEGIES.includes(strategy)) { + throw new ArgvError(`--strategy must be one of: ${STRATEGIES.join(", ")} (got "${args.strategy}")`); + } + + const limit = args.limit !== undefined ? Number.parseInt(args.limit, 10) : 10; + if (!Number.isInteger(limit) || limit <= 0) { + throw new ArgvError(`--limit must be a positive integer (got "${args.limit}")`); + } + const threshold = args.threshold !== undefined ? Number.parseFloat(args.threshold) : undefined; + if (threshold !== undefined && (Number.isNaN(threshold) || threshold < 0 || threshold > 1)) { + throw new ArgvError(`--threshold must be in [0, 1] (got "${args.threshold}")`); + } + const resourceTypes = args.resourceType + ? args.resourceType + .split(",") + .map((s) => s.trim()) + .filter(Boolean) + : undefined; + const hops = args.hops !== undefined ? Number.parseInt(args.hops, 10) : undefined; + if (hops !== undefined && hops !== 1 && hops !== 2) { + throw new ArgvError(`--hops must be 1 or 2 (got "${args.hops}")`); + } + + const ctx_rt: RuntimeContext = { + user_id: "default", + request_id: randomUUID(), + }; + + const store = await getSQLiteWorkspaceStore(); + const result = await searchWorkspaceContext(ctx_rt, store, { + workspace_id: workspaceId, + query, + strategy, + options: { + limit, + threshold, + resource_types: resourceTypes, + hops, + }, + }); + + if (args.json) { + process.stdout.write(`${JSON.stringify(result)}\n`); + } else { + process.stdout.write(`${result.total} hit(s) (strategy=${strategy})\n`); + for (const hit of result.hits) { + const snippet = hit.snippet.replace(/\s+/g, " ").slice(0, 160); + process.stdout.write( + ` - [${hit.resource_type}] ${hit.resource_title} :: ${snippet} (score=${hit.score.toFixed(3)})\n`, + ); + } + } + return 0; +} + +// ──────────────────────────────────────────────────────────────────────────── +// Subcommand: list +// ──────────────────────────────────────────────────────────────────────────── + +interface ListArgs { + workspaceId?: string; + resourceType?: string; + indexStatus?: string; + limit?: string; + offset?: string; + json?: boolean; +} + +function parseListArgs(argv: string[]): ListArgs { + return parseFlags(argv); +} + +async function runList(args: ListArgs): Promise { + const workspaceId = args.workspaceId; + if (!workspaceId) throw new ArgvError("--workspace-id is required"); + if (args.indexStatus && !(INDEX_STATUSES as readonly string[]).includes(args.indexStatus)) { + throw new ArgvError( + `--index-status must be one of: ${INDEX_STATUSES.join(", ")} (got "${args.indexStatus}")`, + ); + } + const limit = args.limit !== undefined ? Number.parseInt(args.limit, 10) : 50; + const offset = args.offset !== undefined ? Number.parseInt(args.offset, 10) : 0; + if (!Number.isInteger(limit) || limit <= 0) { + throw new ArgvError(`--limit must be a positive integer (got "${args.limit}")`); + } + if (!Number.isInteger(offset) || offset < 0) { + throw new ArgvError(`--offset must be a non-negative integer (got "${args.offset}")`); + } + + const ctx_rt: RuntimeContext = { + user_id: "default", + request_id: randomUUID(), + }; + + const store = await getSQLiteWorkspaceStore(); + const result = await listWorkspaceResources(ctx_rt, store, { + workspace_id: workspaceId, + resource_type: args.resourceType, + index_status: args.indexStatus as "pending" | "partial" | "ready" | "failed" | undefined, + limit, + offset, + }); + + if (args.json) { + process.stdout.write(`${JSON.stringify(result)}\n`); + } else { + process.stdout.write(`${result.total} resource(s)\n`); + for (const r of result.resources) { + process.stdout.write( + ` - [${r.index_status}] id=${r.id} ${r.resource_type} :: ${r.title} (${r.canonical_key})\n`, + ); + } + } + return 0; +} + +// ──────────────────────────────────────────────────────────────────────────── +// Help +// ──────────────────────────────────────────────────────────────────────────── + +function printHelp(): void { + const dbPath = resolveWorkspaceDbPath(); + process.stdout.write(`opencontext workspace — versioned, cross-file folder knowledge. + +Usage: + opencontext workspace [options] + +Subcommands: + update Index an OKF folder into the workspace SQLite store + search Run lexical | semantic | hybrid | cross-file search + list Enumerate indexed resources for a workspace + +Storage: + --db-path Override the SQLite path (default: ${dbPath}) + +Common: + --workspace-id Workspace identifier (required) + --user User / tenant id (default: "default") + --json Emit JSON envelope instead of a human line + +Examples: + opencontext workspace update --workspace-id proj-1 --path ~/notes/wiki + opencontext workspace update proj-1 --path ~/notes/wiki --user alice + opencontext workspace search --workspace-id proj-1 --query "limitation" + opencontext workspace search --workspace-id proj-1 --query "x" --strategy cross-file --hops 2 + opencontext workspace list --workspace-id proj-1 --index-status ready + +Run "opencontext workspace --help" for subcommand-specific options. +`); +} + +function printSubcommandHelp(sub: string): void { + switch (sub) { + case "update": + process.stdout.write(`opencontext workspace update — index an OKF folder. + +Required: + --workspace-id Workspace identifier + --path Path to the OKF folder (will be scanned recursively) + +Optional: + --user User / tenant id (default: "default") + --await-embeddings Wait for the in-process embedding queue to drain + before exiting (default: on) + --no-await-embeddings Return as soon as synchronous indexing completes + --drain-timeout-ms Upper bound on --await-embeddings (default 120000) + --json Emit JSON envelope + +Example: + opencontext workspace update --workspace-id proj-1 --path ./wiki +`); + return; + case "search": + process.stdout.write(`opencontext workspace search — multi-strategy hybrid search. + +Required: + --workspace-id Workspace identifier + --query Search query + +Strategy: + --strategy lexical | semantic | hybrid | cross-file + (default: hybrid) + --limit Top-N hits (default: 10, max: 50) + --threshold Semantic similarity threshold, 0..1 (default: 0.7) + --resource-type Comma-separated filter, e.g. "note,statute" + --hops <1|2> Cross-file BFS depth (cross-file strategy only) + +Output: + --json Emit JSON envelope with full WorkspaceSearchHit[] + +Example: + opencontext workspace search --workspace-id proj-1 --query "limitation" \\ + --strategy cross-file --hops 1 --json +`); + return; + case "list": + process.stdout.write(`opencontext workspace list — enumerate indexed resources. + +Required: + --workspace-id Workspace identifier + +Filters: + --resource-type Filter by resource type + --index-status pending | partial | ready | failed + --limit Max rows (default: 50) + --offset Skip N rows (default: 0) + +Output: + --json Emit JSON envelope + +Example: + opencontext workspace list --workspace-id proj-1 --index-status ready --limit 20 +`); + return; + default: + printHelp(); + } +} + +// ──────────────────────────────────────────────────────────────────────────── +// Entry +// ──────────────────────────────────────────────────────────────────────────── + +async function main(): Promise { + const argv = process.argv.slice(2); + const sub = argv[0]; + if (!sub || sub === "--help" || sub === "-h") { + printHelp(); + return; + } + if (sub === "help") { + printSubcommandHelp(argv[1] ?? ""); + return; + } + + const rest = argv.slice(1); + // Short-circuit `--help` / `-h` before parseFlags throws, so the + // subcommand-level help text shows up instead of an error. + if (rest.includes("--help") || rest.includes("-h")) { + printSubcommandHelp(sub); + process.exit(0); + } + + let exitCode = 0; + try { + switch (sub) { + case "update": + exitCode = await runUpdate(parseUpdateArgs(rest)); + break; + case "search": + exitCode = await runSearch(parseSearchArgs(rest)); + break; + case "list": + exitCode = await runList(parseListArgs(rest)); + break; + default: + throw new ArgvError(`unknown subcommand: ${sub}`); + } + } catch (error) { + if (error instanceof ArgvError) { + process.stderr.write(`${error.message}\n`); + // Force-exit on argv errors — see SIGABRT note below. + process.exit(2); + } + throw error; + } + // Force-exit instead of letting Node's event loop drain: the + // `@huggingface/transformers` ONNX worker thread spawned by the + // local embedder leaves a `sqlite-vec` mutex in a state that + // triggers `libc++abi: ... mutex lock failed: Invalid argument` + // (SIGABRT) during natural teardown on macOS. Output has already + // been written to stdout at this point, so a fast exit is safe. + process.exit(exitCode); +} + +// Guard against importing this file as a library (e.g. from +// `opencontext/cli/opencontext.ts`) — `runWorkspaceCli` is the +// programmatic entry, so only run the side-effecting `main()` when this +// file is the actual entry point. +const isDirectInvocation = + typeof process !== "undefined" && + Array.isArray(process.argv) && + process.argv[1] !== undefined && + import.meta.url === `file://${process.argv[1]}`; + +// Ignore SIGPIPE so a piped `head -n 1` doesn't show up as a fatal. +process.on("SIGPIPE", () => { + process.exit(0); +}); + +if (isDirectInvocation) { + main().catch((error: unknown) => { + const message = error instanceof Error ? (error.stack ?? error.message) : String(error); + process.stderr.write(`${logPrefix} fatal: ${message}\n`); + process.exit(1); + }); +} + +/** + * Programmatic entry point for hosts (e.g. `opencontext workspace …`) + * that want to delegate argv parsing + subcommand dispatch without + * re-importing the side-effecting `main()` above. Returns the desired + * process exit code; never throws. + */ +export async function runWorkspaceCli(argv: string[]): Promise { + const sub = argv[0]; + if (!sub || sub === "--help" || sub === "-h") { + printHelp(); + return 0; + } + if (sub === "help") { + printSubcommandHelp(argv[1] ?? ""); + return 0; + } + const rest = argv.slice(1); + if (rest.includes("--help") || rest.includes("-h")) { + printSubcommandHelp(sub); + return 0; + } + try { + switch (sub) { + case "update": + return await runUpdate(parseUpdateArgs(rest)); + case "search": + return await runSearch(parseSearchArgs(rest)); + case "list": + return await runList(parseListArgs(rest)); + default: + process.stderr.write(`${logPrefix} unknown subcommand: ${sub}\n`); + return 2; + } + } catch (error) { + if (error instanceof ArgvError) { + process.stderr.write(`${error.message}\n`); + return 2; + } + const message = error instanceof Error ? (error.stack ?? error.message) : String(error); + process.stderr.write(`${logPrefix} fatal: ${message}\n`); + return 1; + } finally { + // Background-close so sqlite-vec's native mutex destructor + // (occasionally SIGABRTs on certain macOS configs) doesn't + // poison the main process exit. The host program invokes us + // synchronously and then calls `process.exit` itself. + closeSQLiteWorkspaceStore().catch(() => { + // Best-effort cleanup; ignore secondary errors. + }); + } +} diff --git a/packages/workspace/src/embedding-provider.ts b/packages/workspace/src/embedding-provider.ts new file mode 100644 index 00000000..7669fe74 --- /dev/null +++ b/packages/workspace/src/embedding-provider.ts @@ -0,0 +1,133 @@ +/** + * `@melandlabs/workspace` — embedding provider adapter. + * + * The JS API and CLI want a single "give me a vector for this text" / + * "give me vectors for these N texts" surface that honours the + * `EMBEDDING_PROVIDER` env var (`local` | `cloud`). Without this adapter + * `generateEmbedding` / `generateEmbeddings` from `@melandlabs/rag` + * always hit OpenRouter, which makes `hybrid` / `cross-file` / `semantic` + * search unusable in a fully-offline setup. + * + * The provider lives in `@melandlabs/ai-rag`, which is an **optional** + * peer dep: hosts that only want lexical search don't need to install + * it. The adapter is therefore a lazy dynamic import — if the host never + * sets `EMBEDDING_PROVIDER=local` *and* never calls into the embedding + * path with `local` semantics, the import never resolves. + * + * Hosts that *do* set `EMBEDDING_PROVIDER=local` must install + * `@melandlabs/ai-rag` themselves; the workspace package surfaces a + * clear error in that case instead of crashing the module load. + */ + +export interface WorkspaceEmbeddingProvider { + embedQuery(text: string): Promise; + embedDocuments(texts: string[]): Promise; + getModelName(): string; + getDimensions(): number | undefined; +} + +export type WorkspaceEmbeddingProviderType = "cloud" | "local"; + +let _provider: WorkspaceEmbeddingProvider | undefined; + +function resolveProviderType(): WorkspaceEmbeddingProviderType { + // Default to `local` for the workspace CLI: folder indexing is + // positioned as an offline-first feature and most demos / OKF review + // workflows don't carry an OPENROUTER_API_KEY. Set + // `EMBEDDING_PROVIDER=cloud` to opt into the 1536-dim OpenRouter path. + const raw = (process.env.EMBEDDING_PROVIDER ?? "local").trim().toLowerCase(); + return raw === "local" ? "local" : "cloud"; +} + +/** + * Returns the singleton embedding provider, instantiating it on first + * call. Honours `EMBEDDING_PROVIDER` at construction time so the model + * weight download (local) or API key validation (cloud) only happens + * once per process. + * + * Throws if `EMBEDDING_PROVIDER=local` is set but `@melandlabs/ai-rag` + * isn't installed. + */ +export async function getWorkspaceEmbeddingProvider(): Promise { + if (_provider) return _provider; + const providerType = resolveProviderType(); + if (providerType === "local") { + // Dynamic import keeps `@melandlabs/ai-rag` out of the static + // dependency graph for hosts that don't need local embeddings. + const specifier = "@melandlabs/ai-rag/embedding-provider"; + let mod: typeof import("@melandlabs/ai-rag/embedding-provider"); + try { + mod = await import(specifier); + } catch (error) { + throw new Error( + `EMBEDDING_PROVIDER=local but "${specifier}" could not be resolved. Install @melandlabs/ai-rag to enable local ONNX embeddings (Xenova/all-MiniLM-L6-v2, 384 dims by default). Underlying error: ${error instanceof Error ? error.message : String(error)}`, + ); + } + const provider = mod.getConfiguredEmbeddingProvider({ providerType: "local" }); + if (!provider) { + throw new Error("EMBEDDING_PROVIDER=local but getConfiguredEmbeddingProvider returned no provider"); + } + _provider = provider; + } else { + const specifier = "@melandlabs/ai-rag/embedding-provider"; + let mod: typeof import("@melandlabs/ai-rag/embedding-provider"); + try { + mod = await import(specifier); + } catch (error) { + throw new Error( + `Cloud embedding provider requested but "${specifier}" could not be resolved. Install @melandlabs/ai-rag to enable OpenRouter-backed embeddings (requires OPENROUTER_API_KEY). Underlying error: ${error instanceof Error ? error.message : String(error)}`, + ); + } + const provider = mod.getConfiguredEmbeddingProvider({ providerType: "cloud" }); + if (!provider) { + throw new Error( + "Cloud embedding provider requested but getConfiguredEmbeddingProvider returned no provider", + ); + } + _provider = provider; + } + return _provider; +} + +/** + * Convenience: embed a single text and return just the vector. + * Used by `searchWorkspaceContext` for the query side. + */ +export async function workspaceEmbedQuery(text: string): Promise { + const provider = await getWorkspaceEmbeddingProvider(); + return provider.embedQuery(text); +} + +/** + * Convenience: embed a batch of texts and return parallel vectors. + * Used by `createEmbeddingQueue` for the chunk fan-out. + */ +export async function workspaceEmbedDocuments(texts: string[]): Promise { + if (texts.length === 0) return []; + const provider = await getWorkspaceEmbeddingProvider(); + return provider.embedDocuments(texts); +} + +/** + * Read the model name of the active provider without instantiating it. + * Used for the `embedding_model` column on `workspace_chunks`. + */ +export async function workspaceEmbeddingModelName(): Promise { + const provider = await getWorkspaceEmbeddingProvider(); + return provider.getModelName(); +} + +/** + * Read the dimension of the active provider without instantiating it + * (the cloud provider only knows its dim after the first embed call). + * Returns `undefined` for the cloud provider pre-warmup. + */ +export async function workspaceEmbeddingDimensions(): Promise { + const provider = await getWorkspaceEmbeddingProvider(); + return provider.getDimensions(); +} + +/** Test-only: drop the singleton so the next call re-instantiates. */ +export function __resetWorkspaceEmbeddingProviderForTests(): void { + _provider = undefined; +} diff --git a/packages/workspace/src/embedding-queue.ts b/packages/workspace/src/embedding-queue.ts new file mode 100644 index 00000000..f220adaa --- /dev/null +++ b/packages/workspace/src/embedding-queue.ts @@ -0,0 +1,99 @@ +import { workspaceEmbedDocuments, workspaceEmbeddingModelName } from "./embedding-provider"; +import type { SqliteWorkspaceStore } from "./sqlite"; + +const DEFAULT_BATCH_SIZE = 100; +// 384 = Xenova/all-MiniLM-L6-v2 (local). Cloud picks 1536 dynamically. +const DEFAULT_DIMENSIONS = 384; + +export interface EmbeddingQueueDeps { + store: SqliteWorkspaceStore; + batchSize?: number; + dimensions?: number; +} + +export interface EmbeddingQueue { + /** + * Enqueue an embedding job for a specific `(resource_id, version_id)` pair. + * Resolves once the job has been scheduled into the serial queue; it + * does NOT wait for embedding completion. + */ + enqueue(input: { resource_id: number; version_id: number; jobId?: number }): Promise; + /** Test-only: drain the serial queue and wait for all pending jobs. */ + drain(): Promise; +} + +class DefaultEmbeddingQueue implements EmbeddingQueue { + private readonly store: SqliteWorkspaceStore; + private readonly batchSize: number; + private readonly dimensions: number; + private queue: Promise = Promise.resolve(); + + constructor(deps: EmbeddingQueueDeps) { + this.store = deps.store; + this.batchSize = deps.batchSize ?? DEFAULT_BATCH_SIZE; + this.dimensions = deps.dimensions ?? DEFAULT_DIMENSIONS; + } + + enqueue(input: { resource_id: number; version_id: number; jobId?: number }): Promise { + const next = this.queue.then(async () => { + await this.runJob(input); + }); + // Swallow rejections on the chained promise so one failed job does + // not poison subsequent ones. Per-job errors are persisted into + // `workspace_jobs.error` and the per-resource `index_status`. + this.queue = next.catch(() => undefined); + return next.catch(() => undefined); + } + + drain(): Promise { + return this.queue.then(() => undefined); + } + + private async runJob(input: { resource_id: number; version_id: number; jobId?: number }): Promise { + try { + let totalProcessed = 0; + // Loop until no more unembedded chunks remain for this version. + // Each iteration pulls one batch worth; the loop terminates when + // the SELECT returns fewer rows than the batch size. + while (true) { + const batch = this.store.fetchPendingChunks(input.version_id, this.batchSize); + if (batch.length === 0) break; + const texts = batch.map((row) => row.content); + let embeddings: number[][]; + let model = "unknown"; + try { + embeddings = await workspaceEmbedDocuments(texts); + model = await workspaceEmbeddingModelName(); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + this.store.markJobFailed(input.jobId ?? null, message); + // Partial failure: leave whatever was already written in place; + // the resource is marked `partial` so callers can retry. + this.store.markVersionEmbeddingPartial(input.resource_id, input.version_id, message); + return; + } + const dimensions = embeddings[0]?.length ?? this.dimensions; + this.store.ensureChildVectorTable(dimensions); + this.store.writeChunkEmbeddings( + batch.map((row, i) => ({ chunkId: row.chunk_id, embedding: embeddings[i] ?? [] })), + model, + dimensions, + ); + totalProcessed += batch.length; + if (batch.length < this.batchSize) break; + } + this.store.markVersionEmbeddingReady(input.resource_id, input.version_id); + if (input.jobId !== undefined) { + this.store.completeJob(input.jobId, totalProcessed); + } + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + this.store.markJobFailed(input.jobId ?? null, message); + this.store.markVersionEmbeddingFailed(input.resource_id, input.version_id, message); + } + } +} + +export function createEmbeddingQueue(deps: EmbeddingQueueDeps): EmbeddingQueue { + return new DefaultEmbeddingQueue(deps); +} diff --git a/packages/workspace/src/index.ts b/packages/workspace/src/index.ts new file mode 100644 index 00000000..eb8c3714 --- /dev/null +++ b/packages/workspace/src/index.ts @@ -0,0 +1,67 @@ +/** + * `@melandlabs/workspace` — barrel re-export. + * + * Mirrors the OKF subpath shape: + * - root: low-level API surface (api, types, schema, sqlite) + * - `/sqlite` `getSQLiteWorkspaceStore()` / `closeSQLiteWorkspaceStore()` + * - `/cli` `opencontext workspace …` subcommand + */ + +export { + getSQLiteWorkspaceStore, + closeSQLiteWorkspaceStore, + resolveWorkspaceDbPath, + createSqliteWorkspaceStore, + __resetSQLiteWorkspaceStoreForTests, +} from "./sqlite"; +export type { SqliteWorkspaceStore, SqliteWorkspaceStoreOptions } from "./sqlite"; + +export { updateWorkspaceContext, searchWorkspaceContext, listWorkspaceResources } from "./api"; +export type { SearchWorkspaceContextDeps } from "./api"; + +export { indexOkfFolder, listOkfFolderResources } from "./okf-backend"; + +export { searchLexical } from "./search/lexical"; +export { searchSemantic } from "./search/semantic"; +export { fuseHybridHits } from "./search/hybrid"; +export { searchCrossFile } from "./search/cross-file"; + +export { createEmbeddingQueue } from "./embedding-queue"; +export type { EmbeddingQueue, EmbeddingQueueDeps } from "./embedding-queue"; + +export { + getWorkspaceEmbeddingProvider, + workspaceEmbedQuery, + workspaceEmbedDocuments, + workspaceEmbeddingModelName, + workspaceEmbeddingDimensions, + __resetWorkspaceEmbeddingProviderForTests, +} from "./embedding-provider"; +export type { + WorkspaceEmbeddingProvider, + WorkspaceEmbeddingProviderType, +} from "./embedding-provider"; + +export { initializeWorkspaceSchema, WORKSPACE_SCHEMA_VERSION } from "./schema"; + +export type { + RuntimeContext, + WorkspaceStorageKind, + WorkspaceEdgeType, + WorkspaceIndexStatus, + WorkspaceSearchStrategy, + WorkspaceResource, + WorkspaceResourceVersion, + WorkspaceChunk, + WorkspaceReferenceEdge, + WorkspaceJob, + WorkspaceSearchHit, + SearchWorkspaceContextOptions, + UpdateWorkspaceContextInput, + UpdateWorkspaceContextResult, + SearchWorkspaceContextInput, + SearchWorkspaceContextResult, + ListWorkspaceResourcesInput, + ListWorkspaceResourcesResult, + OkfFolderResource, +} from "./types"; diff --git a/packages/workspace/src/okf-backend.ts b/packages/workspace/src/okf-backend.ts new file mode 100644 index 00000000..19758afe --- /dev/null +++ b/packages/workspace/src/okf-backend.ts @@ -0,0 +1,223 @@ +/** + * `@melandlabs/workspace` — OKF folder backend. + * + * Wires the multi-format text extractor to `SqliteWorkspaceStore.indexResource` + * and the OKF graph builder to `SqliteWorkspaceStore.upsertReferenceEdges`. + * + * Only `cites` edges are written (the markdown-link resolver in + * `buildGraphFromDir`). `supersedes` / `amends` / `relates-to` are + * reserved in the schema enum but never produced here. + */ + +import { stat } from "node:fs/promises"; +import { readdir } from "node:fs/promises"; +import { join, relative, sep } from "node:path"; +import { extname } from "node:path"; +import { type WikiGraph, type WikiNode, buildGraphFromDir } from "@melandlabs/okf"; +import { extractText } from "./parsers-adapter"; +import type { SqliteWorkspaceStore } from "./sqlite"; +import type { OkfFolderResource, UpdateWorkspaceContextResult, WorkspaceEdgeType } from "./types"; + +const SUPPORTED_EXTENSIONS = new Set([ + ".md", + ".markdown", + ".txt", + ".pdf", + ".docx", + ".xlsx", + ".xls", + ".numbers", + ".pages", +]); + +async function walk(dir: string): Promise { + const out: string[] = []; + const stack = [dir]; + while (stack.length > 0) { + const head = stack.pop(); + if (!head) break; + let entries: import("node:fs").Dirent[]; + try { + entries = (await readdir(head, { withFileTypes: true })) as unknown as import("node:fs").Dirent[]; + } catch { + continue; + } + for (const entry of entries) { + const full = join(head, entry.name); + if (entry.isDirectory()) { + stack.push(full); + } else if (entry.isFile() && SUPPORTED_EXTENSIONS.has(extname(entry.name).toLowerCase())) { + out.push(full); + } + } + } + return out; +} + +function resourceTypeForExtension(ext: string): string { + switch (ext) { + case ".md": + case ".markdown": + return "note"; + case ".txt": + return "note"; + case ".pdf": + return "document"; + case ".docx": + return "document"; + case ".xlsx": + case ".xls": + case ".numbers": + return "spreadsheet"; + case ".pages": + return "document"; + default: + return "document"; + } +} + +/** + * List every supported file under `dir`, parse it, and return the + * `OkfFolderResource[]` shape that `indexResource` consumes. Errors + * per-file are swallowed (logged via stderr) so one broken file doesn't + * abort the whole scan. + */ +export async function listOkfFolderResources(dir: string): Promise { + const files = await walk(dir); + const results: OkfFolderResource[] = []; + for (const absolute of files) { + try { + const extracted = await extractText(absolute); + const ext = extname(absolute).toLowerCase(); + const canonical = relative(dir, absolute).split(sep).join("/"); + const statResult = await stat(absolute); + const resourceType = ext === ".md" || ext === ".markdown" ? "note" : resourceTypeForExtension(ext); + results.push({ + canonical_key: canonical, + absolute_path: absolute, + title: canonical.replace(/\.[^.]+$/, ""), + resource_type: resourceType, + body: extracted.text, + size_bytes: statResult.size, + }); + } catch (error) { + // biome-ignore lint/suspicious/noConsole: server-side warning surfaced to ops + console.warn(`[workspace/okf] failed to read ${absolute}:`, error); + } + } + return results; +} + +/** + * Walk an OKF folder, index every file, then build the `cites` edge + * graph from the markdown-link resolver and persist it. + * + * The job id is created up-front so the embedding queue can attach + * completion / failure to it. + */ +export async function indexOkfFolder( + store: SqliteWorkspaceStore, + input: { + workspace_id: string; + user_id: string; + path: string; + enqueueEmbedding: (input: { resource_id: number; version_id: number; jobId?: number }) => Promise; + }, +): Promise { + const resources = await listOkfFolderResources(input.path); + const job = store.createJob({ workspace_id: input.workspace_id, kind: "index", total: resources.length }); + store.updateJobTotal(job.id, resources.length); + + const presentKeys = new Set(); + let filesAdded = 0; + let filesModified = 0; + let filesUnchanged = 0; + + const indexedIds: Array<{ resource_id: number; version_id: number; canonical_key: string }> = []; + + for (const resource of resources) { + presentKeys.add(resource.canonical_key); + const result = await store.indexResource({ + workspace_id: input.workspace_id, + user_id: input.user_id, + resource, + }); + indexedIds.push({ + resource_id: result.resource_id, + version_id: result.version_id, + canonical_key: resource.canonical_key, + }); + if (result.change_kind === "created") filesAdded += 1; + else if (result.change_kind === "modified") filesModified += 1; + else filesUnchanged += 1; + await input.enqueueEmbedding({ + resource_id: result.resource_id, + version_id: result.version_id, + jobId: job.id, + }); + } + + // Edge pass: rebuild cites edges from the on-disk graph so a rename / + // delete in the OKF folder immediately reflects in the index. + let graph: WikiGraph; + try { + graph = await buildGraphFromDir(input.path); + } catch (error) { + // biome-ignore lint/suspicious/noConsole: server-side warning surfaced to ops + console.warn(`[workspace/okf] buildGraphFromDir failed for ${input.path}:`, error); + graph = { nodes: [], edges: [], types: [], generatedAt: new Date().toISOString(), root: input.path }; + } + const idByCanonical = new Map(); + const canonicalByResourceId = new Map(); + for (const indexed of indexedIds) { + idByCanonical.set(indexed.canonical_key, indexed.resource_id); + canonicalByResourceId.set(indexed.resource_id, indexed.canonical_key); + } + const wikiEdges: Array<{ + source_resource_id: number; + target_resource_id: number; + edge_type: WorkspaceEdgeType; + }> = []; + for (const edge of graph.edges) { + const sourceCanonical = `${edge.source}.md`; + const targetCanonical = `${edge.target}.md`; + const sourceId = idByCanonical.get(sourceCanonical); + const targetId = idByCanonical.get(targetCanonical); + if (sourceId === undefined || targetId === undefined) continue; + wikiEdges.push({ source_resource_id: sourceId, target_resource_id: targetId, edge_type: "cites" }); + } + if (wikiEdges.length > 0) { + store.upsertReferenceEdges({ + workspace_id: input.workspace_id, + edges: wikiEdges.map((edge) => ({ ...edge, source_version_id: null, target_version_id: null })), + }); + } + + // Soft-delete detection: any canonical key not in `presentKeys` and + // not already marked `deleted_at` is flipped to soft-deleted. + const deleted = store.softDeleteMissingResources({ + workspace_id: input.workspace_id, + presentKeys, + }); + + return { + jobId: job.id, + triggered: true, + status: "pending", + filesScanned: resources.length, + filesAdded, + filesModified, + filesUnchanged, + filesDeleted: deleted.length, + }; +} + +/** + * Cheap helper used by callers that only need the wiki node titles + * (e.g. the indexer's title enrichment). Mirrors `buildGraphFromDir`'s + * node shape but stops short of the full edge / backlink pass. + */ +export async function readOkfFolderTitles(dir: string): Promise { + const graph = await buildGraphFromDir(dir); + return graph.nodes; +} diff --git a/packages/workspace/src/parsers-adapter.ts b/packages/workspace/src/parsers-adapter.ts new file mode 100644 index 00000000..f786e0d9 --- /dev/null +++ b/packages/workspace/src/parsers-adapter.ts @@ -0,0 +1,221 @@ +/** + * `@melandlabs/workspace` — multi-format text extractor. + * + * Supported inputs: + * - `.md` / `.markdown` — pass-through (front-matter is parsed by + * `okf-backend` separately) + * - `.txt` — pass-through + * - `.pdf` — `parseFileToDocument` (`@melandlabs/rag`) + * - `.docx` — `parseFileToDocument` + * - `.pages` — `parseFileToDocument` (via `AppleDocumentLoader`) + * - `.xlsx` / `.xls` — SheetJS (`xlsx`) — round-trips every sheet to + * CSV. `@langchain/community` does not ship an + * Excel loader, so we go straight to SheetJS + * instead of paying the langchain dependency + * tax. + * - `.numbers` — `textutil -convert xlsx` (macOS only), + * then the `.xlsx` path above. + * + * Binary raster files (`.png`, `.jpg`, …) intentionally raise an explicit + * "unsupported" error so the caller can fall back to manual transcription. + */ + +import { execFile } from "node:child_process"; +import { readFile, unlink } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { extname } from "node:path"; +import { promisify } from "node:util"; +import { parseFile, parseFileToDocument } from "@melandlabs/rag"; +import { estimateTokens } from "@melandlabs/shared"; + +const execFileAsync = promisify(execFile); + +let _parsersConfigured = false; + +function ensureParsersConfigured(): void { + if (_parsersConfigured) return; + // `parseFile` only uses `estimateTokens` for chunk-count estimation, + // which we don't currently invoke; provide a stub anyway so future + // callers that hit `estimateChunkCount` don't crash. + try { + const { configureParsers } = require("@melandlabs/rag") as { + configureParsers?: (config: { estimateTokens: (text: string) => number }) => void; + }; + configureParsers?.({ estimateTokens }); + } catch { + // No-op: some builds don't expose `configureParsers` directly. + } + _parsersConfigured = true; +} + +export interface ExtractedText { + text: string; + mimeType: string; + metadata?: Record; +} + +const MIME_BY_EXTENSION: Record = { + ".md": "text/markdown", + ".markdown": "text/markdown", + ".txt": "text/plain", + ".pdf": "application/pdf", + ".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + ".doc": "application/msword", + ".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + ".xls": "application/vnd.ms-excel", + ".pages": "application/x-iwork-pages-sffpages", + ".numbers": "application/x-iwork-numbers-sffnumbers", + ".keynote": "application/x-iwork-keynote-sffkeynote", + ".html": "text/html", + ".htm": "text/html", +}; + +/** + * Best-effort MIME guess from the file extension. Falls back to + * `application/octet-stream` so `parseFile` throws a clear "unsupported + * content type" error rather than silently producing garbage. + */ +export function detectMimeType(sourcePath: string): string { + const ext = extname(sourcePath).toLowerCase(); + return MIME_BY_EXTENSION[ext] ?? "application/octet-stream"; +} + +/** + * Render a SheetJS workbook buffer as one CSV block per sheet. Used for + * `.xlsx` / `.xls` (and `.numbers` after a one-shot `textutil` conversion). + * + * Sheets are separated by `\n\n# Sheet: \n` so the chunker can + * preserve sheet boundaries without any extra metadata plumbing. + */ +async function extractSpreadsheet( + buffer: Buffer, + mimeType: string, + sourcePath: string, +): Promise { + const xlsxModule = await import("xlsx"); + // SheetJS exposes its surface as a namespace; some build entries also + // re-export it under `.default`. Coalesce both shapes here so we don't + // care which one `pnpm install` resolved at runtime. + const XLSX = ((xlsxModule as unknown as { default?: typeof xlsxModule }).default ?? + xlsxModule) as typeof xlsxModule; + type SheetJSModule = typeof import("xlsx"); + type Workbook = ReturnType; + const workbook = XLSX.read(buffer, { type: "buffer" }) as Workbook; + const blocks: string[] = []; + const sheetNames = workbook.SheetNames as string[]; + for (const name of sheetNames) { + const sheet = workbook.Sheets[name]; + if (!sheet) continue; + const csv = XLSX.utils.sheet_to_csv(sheet, { blankrows: false }); + if (!csv.trim()) continue; + blocks.push(`# Sheet: ${name}\n${csv.replace(/\n+$/, "")}`); + } + return { + text: blocks.join("\n\n"), + mimeType, + metadata: { + source: sourcePath, + sheets: sheetNames, + sheet_count: sheetNames.length, + }, + }; +} + +/** + * Convert an Apple Numbers workbook to .xlsx via the bundled + * `textutil` (macOS only). Returns the converted file path or null + * if the host is not macOS or `textutil` is missing. + */ +async function convertNumbersToXlsx(sourcePath: string): Promise { + if (process.platform !== "darwin") return null; + const outDir = tmpdir(); + try { + await execFileAsync("textutil", ["-convert", "xlsx", "-output", outDir, sourcePath]); + return `${outDir}/${ + sourcePath + .split("/") + .pop() + ?.replace(/\.numbers$/, ".xlsx") ?? "fixture.xlsx" + }`; + } catch { + return null; + } +} + +/** + * Read the source file, dispatch to the appropriate loader, and return + * the canonical text body used by both the chunker and the embedder. + */ +export async function extractText(sourcePath: string, mimeType?: string): Promise { + ensureParsersConfigured(); + const ext = extname(sourcePath).toLowerCase(); + const mime = mimeType ?? detectMimeType(sourcePath); + if (ext === ".md" || ext === ".markdown" || ext === ".txt") { + const text = await readFile(sourcePath, "utf8"); + return { text, mimeType: mime }; + } + const buffer = await readFile(sourcePath); + // Spreadsheets: SheetJS handles .xlsx / .xls directly. .numbers + // gets a one-shot textutil conversion on macOS, then falls through + // to the SheetJS path. Other platforms reject .numbers explicitly. + if (ext === ".xlsx" || ext === ".xls") { + return extractSpreadsheet(buffer, mime, sourcePath); + } + if (ext === ".numbers") { + const converted = await convertNumbersToXlsx(sourcePath); + if (!converted) { + throw new Error( + `.numbers parsing requires macOS (textutil); not available on ${process.platform}. Convert the file to .xlsx first.`, + ); + } + try { + const xlsxBuffer = await readFile(converted); + return extractSpreadsheet(xlsxBuffer, MIME_BY_EXTENSION[".xlsx"], sourcePath); + } finally { + unlink(converted).catch(() => undefined); + } + } + // The RAG layer exposes `parseFile` (returns `{text, metadata}`) and + // `parseFileToDocument` (returns a LangChain `Document`). The latter + // also surfaces page-level metadata; prefer it when callers can + // surface metadata downstream. + const document = await parseFileToDocument(buffer, mime, sourcePath); + return { + text: document.pageContent, + mimeType: mime, + metadata: document.metadata as Record, + }; +} + +/** + * Convenience re-export of `parseFile` so the OKF backend can opt into + * the lighter (no-LangChain-Document) path when it only needs text. + */ +export async function extractTextRaw(sourcePath: string, mimeType?: string): Promise { + ensureParsersConfigured(); + const ext = extname(sourcePath).toLowerCase(); + const mime = mimeType ?? detectMimeType(sourcePath); + if (ext === ".md" || ext === ".markdown" || ext === ".txt") { + const text = await readFile(sourcePath, "utf8"); + return { text, mimeType: mime }; + } + if (ext === ".xlsx" || ext === ".xls") { + const buffer = await readFile(sourcePath); + return extractSpreadsheet(buffer, mime, sourcePath); + } + if (ext === ".numbers") { + const converted = await convertNumbersToXlsx(sourcePath); + if (!converted) { + throw new Error(`.numbers parsing requires macOS (textutil); not available on ${process.platform}.`); + } + try { + const xlsxBuffer = await readFile(converted); + return extractSpreadsheet(xlsxBuffer, MIME_BY_EXTENSION[".xlsx"], sourcePath); + } finally { + unlink(converted).catch(() => undefined); + } + } + const buffer = await readFile(sourcePath); + const { text, metadata } = await parseFile(buffer, mime); + return { text, mimeType: mime, metadata }; +} diff --git a/packages/workspace/src/schema.ts b/packages/workspace/src/schema.ts new file mode 100644 index 00000000..6eb6bb41 --- /dev/null +++ b/packages/workspace/src/schema.ts @@ -0,0 +1,178 @@ +/** + * `@melandlabs/workspace` — schema bootstrap. + * + * Workspace tables live in the shared SQLite DB that the memory-store + * already owns (`~/.opencontext/memory/store.db`), in their own + * `workspace_*` namespace so they don't collide with `raw_messages` / + * `facts` / `entities`. + * + * The `addColumnIfMissing` helper from `@melandlabs/sqlite/src/schema` + * is not exported; we replicate the lightweight version here because + * the workspace tables are created from scratch. When a v2 migration + * arrives, the same `pragma_table_info`-based detection pattern from + * the memory schema should be applied — keeping this file's behaviour + * consistent with `packages/sqlite/src/schema.ts:11-17`. + */ + +import type Database from "better-sqlite3"; + +/** + * Current schema version. Bump when adding a new table or column; + * the boot path reads this constant to decide whether to run a + * migration. + */ +export const WORKSPACE_SCHEMA_VERSION = 1; + +/** + * Idempotent column-add helper for SQLite (which lacks `ADD COLUMN IF NOT + * EXISTS`). Mirrors `packages/sqlite/src/schema.ts:11-17`. + */ +function addColumnIfMissing(db: Database.Database, table: string, column: string, definition: string): void { + const rows = db.prepare(`PRAGMA table_info(${table})`).all() as Array<{ name: string }>; + if (rows.some((row) => row.name === column)) return; + db.exec(`ALTER TABLE ${table} ADD COLUMN ${column} ${definition};`); +} + +/** + * Run every idempotent DDL statement that backs the workspace + * feature. Safe to call repeatedly. + * + * - `workspace_resources` — one row per (workspace, canonical_key) + * - `workspace_resource_versions` — version chain per resource (sha256, parent) + * - `workspace_chunks` — search-only child chunks per version + * - `workspace_chunks_fts` — FTS5 mirror of `workspace_chunks.content` + * - `workspace_reference_edges` — cites / supersedes / amends / relates-to + * - `workspace_jobs` — async indexing job status + * + * The vec0 child table `workspace_chunks_vec_d1536` is created lazily by + * `SqliteWorkspaceStore.ensureChildVectorTable(dimensions)` so an empty + * workspace (no embeddings yet) never wastes disk on an empty vector table. + */ +export function initializeWorkspaceSchema(db: Database.Database): void { + db.pragma("journal_mode = WAL"); + db.pragma("busy_timeout = 30000"); + db.pragma("foreign_keys = ON"); + + db.exec(` + CREATE TABLE IF NOT EXISTS workspace_resources ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + workspace_id TEXT NOT NULL, + user_id TEXT NOT NULL, + resource_type TEXT NOT NULL, + canonical_key TEXT NOT NULL, + title TEXT NOT NULL, + storage_kind TEXT NOT NULL, + current_version_id INTEGER, + index_status TEXT NOT NULL DEFAULT 'pending', + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + metadata TEXT, + UNIQUE(workspace_id, canonical_key) + ); + + CREATE INDEX IF NOT EXISTS idx_workspace_resources_workspace_type + ON workspace_resources(workspace_id, resource_type); + CREATE INDEX IF NOT EXISTS idx_workspace_resources_workspace_status + ON workspace_resources(workspace_id, index_status); + CREATE INDEX IF NOT EXISTS idx_workspace_resources_user + ON workspace_resources(user_id); + + CREATE TABLE IF NOT EXISTS workspace_resource_versions ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + resource_id INTEGER NOT NULL, + version_number INTEGER NOT NULL, + sha256 TEXT NOT NULL, + change_kind TEXT NOT NULL, + size_bytes INTEGER NOT NULL, + parent_version_id INTEGER, + source_path TEXT, + created_at INTEGER NOT NULL, + metadata TEXT, + UNIQUE(resource_id, version_number) + ); + + CREATE INDEX IF NOT EXISTS idx_workspace_resource_versions_resource + ON workspace_resource_versions(resource_id, version_number DESC); + + CREATE TABLE IF NOT EXISTS workspace_chunks ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + chunk_id TEXT UNIQUE NOT NULL, + resource_id INTEGER NOT NULL, + version_id INTEGER NOT NULL, + workspace_id TEXT NOT NULL, + chunk_index INTEGER NOT NULL, + chunk_count INTEGER NOT NULL, + start_position INTEGER NOT NULL, + end_position INTEGER NOT NULL, + content TEXT NOT NULL, + content_hash TEXT NOT NULL, + embedding BLOB, + embedding_model TEXT, + embedding_dimensions INTEGER, + embedding_updated_at INTEGER, + UNIQUE(resource_id, version_id, chunk_index) + ); + + CREATE INDEX IF NOT EXISTS idx_workspace_chunks_workspace + ON workspace_chunks(workspace_id); + CREATE INDEX IF NOT EXISTS idx_workspace_chunks_resource + ON workspace_chunks(resource_id, version_id); + + CREATE VIRTUAL TABLE IF NOT EXISTS workspace_chunks_fts USING fts5( + content, + content='workspace_chunks', + content_rowid='id' + ); + + CREATE TRIGGER IF NOT EXISTS workspace_chunks_ai AFTER INSERT ON workspace_chunks BEGIN + INSERT INTO workspace_chunks_fts(rowid, content) VALUES (new.id, new.content); + END; + CREATE TRIGGER IF NOT EXISTS workspace_chunks_ad AFTER DELETE ON workspace_chunks BEGIN + INSERT INTO workspace_chunks_fts(workspace_chunks_fts, rowid, content) + VALUES('delete', old.id, old.content); + END; + CREATE TRIGGER IF NOT EXISTS workspace_chunks_au AFTER UPDATE ON workspace_chunks BEGIN + INSERT INTO workspace_chunks_fts(workspace_chunks_fts, rowid, content) + VALUES('delete', old.id, old.content); + INSERT INTO workspace_chunks_fts(rowid, content) VALUES (new.id, new.content); + END; + + CREATE TABLE IF NOT EXISTS workspace_reference_edges ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + workspace_id TEXT NOT NULL, + source_resource_id INTEGER NOT NULL, + source_version_id INTEGER, + target_resource_id INTEGER NOT NULL, + target_version_id INTEGER, + edge_type TEXT NOT NULL, + quote TEXT, + created_at INTEGER NOT NULL, + UNIQUE(workspace_id, source_resource_id, target_resource_id, edge_type) + ); + + CREATE INDEX IF NOT EXISTS idx_workspace_reference_edges_workspace + ON workspace_reference_edges(workspace_id); + CREATE INDEX IF NOT EXISTS idx_workspace_reference_edges_source + ON workspace_reference_edges(source_resource_id); + + CREATE TABLE IF NOT EXISTS workspace_jobs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + workspace_id TEXT NOT NULL, + kind TEXT NOT NULL, + status TEXT NOT NULL, + total INTEGER NOT NULL DEFAULT 0, + done INTEGER NOT NULL DEFAULT 0, + error TEXT, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_workspace_jobs_workspace + ON workspace_jobs(workspace_id, created_at DESC); + `); + + // Future-proof: keep `addColumnIfMissing` available for v2 migrations + // without TypeScript flagging it as unused. Same pattern as the + // memory-store schema's v2–v5 column upgrades. + void addColumnIfMissing; +} diff --git a/packages/workspace/src/search/cross-file.ts b/packages/workspace/src/search/cross-file.ts new file mode 100644 index 00000000..e4a6058d --- /dev/null +++ b/packages/workspace/src/search/cross-file.ts @@ -0,0 +1,73 @@ +/** + * `@melandlabs/workspace/search/cross-file` — hybrid + BFS over cites edges. + * + * 1. Run hybrid search to collect the top-`limit * 4` candidates. + * 2. BFS 1–2 hops over `workspace_reference_edges` to surface neighbour + * resources (typically the law clause a contract cites, or the + * reference a review result depends on). + * 3. Re-rank with `edge_boost = 0.1 × (out_degree + in_degree)` so + * heavily linked resources surface higher. + */ + +import type { SqliteWorkspaceStore } from "../sqlite"; +import type { WorkspaceSearchHit } from "../types"; +import { fuseHybridHits } from "./hybrid"; + +export interface CrossFileSearchInput { + workspace_id: string; + user_id: string; + query: string; + resource_types?: string[]; + limit: number; + threshold?: number; + hops?: 1 | 2; + lexicalSearch: (input: { + workspace_id: string; + user_id: string; + query: string; + resource_types?: string[]; + limit: number; + }) => WorkspaceSearchHit[]; + semanticSearch: (input: { + workspace_id: string; + user_id: string; + queryEmbedding: number[]; + resource_types?: string[]; + limit: number; + threshold: number; + }) => WorkspaceSearchHit[]; + generateEmbedding?: (text: string) => Promise; +} + +export async function searchCrossFile( + store: SqliteWorkspaceStore, + input: CrossFileSearchInput, +): Promise { + const candidateLimit = input.limit * 4; + const lexicalHits = input.lexicalSearch({ + workspace_id: input.workspace_id, + user_id: input.user_id, + query: input.query, + resource_types: input.resource_types, + limit: candidateLimit, + }); + let semanticHits: WorkspaceSearchHit[] = []; + if (input.generateEmbedding) { + const embedding = await input.generateEmbedding(input.query); + semanticHits = input.semanticSearch({ + workspace_id: input.workspace_id, + user_id: input.user_id, + queryEmbedding: embedding, + resource_types: input.resource_types, + limit: candidateLimit, + threshold: input.threshold ?? 0.7, + }); + } + const fused = fuseHybridHits({ lexical: lexicalHits, semantic: semanticHits, limit: candidateLimit }); + return store.expandNeighbors({ + workspace_id: input.workspace_id, + hits: fused, + hops: input.hops ?? 1, + limit: input.limit, + }); +} diff --git a/packages/workspace/src/search/hybrid.ts b/packages/workspace/src/search/hybrid.ts new file mode 100644 index 00000000..3ed6606c --- /dev/null +++ b/packages/workspace/src/search/hybrid.ts @@ -0,0 +1,87 @@ +/** + * `@melandlabs/workspace/search/hybrid` — lexical × semantic fusion. + * + * Wraps `fuseHybridResults` (`@melandlabs/rag`) with the RRF k=60 default + * the plan specifies. The store-side rank list is normalised to the + * `VectorSearchResult` shape that the RAG fusion helper expects. + */ + +import { type VectorSearchResult, fuseHybridResults } from "@melandlabs/rag"; +import type { WorkspaceSearchHit } from "../types"; + +function toVectorResult(hit: WorkspaceSearchHit): VectorSearchResult { + return { + id: hit.chunk_id, + documentId: hit.canonical_key, + content: hit.snippet, + score: hit.score, + metadata: { + resource_id: hit.resource_id, + version_id: hit.version_id, + resource_type: hit.resource_type, + resource_title: hit.resource_title, + }, + }; +} + +function fromVectorResult(result: VectorSearchResult, original: WorkspaceSearchHit): WorkspaceSearchHit { + const meta = result.metadata ?? {}; + return { + ...original, + score: result.score, + signals: { + ...original.signals, + lexical: original.signals.lexical, + semantic: original.signals.semantic, + edge_boost: original.signals.edge_boost, + }, + chunk_id: result.id, + canonical_key: (meta.documentId as string) ?? original.canonical_key, + }; +} + +export interface HybridSearchInput { + lexical: WorkspaceSearchHit[]; + semantic: WorkspaceSearchHit[]; + limit: number; + rrfK?: number; + alpha?: number; +} + +export function fuseHybridHits(input: HybridSearchInput): WorkspaceSearchHit[] { + const dense = input.semantic.map(toVectorResult); + const lexical = input.lexical.map(toVectorResult); + const fused = fuseHybridResults({ + dense, + lexical, + strategy: "rrf", + rrfK: input.rrfK ?? 60, + alpha: input.alpha ?? 0.5, + limit: input.limit, + }); + // `fuseHybridResults` mutates `result.score`; look up the original + // WorkspaceSearchHit by chunk_id so the return shape carries the + // signals / edges / metadata that the lexical / semantic passes built. + const byChunkId = new Map(); + for (const hit of input.lexical) byChunkId.set(hit.chunk_id, hit); + for (const hit of input.semantic) byChunkId.set(hit.chunk_id, hit); + return fused.map((result) => { + const original = byChunkId.get(result.id); + if (!original) { + return { + chunk_id: result.id, + resource_id: 0, + version_id: 0, + resource_type: "document", + resource_title: "", + canonical_key: result.documentId, + snippet: result.content, + matched_terms: [], + score: result.score, + signals: {}, + reference_edges: [], + }; + } + return fromVectorResult(result, original); + }); +} diff --git a/packages/workspace/src/search/lexical.ts b/packages/workspace/src/search/lexical.ts new file mode 100644 index 00000000..67da1f47 --- /dev/null +++ b/packages/workspace/src/search/lexical.ts @@ -0,0 +1,24 @@ +/** + * `@melandlabs/workspace/search/lexical` — FTS5 lexical search. + * + * Mirrors `packages/sqlite/src/raw-message-manager.ts:332-338` for the + * FTS5 query construction (`"" OR ""`) and `bm25()` ranking. + * Per the plan, the lexical pass is the fallback when embeddings haven't + * been written yet, so it must remain functional before the embedding + * queue drains. + */ + +import type { SqliteWorkspaceStore } from "../sqlite"; +import type { WorkspaceSearchHit } from "../types"; + +export interface LexicalSearchInput { + workspace_id: string; + user_id: string; + query: string; + resource_types?: string[]; + limit: number; +} + +export function searchLexical(store: SqliteWorkspaceStore, input: LexicalSearchInput): WorkspaceSearchHit[] { + return store.searchLexical(input); +} diff --git a/packages/workspace/src/search/semantic.ts b/packages/workspace/src/search/semantic.ts new file mode 100644 index 00000000..32483cd2 --- /dev/null +++ b/packages/workspace/src/search/semantic.ts @@ -0,0 +1,26 @@ +/** + * `@melandlabs/workspace/search/semantic` — sqlite-vec KNN search. + * + * Wraps `SqliteWorkspaceStore.searchSemantic` so the API layer can stay + * store-agnostic. The store handles the widen-and-retry loop and the + * "no embeddings yet → empty result" fallback. + */ + +import type { SqliteWorkspaceStore } from "../sqlite"; +import type { WorkspaceSearchHit } from "../types"; + +export interface SemanticSearchInput { + workspace_id: string; + user_id: string; + queryEmbedding: number[]; + resource_types?: string[]; + limit: number; + threshold: number; +} + +export function searchSemantic( + store: SqliteWorkspaceStore, + input: SemanticSearchInput, +): WorkspaceSearchHit[] { + return store.searchSemantic(input); +} diff --git a/packages/workspace/src/sqlite.ts b/packages/workspace/src/sqlite.ts new file mode 100644 index 00000000..ca2e6842 --- /dev/null +++ b/packages/workspace/src/sqlite.ts @@ -0,0 +1,1269 @@ +/** + * `@melandlabs/workspace/sqlite` — the SQLite-backed store. + * + * Public surface: + * + * - `SqliteWorkspaceStore` — CRUD + indexing pipeline + * - `getSQLiteWorkspaceStore()` — singleton accessor (lazy open) + * - `closeSQLiteWorkspaceStore()`— test-only teardown + * - `resolveWorkspaceDbPath(dbPath?)` — same env-var fallback as + * `@melandlabs/memory-store`'s `resolveSQLiteRawMessageDbPath` + * (`MEMORY_STORE_DB_PATH` → `~/.opencontext/memory/store.db`) + * + * Design notes (mirrors `SQLiteRawMessageManager` patterns): + * + * - One SQLite file is shared with the rest of the memory store so + * the user's local data lives under a single `~/.opencontext/memory/store.db`. + * The vec0 child table is created lazily by + * `ensureChildVectorTable(dimensions)` — same as + * `packages/sqlite/src/raw-message-manager.ts:1859-1874`. + * - `indexResource` runs the **sync** portion of the indexing pipeline + * (sha256, version chain, chunk insert, FTS5 mirror) inside one + * `better-sqlite3` transaction. The async embedding fan-out is + * delegated to `EmbeddingQueue`, which writes the embedding + + * vec0 row + `index_status` flip in subsequent transactions. + */ + +import { createHash } from "node:crypto"; +import { mkdirSync } from "node:fs"; +import { dirname } from "node:path"; +import { getOpenContextPath } from "@melandlabs/env-config"; +import { + RAW_MESSAGE_CHUNK_MAX_TOKENS, + RAW_MESSAGE_CHUNK_OVERLAP_TOKENS, + chunkTextByEstimatedTokens, +} from "@melandlabs/shared"; +import { floatArrayToBuffer } from "@melandlabs/sqlite"; +import Database from "better-sqlite3"; +import * as sqliteVec from "sqlite-vec"; +import { initializeWorkspaceSchema } from "./schema"; +import type { + ListWorkspaceResourcesInput, + ListWorkspaceResourcesResult, + OkfFolderResource, + SearchWorkspaceContextInput, + SearchWorkspaceContextResult, + UpdateWorkspaceContextInput, + UpdateWorkspaceContextResult, + WorkspaceEdgeType, + WorkspaceIndexStatus, + WorkspaceJob, + WorkspaceResource, + WorkspaceResourceVersion, + WorkspaceSearchHit, + WorkspaceSearchStrategy, +} from "./types"; + +type DatabaseLike = Database.Database; + +interface WorkspaceResourceRow { + id: number; + workspace_id: string; + user_id: string; + resource_type: string; + canonical_key: string; + title: string; + storage_kind: string; + current_version_id: number | null; + index_status: string; + created_at: number; + updated_at: number; + metadata: string | null; +} + +interface WorkspaceChunkRow { + id: number; + chunk_id: string; + resource_id: number; + version_id: number; + workspace_id: string; + chunk_index: number; + chunk_count: number; + start_position: number; + end_position: number; + content: string; + content_hash: string; + embedding: Buffer | null; + embedding_model: string | null; + embedding_dimensions: number | null; + embedding_updated_at: number | null; +} + +interface WorkspaceJobRow { + id: number; + workspace_id: string; + kind: string; + status: string; + total: number; + done: number; + error: string | null; + created_at: number; + updated_at: number; +} + +function toWorkspaceResource(row: WorkspaceResourceRow): WorkspaceResource { + return { + id: row.id, + workspace_id: row.workspace_id, + user_id: row.user_id, + resource_type: row.resource_type, + canonical_key: row.canonical_key, + title: row.title, + storage_kind: row.storage_kind as WorkspaceResource["storage_kind"], + current_version_id: row.current_version_id, + index_status: row.index_status as WorkspaceIndexStatus, + created_at: row.created_at, + updated_at: row.updated_at, + metadata: parseJson>(row.metadata, {} as Record), + }; +} + +function toWorkspaceJob(row: WorkspaceJobRow): WorkspaceJob { + return { + id: row.id, + workspace_id: row.workspace_id, + kind: row.kind as WorkspaceJob["kind"], + status: row.status as WorkspaceIndexStatus, + total: row.total, + done: row.done, + error: row.error, + created_at: row.created_at, + updated_at: row.updated_at, + }; +} + +function currentUnixSeconds(): number { + return Math.floor(Date.now() / 1000); +} + +function sha256(value: string): string { + return createHash("sha256").update(value).digest("hex"); +} + +/** + * Local copies of `parseJson` / `stringifyJson` from + * `packages/sqlite/src/raw-message-manager.ts:180-196`. Those helpers + * are not exported from `@melandlabs/sqlite`, so the workspace + * package keeps its own copy in scope. Behaviour matches the memory + * store: missing / invalid JSON falls back to the caller-provided + * default rather than throwing. + */ +function parseJson(value: string | null | undefined, fallback: T): T { + if (!value) return fallback; + try { + return JSON.parse(value) as T; + } catch { + return fallback; + } +} + +function stringifyJson(value: unknown): string | null { + if (value === undefined) return null; + return JSON.stringify(value); +} + +/** + * Resolve the SQLite DB file path that backs the workspace store. + * Mirrors `packages/memory-store/src/storage/sqlite-raw-message-store.ts:27-31` + * so the workspace tables live in the same file as the rest of the + * memory store (no separate DB to coordinate backups / migrations). + */ +export function resolveWorkspaceDbPath(dbPath?: string): string { + if (dbPath && dbPath.length > 0) return dbPath; + const fromEnv = process.env.MEMORY_STORE_DB_PATH?.trim(); + return fromEnv && fromEnv.length > 0 ? fromEnv : getOpenContextPath("memory", "store.db"); +} + +export interface SqliteWorkspaceStoreOptions { + dbPath?: string; + db?: DatabaseLike; + /** + * Lazily import a custom embedding function. Tests inject a deterministic + * mock here to avoid hitting OpenRouter; production leaves this unset + * and uses `@melandlabs/rag`'s `generateEmbeddings` directly. + */ + embeddingQueueFactory?: (store: SqliteWorkspaceStore) => { + enqueue(input: { resource_id: number; version_id: number; jobId?: number }): Promise; + drain(): Promise; + }; +} + +/** + * Lightweight stub interface that `EmbeddingQueue` consumes via duck typing + * (`store.fetchPendingChunks`, `store.writeChunkEmbeddings`, …). Keeping it + * here (rather than re-importing the queue file) avoids a circular type + * reference between `embedding-queue.ts` and `sqlite.ts`. + */ +export interface ISqliteWorkspaceStore { + readonly __testDb: DatabaseLike; + init(): Promise; + close(): Promise; + + fetchPendingChunks(versionId: number, limit: number): Array; + writeChunkEmbeddings( + entries: Array<{ chunkId: string; embedding: number[] }>, + model: string, + dimensions: number, + ): void; + ensureChildVectorTable(dimensions: number): void; + markVersionEmbeddingReady(resourceId: number, versionId: number): void; + markVersionEmbeddingPartial(resourceId: number, versionId: number, errorMessage: string): void; + markVersionEmbeddingFailed(resourceId: number, versionId: number, errorMessage: string): void; + markJobFailed(jobId: number | null, errorMessage: string): void; + completeJob(jobId: number, done: number): void; + + indexResource(input: { + workspace_id: string; + user_id: string; + resource: OkfFolderResource; + }): Promise<{ + resource_id: number; + version_id: number; + change_kind: WorkspaceResourceVersion["change_kind"]; + }>; + + upsertReferenceEdges(input: { + workspace_id: string; + edges: Array<{ + source_resource_id: number; + source_version_id: number | null; + target_resource_id: number; + target_version_id: number | null; + edge_type: WorkspaceEdgeType; + quote?: string | null; + }>; + }): void; + + softDeleteMissingResources(input: { + workspace_id: string; + presentKeys: Set; + }): Array<{ canonical_key: string }>; + + listResources(input: ListWorkspaceResourcesInput): ListWorkspaceResourcesResult; + + searchLexical(input: { + workspace_id: string; + user_id: string; + query: string; + resource_types?: string[]; + limit: number; + }): WorkspaceSearchHit[]; + + searchSemantic(input: { + workspace_id: string; + user_id: string; + queryEmbedding: number[]; + resource_types?: string[]; + limit: number; + threshold: number; + }): WorkspaceSearchHit[]; + + expandNeighbors(input: { + workspace_id: string; + hits: WorkspaceSearchHit[]; + hops: 1 | 2; + limit: number; + }): WorkspaceSearchHit[]; + + findResourceByCanonicalKey(input: { + workspace_id: string; + canonical_key: string; + }): WorkspaceResource | null; + + createJob(input: { workspace_id: string; kind: WorkspaceJob["kind"]; total: number }): WorkspaceJob; + updateJobTotal(jobId: number, total: number): void; +} + +/** + * `SqliteWorkspaceStore` — full implementation. The class body is split + * into clearly labelled sections (`init/close`, `index pipeline`, + * `search pipeline`, `cross-file expansion`) to keep the file readable + * when each section grows in later phases. + */ +export class SqliteWorkspaceStore implements ISqliteWorkspaceStore { + readonly __testDb!: DatabaseLike; + private readonly db: DatabaseLike; + private readonly ownsConnection: boolean; + private readonly embeddingQueueFactory?: SqliteWorkspaceStoreOptions["embeddingQueueFactory"]; + private initialized = false; + private vectorSearchAvailable = false; + + constructor(options: SqliteWorkspaceStoreOptions | string = ":memory:") { + if (typeof options === "string") { + this.db = new Database(options); + this.ownsConnection = true; + this.embeddingQueueFactory = undefined; + } else if (options.db) { + this.db = options.db; + this.ownsConnection = false; + this.embeddingQueueFactory = options.embeddingQueueFactory; + } else { + this.db = new Database(options.dbPath ?? ":memory:"); + this.ownsConnection = true; + this.embeddingQueueFactory = options.embeddingQueueFactory; + } + // `__testDb` is intentionally a public readonly handle (mirrors + // `SQLiteVsaStore.__testDb`). Assigning it through `this.__testDb` + // in a `readonly` declaration trips the DTS build's strictness, + // so we set it via `Object.defineProperty` here. This is only + // touched in tests — production code goes through the typed + // surface above. + Object.defineProperty(this, "__testDb", { value: this.db, writable: false, enumerable: true }); + } + + async init(): Promise { + if (this.initialized) return; + initializeWorkspaceSchema(this.db); + // Load sqlite-vec into this connection so vec0 tables can be + // created/queried. Mirrors `SQLiteRawMessageManager.initializeVectorSearch` + // (`packages/sqlite/src/raw-message-manager.ts:1683-1695`). If the + // extension can't load (e.g. the binary wasn't built for the host + // platform) we leave `vectorSearchAvailable = false` so the + // embedding fan-out degrades to lexical-only instead of crashing. + try { + sqliteVec.load(this.db); + this.vectorSearchAvailable = true; + } catch { + this.vectorSearchAvailable = false; + } + this.initialized = true; + } + + async close(): Promise { + // Best-effort: drop the singleton reference and let the OS reap the + // native handle on process exit. `db.close()` is intentionally NOT + // called here because `sqlite-vec`'s native destructor occasionally + // raises SIGABRT during process teardown on macOS, which would + // surface to the user as a confusing abort right after we've + // already printed the search result. The trade-off is that the + // process exit takes a few extra ms (sqlite writes its WAL flush) + // but no abort noise. + this.initialized = false; + if (!this.ownsConnection) return; + // We intentionally do NOT call `this.db.close()` — see comment above. + // Tests that need a clean teardown should use `__resetSQLiteWorkspaceStoreForTests`. + } + + // ------------------------------------------------------------------------- + // Vector table helpers (mirrors raw-message-manager.ts:1859-1914) + // ------------------------------------------------------------------------- + + private getChildVectorTableName(dimensions: number): string { + if (!Number.isInteger(dimensions) || dimensions <= 0) { + throw new Error(`Invalid project chunk embedding dimensions: ${dimensions}`); + } + return `workspace_chunks_vec_d${dimensions}`; + } + + ensureChildVectorTable(dimensions: number): void { + if (!this.vectorSearchAvailable) { + // sqlite-vec didn't load — skip vec0 creation. The embedding + // queue's `writeChunkEmbeddings` will still persist the raw + // vector into `workspace_chunks.embedding` for later replay + // once the extension is available, and semantic search will + // fall back to lexical. + return; + } + const tableName = this.getChildVectorTableName(dimensions); + this.db.exec(` + CREATE VIRTUAL TABLE IF NOT EXISTS ${tableName} + USING vec0( + embedding float[${dimensions}], + chunk_id TEXT PRIMARY KEY + ); + `); + } + + private listChildVectorTables(): string[] { + return ( + this.db + .prepare( + "SELECT name FROM sqlite_master WHERE type = 'table' AND name LIKE 'workspace_chunks_vec_d%'", + ) + .all() as Array<{ name: string }> + ) + .map((row) => row.name) + .filter((name) => /^workspace_chunks_vec_d\d+$/.test(name)); + } + + // ------------------------------------------------------------------------- + // Embedding queue helpers + // ------------------------------------------------------------------------- + + fetchPendingChunks(versionId: number, limit: number): Array { + return this.db + .prepare( + `SELECT id, chunk_id, resource_id, version_id, workspace_id, chunk_index, chunk_count, + start_position, end_position, content, content_hash, + embedding, embedding_model, embedding_dimensions, embedding_updated_at + FROM workspace_chunks + WHERE version_id = ? AND embedding IS NULL + ORDER BY chunk_index + LIMIT ?`, + ) + .all(versionId, limit) as Array; + } + + writeChunkEmbeddings( + entries: Array<{ chunkId: string; embedding: number[] }>, + model: string, + dimensions: number, + ): void { + if (entries.length === 0) return; + const writeVec = this.vectorSearchAvailable; + const dimensionsTable = this.getChildVectorTableName(dimensions); + this.db.exec("BEGIN"); + try { + const updateStmt = this.db.prepare( + `UPDATE workspace_chunks + SET embedding = ?, embedding_model = ?, embedding_dimensions = ?, embedding_updated_at = ? + WHERE chunk_id = ?`, + ); + const insertVecStmt = writeVec + ? this.db.prepare(`INSERT OR REPLACE INTO ${dimensionsTable}(embedding, chunk_id) VALUES (?, ?)`) + : null; + const now = currentUnixSeconds(); + for (const entry of entries) { + const buffer = floatArrayToBuffer(entry.embedding); + updateStmt.run(buffer, model, dimensions, now, entry.chunkId); + if (writeVec && buffer && insertVecStmt) insertVecStmt.run(buffer, entry.chunkId); + } + this.db.exec("COMMIT"); + } catch (error) { + this.db.exec("ROLLBACK"); + throw error; + } + } + + markVersionEmbeddingReady(resourceId: number, versionId: number): void { + const stillMissing = this.db + .prepare("SELECT 1 FROM workspace_chunks WHERE version_id = ? AND embedding IS NULL LIMIT 1") + .get(versionId); + if (stillMissing) { + this.db.prepare(`UPDATE workspace_resources SET index_status = 'partial' WHERE id = ?`).run(resourceId); + return; + } + this.db.prepare(`UPDATE workspace_resources SET index_status = 'ready' WHERE id = ?`).run(resourceId); + } + + markVersionEmbeddingPartial(resourceId: number, versionId: number, errorMessage: string): void { + // `versionId` and `errorMessage` are reserved for a future + // per-version error column on `workspace_jobs`. For now we just + // flip the resource status to `partial` so callers can see the + // resource is half-indexed. + void versionId; + void errorMessage; + this.db.prepare(`UPDATE workspace_resources SET index_status = 'partial' WHERE id = ?`).run(resourceId); + } + + markVersionEmbeddingFailed(resourceId: number, versionId: number, errorMessage: string): void { + // Same reservation as `markVersionEmbeddingPartial`. Future phases + // will surface the error string on the resource / version row. + void versionId; + void errorMessage; + this.db.prepare(`UPDATE workspace_resources SET index_status = 'failed' WHERE id = ?`).run(resourceId); + } + + markJobFailed(jobId: number | null, errorMessage: string): void { + if (jobId === null) return; + this.db + .prepare(`UPDATE workspace_jobs SET status = 'failed', error = ?, updated_at = ? WHERE id = ?`) + .run(errorMessage, currentUnixSeconds(), jobId); + } + + completeJob(jobId: number, done: number): void { + this.db + .prepare(`UPDATE workspace_jobs SET status = 'ready', done = ?, updated_at = ? WHERE id = ?`) + .run(done, currentUnixSeconds(), jobId); + } + + // ------------------------------------------------------------------------- + // Indexing pipeline (sync portion — embedding fan-out is enqueued separately) + // ------------------------------------------------------------------------- + + async indexResource(input: { + workspace_id: string; + user_id: string; + resource: OkfFolderResource; + }): Promise<{ + resource_id: number; + version_id: number; + change_kind: WorkspaceResourceVersion["change_kind"]; + }> { + await this.init(); + const { workspace_id, user_id, resource } = input; + const now = currentUnixSeconds(); + const contentHash = sha256(resource.body); + + const tx = this.db.transaction(() => { + const existing = this.db + .prepare( + `SELECT id, current_version_id + FROM workspace_resources + WHERE workspace_id = ? AND canonical_key = ?`, + ) + .get(workspace_id, resource.canonical_key) as + | { id: number; current_version_id: number | null } + | undefined; + + let resourceId: number; + let currentVersionId: number | null = existing?.current_version_id ?? null; + let changeKind: WorkspaceResourceVersion["change_kind"]; + + if (!existing) { + const insertResource = this.db + .prepare( + `INSERT INTO workspace_resources( + workspace_id, user_id, resource_type, canonical_key, title, storage_kind, + current_version_id, index_status, created_at, updated_at, metadata + ) VALUES (?, ?, ?, ?, ?, ?, NULL, 'pending', ?, ?, ?)`, + ) + .run( + workspace_id, + user_id, + resource.resource_type, + resource.canonical_key, + resource.title, + "okf_local_dir", + now, + now, + stringifyJson(resource.front_matter), + ); + resourceId = Number(insertResource.lastInsertRowid); + changeKind = "created"; + } else { + resourceId = existing.id; + if (currentVersionId !== null) { + const prevVersion = this.db + .prepare("SELECT sha256 FROM workspace_resource_versions WHERE id = ?") + .get(currentVersionId) as { sha256: string } | undefined; + if (prevVersion?.sha256 === contentHash) { + changeKind = "unchanged"; + // Touch updated_at so the resource stays "fresh" without + // re-indexing — callers may want to detect staleness + // via `updated_at` vs. `current_version_id.created_at`. + this.db + .prepare("UPDATE workspace_resources SET updated_at = ? WHERE id = ?") + .run(now, resourceId); + return { resource_id: resourceId, version_id: currentVersionId, change_kind: changeKind }; + } + } + changeKind = "modified"; + } + + const maxVersion = this.db + .prepare( + `SELECT COALESCE(MAX(version_number), 0) AS max_version + FROM workspace_resource_versions WHERE resource_id = ?`, + ) + .get(resourceId) as { max_version: number }; + + const insertVersion = this.db + .prepare( + `INSERT INTO workspace_resource_versions( + resource_id, version_number, sha256, change_kind, size_bytes, + parent_version_id, source_path, created_at, metadata + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .run( + resourceId, + maxVersion.max_version + 1, + contentHash, + changeKind, + resource.size_bytes, + currentVersionId, + resource.absolute_path, + now, + stringifyJson(resource.front_matter), + ); + const versionId = Number(insertVersion.lastInsertRowid); + + // Delete any old chunks for this resource; FTS5 mirror is + // kept in sync via the AI/AD/AU triggers. + this.db.prepare("DELETE FROM workspace_chunks WHERE resource_id = ?").run(resourceId); + + const pieces = chunkTextByEstimatedTokens(resource.body, { + maxTokens: RAW_MESSAGE_CHUNK_MAX_TOKENS, + overlapTokens: RAW_MESSAGE_CHUNK_OVERLAP_TOKENS, + }); + const chunkCount = pieces.length; + const insertChunk = this.db.prepare( + `INSERT INTO workspace_chunks( + chunk_id, resource_id, version_id, workspace_id, chunk_index, chunk_count, + start_position, end_position, content, content_hash + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ); + for (const piece of pieces) { + const pieceHash = sha256(piece.content); + const chunkId = `${workspace_id}:${resourceId}:${versionId}:chunk:${piece.chunkIndex}:${pieceHash.slice(0, 16)}`; + insertChunk.run( + chunkId, + resourceId, + versionId, + workspace_id, + piece.chunkIndex, + chunkCount, + piece.startPosition, + piece.endPosition, + piece.content, + pieceHash, + ); + } + + this.db + .prepare( + `UPDATE workspace_resources + SET current_version_id = ?, index_status = 'pending', updated_at = ?, title = ? + WHERE id = ?`, + ) + .run(versionId, now, resource.title, resourceId); + + currentVersionId = versionId; + return { resource_id: resourceId, version_id: versionId, change_kind: changeKind }; + }); + + const result = tx(); + return result; + } + + // ------------------------------------------------------------------------- + // Reference edges + // ------------------------------------------------------------------------- + + upsertReferenceEdges(input: { + workspace_id: string; + edges: Array<{ + source_resource_id: number; + source_version_id: number | null; + target_resource_id: number; + target_version_id: number | null; + edge_type: WorkspaceEdgeType; + quote?: string | null; + }>; + }): void { + const now = currentUnixSeconds(); + const tx = this.db.transaction(() => { + const insertStmt = this.db.prepare( + `INSERT INTO workspace_reference_edges( + workspace_id, source_resource_id, source_version_id, + target_resource_id, target_version_id, edge_type, quote, created_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(workspace_id, source_resource_id, target_resource_id, edge_type) DO UPDATE SET + source_version_id = excluded.source_version_id, + target_version_id = excluded.target_version_id, + quote = excluded.quote`, + ); + for (const edge of input.edges) { + insertStmt.run( + input.workspace_id, + edge.source_resource_id, + edge.source_version_id, + edge.target_resource_id, + edge.target_version_id, + edge.edge_type, + edge.quote ?? null, + now, + ); + } + }); + tx(); + } + + // ------------------------------------------------------------------------- + // Resource listing + // ------------------------------------------------------------------------- + + listResources(input: ListWorkspaceResourcesInput): ListWorkspaceResourcesResult { + const where: string[] = ["workspace_id = ?"]; + const params: Array = [input.workspace_id]; + if (input.resource_type) { + where.push("resource_type = ?"); + params.push(input.resource_type); + } + if (input.index_status) { + where.push("index_status = ?"); + params.push(input.index_status); + } + const limit = Math.max(1, Math.min(200, Math.floor(input.limit ?? 50))); + const offset = Math.max(0, Math.floor(input.offset ?? 0)); + const rows = this.db + .prepare( + `SELECT * FROM workspace_resources + WHERE ${where.join(" AND ")} + ORDER BY updated_at DESC, id DESC + LIMIT ? OFFSET ?`, + ) + .all(...params, limit, offset) as Array; + const totalRow = this.db + .prepare(`SELECT COUNT(*) AS total FROM workspace_resources WHERE ${where.join(" AND ")}`) + .get(...params) as { total: number }; + return { + total: totalRow.total, + resources: rows.map(toWorkspaceResource), + }; + } + + findResourceByCanonicalKey(input: { + workspace_id: string; + canonical_key: string; + }): WorkspaceResource | null { + const row = this.db + .prepare("SELECT * FROM workspace_resources WHERE workspace_id = ? AND canonical_key = ?") + .get(input.workspace_id, input.canonical_key) as WorkspaceResourceRow | undefined; + return row ? toWorkspaceResource(row) : null; + } + + // ------------------------------------------------------------------------- + // Search: lexical + // ------------------------------------------------------------------------- + + searchLexical(input: { + workspace_id: string; + user_id: string; + query: string; + resource_types?: string[]; + limit: number; + }): WorkspaceSearchHit[] { + void input.user_id; + const keywords = tokenizeQuery(input.query); + if (keywords.length === 0) return []; + const ftsQuery = keywords.map((kw) => `"${kw.replace(/"/g, '""')}"`).join(" OR "); + const params: Array = []; + let resourceTypeFilter = ""; + if (input.resource_types && input.resource_types.length > 0) { + resourceTypeFilter = `AND pr.resource_type IN (${input.resource_types.map(() => "?").join(",")})`; + params.push(...input.resource_types); + } + const sql = ` + SELECT pc.chunk_id, + pc.resource_id, + pc.version_id, + pr.resource_type, + pr.title AS resource_title, + pr.canonical_key, + pc.content, + bm25(workspace_chunks_fts) AS bm25_score, + pc.chunk_index + FROM workspace_chunks_fts fts + JOIN workspace_chunks pc ON pc.id = fts.rowid + JOIN workspace_resources pr ON pr.id = pc.resource_id + WHERE workspace_chunks_fts MATCH ? + AND pc.workspace_id = ? + ${resourceTypeFilter} + ORDER BY bm25_score ASC + LIMIT ? + `; + params.unshift(ftsQuery, input.workspace_id, input.limit); + const rows = this.db.prepare(sql).all(...params) as Array<{ + chunk_id: string; + resource_id: number; + version_id: number; + resource_type: string; + resource_title: string; + canonical_key: string; + content: string; + bm25_score: number; + chunk_index: number; + }>; + return rows.map((row) => ({ + chunk_id: row.chunk_id, + resource_id: row.resource_id, + version_id: row.version_id, + resource_type: row.resource_type, + resource_title: row.resource_title, + canonical_key: row.canonical_key, + snippet: buildSnippet(row.content, keywords), + matched_terms: keywords, + score: bm25ToSimilarity(row.bm25_score), + signals: { lexical: bm25ToSimilarity(row.bm25_score) }, + reference_edges: this.edgesForResource(row.resource_id), + })); + } + + // ------------------------------------------------------------------------- + // Search: semantic (sqlite-vec KNN, widen-and-retry) + // ------------------------------------------------------------------------- + + searchSemantic(input: { + workspace_id: string; + user_id: string; + queryEmbedding: number[]; + resource_types?: string[]; + limit: number; + threshold: number; + }): WorkspaceSearchHit[] { + void input.user_id; + const tableName = this.getChildVectorTableName(input.queryEmbedding.length); + if (!this.childVectorTableExists(tableName)) { + // No embeddings written yet — caller should fall back to lexical. + return []; + } + const scanLimit = Math.max(input.limit, input.limit * 4); + const vecKnnMaxK = 4096; + let currentScanLimit = scanLimit; + while (true) { + const vecRows = this.db + .prepare( + `SELECT chunk_id, distance + FROM ${tableName} + WHERE embedding MATCH ? + ORDER BY distance + LIMIT ?`, + ) + .all(floatArrayToBuffer(input.queryEmbedding), currentScanLimit) as Array<{ + chunk_id: string; + distance: number; + }>; + if (vecRows.length === 0) return []; + const resourceTypeFilter = + input.resource_types && input.resource_types.length > 0 + ? `AND pr.resource_type IN (${input.resource_types.map(() => "?").join(",")})` + : ""; + const params: Array = [input.workspace_id, ...vecRows.map((r) => r.chunk_id)]; + if (input.resource_types && input.resource_types.length > 0) { + params.push(...input.resource_types); + } + params.push(input.limit); + const hydrated = this.db + .prepare( + `SELECT pc.chunk_id, + pc.resource_id, + pc.version_id, + pr.resource_type, + pr.title AS resource_title, + pr.canonical_key, + pc.content + FROM workspace_chunks pc + JOIN workspace_resources pr ON pr.id = pc.resource_id + WHERE pc.workspace_id = ? + AND pc.chunk_id IN (${vecRows.map(() => "?").join(",")}) + ${resourceTypeFilter} + ORDER BY pc.id ASC + LIMIT ?`, + ) + .all(...params) as Array<{ + chunk_id: string; + resource_id: number; + version_id: number; + resource_type: string; + resource_title: string; + canonical_key: string; + content: string; + }>; + const byDistance = new Map(vecRows.map((row) => [row.chunk_id, row.distance])); + const candidateHits: WorkspaceSearchHit[] = []; + for (const row of hydrated) { + const distance = byDistance.get(row.chunk_id) ?? Number.POSITIVE_INFINITY; + const similarity = sqliteDistanceToSimilarity(distance); + if (similarity < input.threshold) continue; + candidateHits.push({ + chunk_id: row.chunk_id, + resource_id: row.resource_id, + version_id: row.version_id, + resource_type: row.resource_type, + resource_title: row.resource_title, + canonical_key: row.canonical_key, + snippet: buildSnippet(row.content, []), + matched_terms: [], + score: similarity, + signals: { semantic: similarity }, + reference_edges: this.edgesForResource(row.resource_id), + }); + } + const hits = candidateHits.sort((a, b) => b.score - a.score); + if (hits.length >= input.limit || vecRows.length < currentScanLimit) { + return hits.slice(0, input.limit); + } + if (currentScanLimit >= vecKnnMaxK) { + return hits.slice(0, input.limit); + } + currentScanLimit = Math.min(currentScanLimit * 2, vecKnnMaxK); + } + } + + private childVectorTableExists(name: string): boolean { + if (!this.vectorSearchAvailable) return false; + return Boolean( + this.db.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?").get(name), + ); + } + + // ------------------------------------------------------------------------- + // Cross-file expansion (BFS over workspace_reference_edges) + // ------------------------------------------------------------------------- + + expandNeighbors(input: { + workspace_id: string; + hits: WorkspaceSearchHit[]; + hops: 1 | 2; + limit: number; + }): WorkspaceSearchHit[] { + if (input.hits.length === 0) return input.hits.slice(0, input.limit); + const visited = new Set(input.hits.map((h) => h.resource_id)); + const queue: Array<{ resource_id: number; depth: number }> = input.hits.map((h) => ({ + resource_id: h.resource_id, + depth: 0, + })); + const expansions: WorkspaceSearchHit[] = []; + const edgeBoost = (resourceId: number): number => { + const out = this.db + .prepare( + `SELECT COUNT(*) AS c FROM workspace_reference_edges + WHERE workspace_id = ? AND source_resource_id = ?`, + ) + .get(input.workspace_id, resourceId) as { c: number }; + const inc = this.db + .prepare( + `SELECT COUNT(*) AS c FROM workspace_reference_edges + WHERE workspace_id = ? AND target_resource_id = ?`, + ) + .get(input.workspace_id, resourceId) as { c: number }; + return 0.1 * (out.c + inc.c); + }; + while (queue.length > 0 && expansions.length < input.limit * 2) { + const head = queue.shift(); + if (!head) break; + if (head.depth >= input.hops) continue; + const neighborIds = this.db + .prepare( + `SELECT DISTINCT target_resource_id AS id FROM workspace_reference_edges + WHERE workspace_id = ? AND source_resource_id = ? + UNION + SELECT DISTINCT source_resource_id AS id FROM workspace_reference_edges + WHERE workspace_id = ? AND target_resource_id = ?`, + ) + .all(input.workspace_id, head.resource_id, input.workspace_id, head.resource_id) as Array<{ + id: number; + }>; + for (const { id } of neighborIds) { + if (visited.has(id)) continue; + visited.add(id); + const boost = edgeBoost(id); + const chunkRow = this.db + .prepare( + `SELECT pc.chunk_id, pc.resource_id, pc.version_id, pc.content, pc.chunk_index, + pr.resource_type, pr.title AS resource_title, pr.canonical_key + FROM workspace_chunks pc + JOIN workspace_resources pr ON pr.id = pc.resource_id + WHERE pc.resource_id = ? + ORDER BY pc.chunk_index ASC + LIMIT 1`, + ) + .get(id) as + | { + chunk_id: string; + resource_id: number; + version_id: number; + content: string; + chunk_index: number; + resource_type: string; + resource_title: string; + canonical_key: string; + } + | undefined; + if (!chunkRow) continue; + expansions.push({ + chunk_id: chunkRow.chunk_id, + resource_id: chunkRow.resource_id, + version_id: chunkRow.version_id, + resource_type: chunkRow.resource_type, + resource_title: chunkRow.resource_title, + canonical_key: chunkRow.canonical_key, + snippet: buildSnippet(chunkRow.content, []), + matched_terms: [], + score: boost, + signals: { edge_boost: boost }, + reference_edges: this.edgesForResource(chunkRow.resource_id), + }); + queue.push({ resource_id: id, depth: head.depth + 1 }); + } + } + const seen = new Set(input.hits.map((h) => h.resource_id)); + const dedupedExpansions: WorkspaceSearchHit[] = []; + for (const expansion of expansions) { + if (seen.has(expansion.resource_id)) continue; + seen.add(expansion.resource_id); + dedupedExpansions.push(expansion); + } + const merged = [...input.hits, ...dedupedExpansions] + .sort((a, b) => b.score - a.score) + .slice(0, input.limit); + return merged; + } + + private edgesForResource(resourceId: number): WorkspaceSearchHit["reference_edges"] { + const rows = this.db + .prepare( + `SELECT edge_type, target_resource_id FROM workspace_reference_edges + WHERE source_resource_id = ? OR target_resource_id = ? + LIMIT 16`, + ) + .all(resourceId, resourceId) as Array<{ edge_type: WorkspaceEdgeType; target_resource_id: number }>; + return rows; + } + + // ------------------------------------------------------------------------- + // Soft-delete + // ------------------------------------------------------------------------- + + softDeleteMissingResources(input: { + workspace_id: string; + presentKeys: Set; + }): Array<{ canonical_key: string }> { + const allRows = this.db + .prepare("SELECT id, canonical_key, metadata FROM workspace_resources WHERE workspace_id = ?") + .all(input.workspace_id) as Array<{ id: number; canonical_key: string; metadata: string | null }>; + const missing: Array<{ canonical_key: string }> = []; + const now = currentUnixSeconds(); + for (const row of allRows) { + if (input.presentKeys.has(row.canonical_key)) continue; + const meta = parseJson>(row.metadata, {}) ?? {}; + if (meta.deleted_at) continue; + meta.deleted_at = now; + this.db + .prepare("UPDATE workspace_resources SET metadata = ?, updated_at = ? WHERE id = ?") + .run(stringifyJson(meta), now, row.id); + missing.push({ canonical_key: row.canonical_key }); + } + return missing; + } + + // ------------------------------------------------------------------------- + // Job helpers (used by okf-backend + tests) + // ------------------------------------------------------------------------- + + createJob(input: { workspace_id: string; kind: WorkspaceJob["kind"]; total: number }): WorkspaceJob { + const now = currentUnixSeconds(); + const stmt = this.db + .prepare( + `INSERT INTO workspace_jobs(workspace_id, kind, status, total, done, created_at, updated_at) + VALUES (?, ?, 'pending', ?, 0, ?, ?)`, + ) + .run(input.workspace_id, input.kind, input.total, now, now); + const row = this.db + .prepare("SELECT * FROM workspace_jobs WHERE id = ?") + .get(Number(stmt.lastInsertRowid)) as WorkspaceJobRow; + return toWorkspaceJob(row); + } + + updateJobTotal(jobId: number, total: number): void { + this.db + .prepare("UPDATE workspace_jobs SET total = ?, updated_at = ? WHERE id = ?") + .run(total, currentUnixSeconds(), jobId); + } +} + +function tokenizeQuery(query: string): string[] { + return query + .split(/\s+/u) + .map((token) => token.trim()) + .filter((token) => token.length > 0); +} + +function buildSnippet(content: string, keywords: string[]): string { + const max = 160; + if (content.length <= max) return content; + if (keywords.length === 0) return `${content.slice(0, max)}…`; + const lower = content.toLowerCase(); + for (const keyword of keywords) { + const idx = lower.indexOf(keyword.toLowerCase()); + if (idx >= 0) { + const start = Math.max(0, idx - 40); + const end = Math.min(content.length, idx + keyword.length + 120); + return `${start > 0 ? "…" : ""}${content.slice(start, end)}${end < content.length ? "…" : ""}`; + } + } + return `${content.slice(0, max)}…`; +} + +function bm25ToSimilarity(score: number): number { + if (!Number.isFinite(score)) return 0; + // bm25() returns negative numbers in SQLite FTS5; lower = better. + const normalised = 1 / (1 + Math.abs(score)); + return Math.min(1, Math.max(0, normalised)); +} + +function sqliteDistanceToSimilarity(distance: number): number { + if (!Number.isFinite(distance)) return 0; + // sqlite-vec returns L2 distance. Embeddings are L2-normalised upstream + // (the same assumption `sqliteVectorDistanceToCosineSimilarity` makes in + // `packages/sqlite/src/raw-message-manager.ts`), so the conversion is + // the standard `cosine_similarity = 1 - distance^2 / 2` clamped to [-1, 1]. + return Math.max(-1, 1 - (distance * distance) / 2); +} + +// ------------------------------------------------------------------------- +// Singleton accessor +// ------------------------------------------------------------------------- + +let _instance: SqliteWorkspaceStore | undefined; + +export async function getSQLiteWorkspaceStore( + options: SqliteWorkspaceStoreOptions = {}, +): Promise { + if (!_instance) { + const dbPath = options.dbPath ?? resolveWorkspaceDbPath(); + mkdirSync(dirname(dbPath), { recursive: true }); + const instance = new SqliteWorkspaceStore({ ...options, dbPath }); + await instance.init(); + _instance = instance; + } + return _instance; +} + +export async function closeSQLiteWorkspaceStore(): Promise { + if (!_instance) return; + await _instance.close(); + _instance = undefined; +} + +/** Test-only reset hook; mirrors `__resetSQLiteRawMessageManagerForTests`. */ +export function __resetSQLiteWorkspaceStoreForTests(): void { + _instance = undefined; +} + +/** + * Build a `SqliteWorkspaceStore` from an existing in-memory database handle. + * Convenience for tests that want to share a single connection with the + * raw-message store (e.g. to verify FK / multi-table integration). + */ +export async function createSqliteWorkspaceStore( + options: SqliteWorkspaceStoreOptions = {}, +): Promise { + const instance = new SqliteWorkspaceStore(options); + await instance.init(); + return instance; +} + +// ------------------------------------------------------------------------- +// Public helpers exported for the API layer (api.ts) and OKF backend. +// ------------------------------------------------------------------------- + +export async function runUpdateWorkspaceContext( + _store: SqliteWorkspaceStore, + input: UpdateWorkspaceContextInput & { user_id: string }, + hooks: { + indexOkfFolder: ( + workspace_id: string, + user_id: string, + path: string, + ) => Promise; + }, +): Promise { + return hooks.indexOkfFolder(input.workspace_id, input.user_id, input.path); +} + +export async function runSearchWorkspaceContext( + _store: SqliteWorkspaceStore, + input: SearchWorkspaceContextInput & { user_id: string }, + hooks: { + searchLexical: ( + workspace_id: string, + user_id: string, + query: string, + resource_types?: string[], + limit?: number, + ) => WorkspaceSearchHit[]; + searchSemantic: ( + workspace_id: string, + user_id: string, + queryEmbedding: number[], + resource_types?: string[], + limit?: number, + threshold?: number, + ) => WorkspaceSearchHit[]; + expandNeighbors: ( + workspace_id: string, + hits: WorkspaceSearchHit[], + hops: 1 | 2, + limit: number, + ) => WorkspaceSearchHit[]; + generateEmbedding?: (text: string) => Promise; + fuse?: ( + lexical: WorkspaceSearchHit[], + semantic: WorkspaceSearchHit[], + limit: number, + ) => WorkspaceSearchHit[]; + }, +): Promise { + const strategy: WorkspaceSearchStrategy = input.strategy ?? "hybrid"; + const options = input.options ?? {}; + const limit = Math.max(1, Math.min(50, Math.floor(options.limit ?? 10))); + const threshold = options.threshold ?? 0.7; + const resourceTypes = options.resource_types; + + let hits: WorkspaceSearchHit[] = []; + if (strategy === "lexical") { + hits = hooks.searchLexical(input.workspace_id, input.user_id, input.query, resourceTypes, limit); + } else if (strategy === "semantic") { + if (!hooks.generateEmbedding) { + hits = []; + } else { + const embedding = await hooks.generateEmbedding(input.query); + hits = hooks.searchSemantic( + input.workspace_id, + input.user_id, + embedding, + resourceTypes, + limit, + threshold, + ); + if (hits.length === 0) { + // Semantic fallback to lexical so users still get a hit + // before embeddings are written. + hits = hooks.searchLexical(input.workspace_id, input.user_id, input.query, resourceTypes, limit); + } + } + } else if (strategy === "hybrid") { + const candidateLimit = limit * 4; + const lexicalHits = hooks.searchLexical( + input.workspace_id, + input.user_id, + input.query, + resourceTypes, + candidateLimit, + ); + let semanticHits: WorkspaceSearchHit[] = []; + if (hooks.generateEmbedding) { + const embedding = await hooks.generateEmbedding(input.query); + semanticHits = hooks.searchSemantic( + input.workspace_id, + input.user_id, + embedding, + resourceTypes, + candidateLimit, + threshold, + ); + } + hits = hooks.fuse ? hooks.fuse(lexicalHits, semanticHits, limit) : lexicalHits.slice(0, limit); + } else { + const candidateLimit = limit * 4; + const lexicalHits = hooks.searchLexical( + input.workspace_id, + input.user_id, + input.query, + resourceTypes, + candidateLimit, + ); + let semanticHits: WorkspaceSearchHit[] = []; + if (hooks.generateEmbedding) { + const embedding = await hooks.generateEmbedding(input.query); + semanticHits = hooks.searchSemantic( + input.workspace_id, + input.user_id, + embedding, + resourceTypes, + candidateLimit, + threshold, + ); + } + const fused = hooks.fuse ? hooks.fuse(lexicalHits, semanticHits, candidateLimit) : lexicalHits; + hits = hooks.expandNeighbors(input.workspace_id, fused, options.hops ?? 1, limit); + } + return { + query: input.query, + strategy, + total: hits.length, + hits, + }; +} diff --git a/packages/workspace/src/types.ts b/packages/workspace/src/types.ts new file mode 100644 index 00000000..fa60e9e9 --- /dev/null +++ b/packages/workspace/src/types.ts @@ -0,0 +1,183 @@ +/** + * `@melandlabs/workspace` — core types. + * + * All cross-package inputs / outputs flow through this module so the + * API surface (`updateWorkspaceContext` / `searchWorkspaceContext` / + * `listWorkspaceResources`) and the HTTP / MCP wiring can share one + * canonical definition. + */ + +import type { OkfFrontMatter } from "@melandlabs/contracts"; + +export interface RuntimeContext { + user_id: string; + employee_id?: string; + session_id?: string; + request_id: string; + mode?: "sandbox" | "client"; + auth_token?: string; +} + +export type WorkspaceStorageKind = "okf_local_dir" | "inline"; +export type WorkspaceEdgeType = "cites" | "supersedes" | "amends" | "relates-to"; +export type WorkspaceIndexStatus = "pending" | "partial" | "ready" | "failed"; +export type WorkspaceSearchStrategy = "lexical" | "semantic" | "hybrid" | "cross-file"; + +export interface WorkspaceResource { + id: number; + workspace_id: string; + user_id: string; + resource_type: string; + canonical_key: string; + title: string; + storage_kind: WorkspaceStorageKind; + current_version_id: number | null; + index_status: WorkspaceIndexStatus; + created_at: number; + updated_at: number; + metadata?: Record; +} + +export interface WorkspaceResourceVersion { + id: number; + resource_id: number; + version_number: number; + sha256: string; + change_kind: "created" | "modified" | "unchanged"; + size_bytes: number; + parent_version_id: number | null; + source_path: string | null; + created_at: number; + metadata?: Record; +} + +export interface WorkspaceChunk { + id: number; + chunk_id: string; + resource_id: number; + version_id: number; + workspace_id: string; + chunk_index: number; + chunk_count: number; + start_position: number; + end_position: number; + content: string; + content_hash: string; + embedding?: number[]; + embedding_model?: string; + embedding_dimensions?: number; + embedding_updated_at?: number; +} + +export interface WorkspaceReferenceEdge { + id: number; + workspace_id: string; + source_resource_id: number; + source_version_id: number | null; + target_resource_id: number; + target_version_id: number | null; + edge_type: WorkspaceEdgeType; + quote?: string | null; + created_at: number; +} + +export interface WorkspaceJob { + id: number; + workspace_id: string; + kind: "index" | "update"; + status: WorkspaceIndexStatus; + total: number; + done: number; + error: string | null; + created_at: number; + updated_at: number; +} + +export interface WorkspaceSearchHit { + chunk_id: string; + resource_id: number; + version_id: number; + resource_type: string; + resource_title: string; + canonical_key: string; + snippet: string; + matched_terms: string[]; + score: number; + signals: { + lexical?: number; + semantic?: number; + edge_boost?: number; + }; + reference_edges: Array<{ edge_type: WorkspaceEdgeType; target_resource_id: number }>; +} + +export interface SearchWorkspaceContextOptions { + /** Default 10, max 50. */ + limit?: number; + resource_types?: string[]; + /** Default 0.7. */ + threshold?: number; + /** Only honoured by the `cross-file` strategy. */ + include_edges?: boolean; + /** Only honoured by the `cross-file` strategy. */ + hops?: 1 | 2; +} + +export interface UpdateWorkspaceContextInput { + workspace_id: string; + source: "okf_folder"; + path: string; + metadata?: Record; +} + +export interface UpdateWorkspaceContextResult { + jobId: number; + triggered: true; + status: WorkspaceIndexStatus; + filesScanned: number; + filesAdded: number; + filesModified: number; + filesUnchanged: number; + filesDeleted: number; +} + +export interface SearchWorkspaceContextInput { + workspace_id: string; + query: string; + strategy?: WorkspaceSearchStrategy; + options?: SearchWorkspaceContextOptions; +} + +export interface SearchWorkspaceContextResult { + query: string; + strategy: WorkspaceSearchStrategy; + total: number; + hits: WorkspaceSearchHit[]; +} + +export interface ListWorkspaceResourcesInput { + workspace_id: string; + resource_type?: string; + index_status?: WorkspaceIndexStatus; + limit?: number; + offset?: number; +} + +export interface ListWorkspaceResourcesResult { + total: number; + resources: WorkspaceResource[]; +} + +/** + * Internal: the resource shape produced by `okf-backend.listOkfFolderResources` + * and consumed by `SqliteWorkspaceStore.indexResource`. + */ +export interface OkfFolderResource { + canonical_key: string; + absolute_path: string; + title: string; + resource_type: string; + body: string; + front_matter?: OkfFrontMatter; + size_bytes: number; +} diff --git a/packages/workspace/test/okf-backend.test.ts b/packages/workspace/test/okf-backend.test.ts new file mode 100644 index 00000000..4a3cbef2 --- /dev/null +++ b/packages/workspace/test/okf-backend.test.ts @@ -0,0 +1,123 @@ +/** + * Tests for `okf-backend.indexOkfFolder` — sha256 dedup, cites-edge + * extraction, and soft-delete detection across re-runs. + */ +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { indexOkfFolder } from "../src/okf-backend"; +import { SqliteWorkspaceStore } from "../src/sqlite"; + +let scratchDir: string; + +beforeEach(() => { + scratchDir = mkdtempSync(join(tmpdir(), "workspace-okf-")); +}); + +afterEach(() => { + rmSync(scratchDir, { recursive: true, force: true }); +}); + +function writeFile(folder: string, relativePath: string, content: string): void { + const absolute = join(folder, relativePath); + mkdirSync(join(absolute, ".."), { recursive: true }); + writeFileSync(absolute, content, "utf8"); +} + +describe("indexOkfFolder", () => { + it("indexes every .md file, computes sha256 dedup, and persists cites edges", async () => { + const okfRoot = join(scratchDir, "wiki"); + mkdirSync(okfRoot, { recursive: true }); + writeFile( + okfRoot, + "contract.md", + "---\ntype: Reference\n---\n# Contract\n\nSee [law](./law.md) for the governing provision.\n", + ); + writeFile( + okfRoot, + "law.md", + "---\ntype: Reference\n---\n# Law\n\nCivil code article 123 is the controlling clause.\n", + ); + writeFile( + okfRoot, + "policy.md", + "---\ntype: Reference\n---\n# Policy\n\nThis policy is not linked from any other document.\n", + ); + + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + const enqueued: Array<{ resource_id: number; version_id: number }> = []; + const result = await indexOkfFolder(store, { + workspace_id: "p1", + user_id: "u1", + path: okfRoot, + enqueueEmbedding: async (input) => { + enqueued.push({ resource_id: input.resource_id, version_id: input.version_id }); + }, + }); + expect(result.filesScanned).toBe(3); + expect(result.filesAdded).toBe(3); + expect(result.filesModified).toBe(0); + expect(result.filesUnchanged).toBe(0); + expect(enqueued.length).toBe(3); + // Re-run: every file should be unchanged. + const second = await indexOkfFolder(store, { + workspace_id: "p1", + user_id: "u1", + path: okfRoot, + enqueueEmbedding: async () => {}, + }); + expect(second.filesScanned).toBe(3); + expect(second.filesAdded).toBe(0); + expect(second.filesModified).toBe(0); + expect(second.filesUnchanged).toBe(3); + // Cites edge: contract → law. + const edgeRows = store.__testDb + .prepare( + `SELECT pr1.canonical_key AS source, pr2.canonical_key AS target, edge_type + FROM workspace_reference_edges e + JOIN workspace_resources pr1 ON pr1.id = e.source_resource_id + JOIN workspace_resources pr2 ON pr2.id = e.target_resource_id + WHERE edge_type = 'cites'`, + ) + .all() as Array<{ source: string; target: string; edge_type: string }>; + const contractLaw = edgeRows.find((row) => row.source === "contract.md" && row.target === "law.md"); + expect(contractLaw).toBeDefined(); + await store.close(); + }); + + it("soft-deletes a resource whose file disappeared on a re-run", async () => { + const okfRoot = join(scratchDir, "wiki"); + mkdirSync(okfRoot, { recursive: true }); + writeFile(okfRoot, "keep.md", "keep me"); + writeFile(okfRoot, "drop.md", "delete me next run"); + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + const first = await indexOkfFolder(store, { + workspace_id: "p1", + user_id: "u1", + path: okfRoot, + enqueueEmbedding: async () => {}, + }); + expect(first.filesAdded).toBe(2); + rmSync(join(okfRoot, "drop.md")); + const second = await indexOkfFolder(store, { + workspace_id: "p1", + user_id: "u1", + path: okfRoot, + enqueueEmbedding: async () => {}, + }); + expect(second.filesDeleted).toBe(1); + const rows = store.__testDb + .prepare("SELECT canonical_key, metadata FROM workspace_resources ORDER BY canonical_key") + .all() as Array<{ canonical_key: string; metadata: string | null }>; + const dropped = rows.find((row) => row.canonical_key === "drop.md"); + expect(dropped).toBeDefined(); + const meta = JSON.parse(dropped?.metadata ?? "{}") as { deleted_at?: number }; + expect(typeof meta.deleted_at).toBe("number"); + await store.close(); + }); +}); diff --git a/packages/workspace/test/parsers-adapter.test.ts b/packages/workspace/test/parsers-adapter.test.ts new file mode 100644 index 00000000..01c30176 --- /dev/null +++ b/packages/workspace/test/parsers-adapter.test.ts @@ -0,0 +1,55 @@ +/** + * Tests for `parsers-adapter.extractText`. Validates the text-extraction + * path for plain `.md` / `.txt` files (which should be a UTF-8 pass-through) + * and that the MIME detection is stable across file extensions. + */ +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { detectMimeType, extractText } from "../src/parsers-adapter"; + +let scratchDir: string; + +beforeEach(() => { + scratchDir = mkdtempSync(join(tmpdir(), "workspace-parsers-")); +}); + +afterEach(() => { + rmSync(scratchDir, { recursive: true, force: true }); +}); + +describe("parsers-adapter", () => { + it("detects MIME types by extension", () => { + expect(detectMimeType("a.md")).toBe("text/markdown"); + expect(detectMimeType("a.markdown")).toBe("text/markdown"); + expect(detectMimeType("a.txt")).toBe("text/plain"); + expect(detectMimeType("a.pdf")).toBe("application/pdf"); + expect(detectMimeType("a.docx")).toBe( + "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + ); + expect(detectMimeType("a.pages")).toBe("application/x-iwork-pages-sffpages"); + expect(detectMimeType("a.unknown")).toBe("application/octet-stream"); + }); + + it("extracts text from a plain markdown file", async () => { + const filePath = join(scratchDir, "note.md"); + mkdirSync(scratchDir, { recursive: true }); + writeFileSync(filePath, "# Title\n\nThis is the body.", "utf8"); + const result = await extractText(filePath); + expect(result.text).toContain("# Title"); + expect(result.text).toContain("This is the body."); + expect(result.mimeType).toBe("text/markdown"); + }); + + it("extracts text from a plain text file", async () => { + const filePath = join(scratchDir, "notes.txt"); + mkdirSync(scratchDir, { recursive: true }); + writeFileSync(filePath, "first line\nsecond line\n", "utf8"); + const result = await extractText(filePath); + expect(result.text).toBe("first line\nsecond line\n"); + expect(result.mimeType).toBe("text/plain"); + }); +}); diff --git a/packages/workspace/test/search.test.ts b/packages/workspace/test/search.test.ts new file mode 100644 index 00000000..c4ee5019 --- /dev/null +++ b/packages/workspace/test/search.test.ts @@ -0,0 +1,200 @@ +/** + * Tests for the lexical / semantic / hybrid / cross-file search pipeline. + * Embeddings are mocked with deterministic 1536-dim vectors so we never hit + * OpenRouter and the distance ordering is reproducible. + */ +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { searchWorkspaceContext } from "../src/api"; +import { searchCrossFile } from "../src/search/cross-file"; +import { fuseHybridHits } from "../src/search/hybrid"; +import { searchLexical } from "../src/search/lexical"; +import { searchSemantic } from "../src/search/semantic"; +import { SqliteWorkspaceStore } from "../src/sqlite"; +import type { OkfFolderResource, WorkspaceSearchHit } from "../src/types"; + +let scratchDir: string; + +beforeEach(() => { + scratchDir = mkdtempSync(join(tmpdir(), "workspace-search-")); +}); + +afterEach(() => { + rmSync(scratchDir, { recursive: true, force: true }); +}); + +function makeResource( + overrides: Partial & Pick, +): OkfFolderResource { + return { + absolute_path: `/fake/${overrides.canonical_key}`, + title: overrides.canonical_key, + resource_type: overrides.canonical_key.endsWith(".md") ? "note" : "document", + size_bytes: overrides.body.length, + ...overrides, + }; +} + +const DIMS = 1536; + +function makeDeterministicVector(seed: number): number[] { + // Push the vector onto a unit sphere so cosine distance is + // reproducible. Two resources whose `seed` shares any high bits get + // similar vectors, which mirrors "documents on the same topic". + const vec = new Array(DIMS).fill(0); + let state = seed; + for (let i = 0; i < DIMS; i += 1) { + state = (state * 1103515245 + 12345) >>> 0; + vec[i] = (state / 0xffffffff) * 2 - 1; + } + let norm = 0; + for (const v of vec) norm += v * v; + norm = Math.sqrt(norm) || 1; + return vec.map((v) => v / norm); +} + +async function indexThreeDocuments(store: SqliteWorkspaceStore): Promise { + await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ + canonical_key: "Reference/contract.md", + body: "limitation of liability clauses must conform to applicable law".trim(), + }), + }); + await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ + canonical_key: "Reference/law.md", + body: "civil code article 123: parties may limit liability unless the law forbids it".trim(), + }), + }); + await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ + canonical_key: "Reference/cookie-policy.md", + body: "this document describes cookie usage on the marketing site".trim(), + }), + }); +} + +describe("search pipeline", () => { + it("returns BM25-ranked lexical hits for matching keywords", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + await indexThreeDocuments(store); + const hits = searchLexical(store, { + workspace_id: "p1", + user_id: "u1", + query: "limitation liability", + limit: 5, + }); + expect(hits.length).toBeGreaterThan(0); + expect(hits[0]?.resource_title).toContain("contract.md"); + expect(hits[0]?.signals.lexical).toBeGreaterThan(0); + await store.close(); + }); + + it("returns no semantic hits when embeddings haven't been written", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + await indexThreeDocuments(store); + const hits = searchSemantic(store, { + workspace_id: "p1", + user_id: "u1", + queryEmbedding: makeDeterministicVector(1), + limit: 5, + threshold: 0.0, + }); + expect(hits).toEqual([]); + await store.close(); + }); + + it("fuses lexical + semantic candidates via RRF (k=60)", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + await indexThreeDocuments(store); + const lexical = searchLexical(store, { + workspace_id: "p1", + user_id: "u1", + query: "limitation", + limit: 5, + }); + // Fabricate semantic hits with monotonically decreasing scores so + // the RRF fusion ranks the top one highest. + const semantic: WorkspaceSearchHit[] = lexical.map((hit, index) => ({ + ...hit, + score: 0.9 - index * 0.1, + signals: { semantic: 0.9 - index * 0.1 }, + })); + const fused = fuseHybridHits({ lexical, semantic, limit: 3 }); + expect(fused.length).toBeGreaterThan(0); + expect(fused[0]?.score).toBeGreaterThanOrEqual(fused[1]?.score ?? 0); + await store.close(); + }); + + it("expands hits via cites edges for cross-file search", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + const contract = await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "Reference/contract.md", body: "limitation clause text" }), + }); + const law = await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "Reference/law.md", body: "civil code article 123" }), + }); + store.upsertReferenceEdges({ + workspace_id: "p1", + edges: [ + { + source_resource_id: contract.resource_id, + source_version_id: contract.version_id, + target_resource_id: law.resource_id, + target_version_id: law.version_id, + edge_type: "cites", + }, + ], + }); + const hits = await searchCrossFile(store, { + workspace_id: "p1", + user_id: "u1", + query: "limitation", + limit: 5, + lexicalSearch: (params) => searchLexical(store, params), + semanticSearch: (params) => searchSemantic(store, params), + }); + expect(hits.length).toBeGreaterThan(0); + const lawHit = hits.find((hit) => hit.resource_id === law.resource_id); + expect(lawHit).toBeDefined(); + expect(lawHit?.signals.edge_boost).toBeGreaterThan(0); + await store.close(); + }); + + it("falls back to lexical-only when embeddings fail during hybrid search", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + await indexThreeDocuments(store); + const result = await searchWorkspaceContext( + { user_id: "u1", request_id: "req-1" }, + store, + { workspace_id: "p1", query: "limitation", strategy: "hybrid" }, + { + embed: async () => { + throw new Error("simulated embedding failure"); + }, + }, + ); + expect(result.total).toBeGreaterThan(0); + expect(result.strategy).toBe("hybrid"); + await store.close(); + }); +}); diff --git a/packages/workspace/test/sqlite-project-store.test.ts b/packages/workspace/test/sqlite-project-store.test.ts new file mode 100644 index 00000000..93882f73 --- /dev/null +++ b/packages/workspace/test/sqlite-project-store.test.ts @@ -0,0 +1,159 @@ +/** + * Tests for `SqliteWorkspaceStore` — schema bootstrap, resource/version + * chain, sha256 dedup, unchanged short-circuit, and version parent + * linkage. + */ +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { SqliteWorkspaceStore } from "../src/sqlite"; +import type { OkfFolderResource } from "../src/types"; + +let scratchDir: string; + +beforeEach(() => { + scratchDir = mkdtempSync(join(tmpdir(), "workspace-store-")); +}); + +afterEach(() => { + rmSync(scratchDir, { recursive: true, force: true }); +}); + +function makeResource( + overrides: Partial & Pick, +): OkfFolderResource { + return { + absolute_path: `/fake/${overrides.canonical_key}`, + title: overrides.canonical_key, + resource_type: "note", + size_bytes: overrides.body.length, + ...overrides, + }; +} + +describe("SqliteWorkspaceStore", () => { + it("initialises every schema table on init()", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + const tables = ( + store as unknown as { __testDb: { prepare: (s: string) => { all: () => unknown[] } } } + ).__testDb + .prepare(`SELECT name FROM sqlite_master WHERE type IN ('table', 'view', 'trigger')`) + .all() as Array<{ name: string }>; + const names = tables.map((row) => row.name); + expect(names).toContain("workspace_resources"); + expect(names).toContain("workspace_resource_versions"); + expect(names).toContain("workspace_chunks"); + expect(names).toContain("workspace_chunks_fts"); + expect(names).toContain("workspace_reference_edges"); + expect(names).toContain("workspace_jobs"); + await store.close(); + }); + + it("creates a resource + version + chunks on first indexResource", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + const result = await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ + canonical_key: "docs/intro.md", + body: "Hello world. This is the first document for the project.", + }), + }); + expect(result.change_kind).toBe("created"); + expect(result.resource_id).toBeGreaterThan(0); + const chunks = store.__testDb + .prepare("SELECT chunk_index, chunk_count, content FROM workspace_chunks ORDER BY chunk_index") + .all() as Array<{ chunk_index: number; chunk_count: number; content: string }>; + expect(chunks.length).toBeGreaterThan(0); + expect(chunks[0]?.chunk_count).toBe(chunks.length); + await store.close(); + }); + + it("short-circuits with change_kind='unchanged' on identical re-index", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + const body = "Identical body for sha256 dedup test."; + const first = await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "docs/notes.md", body }), + }); + expect(first.change_kind).toBe("created"); + const second = await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "docs/notes.md", body }), + }); + expect(second.change_kind).toBe("unchanged"); + expect(second.version_id).toBe(first.version_id); + await store.close(); + }); + + it("creates a new version with parent_version_id on modified content", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + const first = await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "docs/changelog.md", body: "version 1 body" }), + }); + const second = await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "docs/changelog.md", body: "version 2 body — modified!" }), + }); + expect(second.change_kind).toBe("modified"); + expect(second.version_id).not.toBe(first.version_id); + const versions = store.__testDb + .prepare( + "SELECT id, version_number, parent_version_id FROM workspace_resource_versions ORDER BY version_number", + ) + .all() as Array<{ id: number; version_number: number; parent_version_id: number | null }>; + expect(versions.length).toBe(2); + expect(versions[1]?.parent_version_id).toBe(versions[0]?.id ?? null); + await store.close(); + }); + + it("soft-deletes missing resources when their canonical_key disappears", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "docs/a.md", body: "alpha" }), + }); + await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "docs/b.md", body: "bravo" }), + }); + const missing = store.softDeleteMissingResources({ + workspace_id: "p1", + presentKeys: new Set(["docs/a.md"]), + }); + expect(missing.map((entry) => entry.canonical_key)).toEqual(["docs/b.md"]); + const afterRows = store.__testDb + .prepare("SELECT canonical_key, metadata FROM workspace_resources ORDER BY canonical_key") + .all() as Array<{ canonical_key: string; metadata: string | null }>; + const deletedRow = afterRows.find((row) => row.canonical_key === "docs/b.md"); + expect(deletedRow).toBeDefined(); + const meta = JSON.parse(deletedRow?.metadata ?? "{}") as { deleted_at?: number }; + expect(typeof meta.deleted_at).toBe("number"); + await store.close(); + }); + + it("reuses init() safely across multiple new instances on the same DB", async () => { + const dbPath = join(scratchDir, "store.db"); + const first = new SqliteWorkspaceStore({ dbPath }); + await first.init(); + await first.close(); + const second = new SqliteWorkspaceStore({ dbPath }); + await expect(second.init()).resolves.toBeUndefined(); + await second.close(); + }); +}); diff --git a/packages/workspace/tsconfig.json b/packages/workspace/tsconfig.json new file mode 100644 index 00000000..b6545e66 --- /dev/null +++ b/packages/workspace/tsconfig.json @@ -0,0 +1,11 @@ +{ + "extends": "../config/src/tsconfig.json", + "compilerOptions": { + "outDir": "./dist", + "rootDir": "..", + "incremental": false, + "tsBuildInfoFile": "./dist/.tsbuildinfo", + "noEmit": false + }, + "include": ["src"] +} diff --git a/packages/workspace/tsup.config.ts b/packages/workspace/tsup.config.ts new file mode 100644 index 00000000..04b6a74f --- /dev/null +++ b/packages/workspace/tsup.config.ts @@ -0,0 +1,40 @@ +import { defineConfig } from "tsup"; + +/** + * Multi-entry build for `@melandlabs/workspace`. Two entry points: + * + * - `index` — public surface (api, types, schema, sqlite helpers) + * - `cli` — `opencontext workspace …` subcommand for local indexing + search + * - `sqlite` — convenience entry so tests and internal callers can + * `import "@melandlabs/workspace/sqlite"` without dragging + * the rest of the barrel + */ +export default defineConfig({ + entry: { + index: "src/index.ts", + sqlite: "src/sqlite.ts", + cli: "src/cli.ts", + }, + format: ["esm"], + dts: true, + sourcemap: false, + clean: true, + splitting: false, + treeshake: true, + external: [ + "react", + "react-dom", + "better-sqlite3", + "sqlite-vec", + "hono", + "zod", + "@modelcontextprotocol/sdk", + "@hono/node-server", + "@melandlabs/ai-rag", + "@melandlabs/contracts", + "@melandlabs/okf", + "@melandlabs/rag", + "@melandlabs/shared", + "@melandlabs/sqlite", + ], +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 1a2b342a..377c1bb7 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -378,7 +378,7 @@ importers: version: 3.8.1 '@langchain/community': specifier: ^1.1.1 - version: 1.1.29(382ed5f2a3e6bc1e3575eeff382cf5cb) + version: 1.1.29(8e07a13ecb0caa2defe560ab6fa2e6f8) '@langchain/core': specifier: ^1.1.8 version: 1.2.5(@opentelemetry/api@1.9.0)(openai@4.104.0(encoding@0.1.13)(ws@8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(zod@4.4.3))(ws@8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10)) @@ -1187,6 +1187,9 @@ importers: '@melandlabs/security': specifier: ^0.3.0 version: link:../security + '@melandlabs/workspace': + specifier: ^0.1.0 + version: link:../workspace '@modelcontextprotocol/sdk': specifier: ^1.25.3 version: 1.30.0(@cfworker/json-schema@4.1.1)(zod@4.4.3) @@ -1313,7 +1316,7 @@ importers: dependencies: '@langchain/community': specifier: ^1.1.1 - version: 1.1.29(905166cb4cd0093f294357964e3103f4) + version: 1.1.29(473a380aa82688979edbb67c4e0acfa5) '@langchain/core': specifier: ^1.1.8 version: 1.2.5(@opentelemetry/api@1.9.0)(openai@4.104.0(encoding@0.1.13)(ws@8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(zod@4.4.3))(ws@8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10)) @@ -1519,6 +1522,70 @@ importers: specifier: ^2.1.0 version: 2.1.9(@types/node@26.2.0)(jiti@2.7.0)(jsdom@24.1.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(tsx@4.23.12)(yaml@2.9.0) + packages/workspace: + dependencies: + '@melandlabs/contracts': + specifier: workspace:* + version: link:../contracts + '@melandlabs/env-config': + specifier: workspace:* + version: link:../env-config + '@melandlabs/okf': + specifier: workspace:* + version: link:../okf + '@melandlabs/rag': + specifier: workspace:* + version: link:../rag + '@melandlabs/shared': + specifier: workspace:* + version: link:../shared + '@melandlabs/sqlite': + specifier: workspace:* + version: link:../sqlite + '@modelcontextprotocol/sdk': + specifier: ^1.25.3 + version: 1.30.0(@cfworker/json-schema@4.1.1)(zod@4.4.3) + better-sqlite3: + specifier: ^11.10.0 + version: 11.10.0 + hono: + specifier: ^4.6.14 + version: 4.13.1 + mammoth: + specifier: ^1.11.0 + version: 1.12.2 + pdf-parse: + specifier: ^2.0.0 + version: 2.4.5 + sqlite-vec: + specifier: ^0.1.9 + version: 0.1.9 + xlsx: + specifier: ^0.18.5 + version: 0.18.5 + zod: + specifier: ^4.3.6 + version: 4.4.3 + devDependencies: + '@melandlabs/ai-rag': + specifier: workspace:* + version: link:../ai/rag + '@melandlabs/config': + specifier: workspace:* + version: link:../config + '@types/better-sqlite3': + specifier: ^7.6.13 + version: 7.6.13 + '@types/node': + specifier: ^22.13.10 + version: 22.20.1 + typescript: + specifier: ^5.6.3 + version: 5.9.3 + vitest: + specifier: ^4.1.10 + version: 4.1.10(@opentelemetry/api@1.9.0)(@types/node@22.20.1)(jsdom@24.1.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(vite@6.4.3(@types/node@22.20.1)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0)) + packages: '@agentclientprotocol/sdk@0.22.1': @@ -3396,54 +3463,108 @@ packages: cpu: [arm64] os: [android] + '@napi-rs/canvas-android-arm64@0.1.80': + resolution: {integrity: sha512-sk7xhN/MoXeuExlggf91pNziBxLPVUqF2CAVnB57KLG/pz7+U5TKG8eXdc3pm0d7Od0WreB6ZKLj37sX9muGOQ==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [android] + '@napi-rs/canvas-darwin-arm64@0.1.100': resolution: {integrity: sha512-2PcswRaC7Ly645DGt88///zuFDhJxJYdKAs1uU3mfk1atYkXufgcgLfBpk6Tm12nCQBaNt1wpybuPZ4qOhTo8A==} engines: {node: '>= 10'} cpu: [arm64] os: [darwin] + '@napi-rs/canvas-darwin-arm64@0.1.80': + resolution: {integrity: sha512-O64APRTXRUiAz0P8gErkfEr3lipLJgM6pjATwavZ22ebhjYl/SUbpgM0xcWPQBNMP1n29afAC/Us5PX1vg+JNQ==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [darwin] + '@napi-rs/canvas-darwin-x64@0.1.100': resolution: {integrity: sha512-ePNZtj7pNIva/siZMg+HmbeozkIjqUIYdoymH8HaA3qK7LfzFN4WMBM8G6HQ9ZC+H3+Dnn5pqtiXpgLykaPOhw==} engines: {node: '>= 10'} cpu: [x64] os: [darwin] + '@napi-rs/canvas-darwin-x64@0.1.80': + resolution: {integrity: sha512-FqqSU7qFce0Cp3pwnTjVkKjjOtxMqRe6lmINxpIZYaZNnVI0H5FtsaraZJ36SiTHNjZlUB69/HhxNDT1Aaa9vA==} + engines: {node: '>= 10'} + cpu: [x64] + os: [darwin] + '@napi-rs/canvas-linux-arm-gnueabihf@0.1.100': resolution: {integrity: sha512-d5cDB48oWFGU8/XPhUOFAlySgb/VAu7D+s8fi55K1Pcfg8aPplHWqMgibhVLU8ky7Pyg/fuiVLz4Nf3JrSTuUA==} engines: {node: '>= 10'} cpu: [arm] os: [linux] + '@napi-rs/canvas-linux-arm-gnueabihf@0.1.80': + resolution: {integrity: sha512-eyWz0ddBDQc7/JbAtY4OtZ5SpK8tR4JsCYEZjCE3dI8pqoWUC8oMwYSBGCYfsx2w47cQgQCgMVRVTFiiO38hHQ==} + engines: {node: '>= 10'} + cpu: [arm] + os: [linux] + '@napi-rs/canvas-linux-arm64-gnu@0.1.100': resolution: {integrity: sha512-rDxgxRu69RvDlX/bh9o22DxLsGr8EqsNgotL9+RwQE1S0b0cqeatqsw6aW45mukm0B42DIAaAacKaYQ8cqS1nw==} engines: {node: '>= 10'} cpu: [arm64] os: [linux] + '@napi-rs/canvas-linux-arm64-gnu@0.1.80': + resolution: {integrity: sha512-qwA63t8A86bnxhuA/GwOkK3jvb+XTQaTiVML0vAWoHyoZYTjNs7BzoOONDgTnNtr8/yHrq64XXzUoLqDzU+Uuw==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [linux] + '@napi-rs/canvas-linux-arm64-musl@0.1.100': resolution: {integrity: sha512-K3mDW66N+xT2/V439u1alFANiBUjdEx2gLiNYnCmUsva5jZMxWTjafBYwTzYK+EMFMHrUoabuU+T1BIP5CgbYQ==} engines: {node: '>= 10'} cpu: [arm64] os: [linux] + '@napi-rs/canvas-linux-arm64-musl@0.1.80': + resolution: {integrity: sha512-1XbCOz/ymhj24lFaIXtWnwv/6eFHXDrjP0jYkc6iHQ9q8oXKzUX1Lc6bu+wuGiLhGh2GS/2JlfORC5ZcXimRcg==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [linux] + '@napi-rs/canvas-linux-riscv64-gnu@0.1.100': resolution: {integrity: sha512-mooqUBTIsccZpnoQC4NgrC1v6C1vof39etLNMnBwCY+p0gajWJvAHLGQ6g/gGyS5YrpDW+GefSN4+Cvcr08UWw==} engines: {node: '>= 10'} cpu: [riscv64] os: [linux] + '@napi-rs/canvas-linux-riscv64-gnu@0.1.80': + resolution: {integrity: sha512-XTzR125w5ZMs0lJcxRlS1K3P5RaZ9RmUsPtd1uGt+EfDyYMu4c6SEROYsxyatbbu/2+lPe7MPHOO/0a0x7L/gw==} + engines: {node: '>= 10'} + cpu: [riscv64] + os: [linux] + '@napi-rs/canvas-linux-x64-gnu@0.1.100': resolution: {integrity: sha512-1eCvkDCazm7FFhsT7DfGOdSaHgZVK3bt/dSBl5EWHOWmnz+I7j8tPseJqqD81NF+MH21jKUK4wQSDjN0mdhnTg==} engines: {node: '>= 10'} cpu: [x64] os: [linux] + '@napi-rs/canvas-linux-x64-gnu@0.1.80': + resolution: {integrity: sha512-BeXAmhKg1kX3UCrJsYbdQd3hIMDH/K6HnP/pG2LuITaXhXBiNdh//TVVVVCBbJzVQaV5gK/4ZOCMrQW9mvuTqA==} + engines: {node: '>= 10'} + cpu: [x64] + os: [linux] + '@napi-rs/canvas-linux-x64-musl@0.1.100': resolution: {integrity: sha512-20arT6lnI19S68qNlii73TSEDbECNgzMz2EpldC1V3mZFuRkeujXkcebRk0LRJe9SEUAooYiLokfMViY8IX7yA==} engines: {node: '>= 10'} cpu: [x64] os: [linux] + '@napi-rs/canvas-linux-x64-musl@0.1.80': + resolution: {integrity: sha512-x0XvZWdHbkgdgucJsRxprX/4o4sEed7qo9rCQA9ugiS9qE2QvP0RIiEugtZhfLH3cyI+jIRFJHV4Fuz+1BHHMg==} + engines: {node: '>= 10'} + cpu: [x64] + os: [linux] + '@napi-rs/canvas-win32-arm64-msvc@0.1.100': resolution: {integrity: sha512-DZFFT1wIAg37LJw37yhMRFfjATd3vTQzjZ1Yki8u2vhO6Hi5VE6BVaGQ1aaDu7xb4iMErz+9EOwjpS7xcxFeBw==} engines: {node: '>= 10'} @@ -3456,10 +3577,20 @@ packages: cpu: [x64] os: [win32] + '@napi-rs/canvas-win32-x64-msvc@0.1.80': + resolution: {integrity: sha512-Z8jPsM6df5V8B1HrCHB05+bDiCxjE9QA//3YrkKIdVDEwn5RKaqOxCJDRJkl48cJbylcrJbW4HxZbTte8juuPg==} + engines: {node: '>= 10'} + cpu: [x64] + os: [win32] + '@napi-rs/canvas@0.1.100': resolution: {integrity: sha512-xglYA6q3XO5P3BNJYxVZ1IV7DLVjp1Py6nwag88YntrS+3vKHyYcMqXVS4ZztJmwz2uGvz1FWhI/4LgbR5uQDA==} engines: {node: '>= 10'} + '@napi-rs/canvas@0.1.80': + resolution: {integrity: sha512-DxuT1ClnIPts1kQx8FBmkk4BQDTfI5kIzywAaMjQSXfNnra5UFU9PwurXrl+Je3bJ6BGsp/zmshVVFbCmyI+ww==} + engines: {node: '>= 10'} + '@napi-rs/lzma-linux-x64-gnu@1.5.1': resolution: {integrity: sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==} engines: {node: ^22.20 || ^24.12 || >=25} @@ -3969,6 +4100,10 @@ packages: node-fetch: optional: true + '@xmldom/xmldom@0.8.15': + resolution: {integrity: sha512-/5NV/vDALVFDXgLmfsy9TRCBlKwO2LNBFzpzvb9iIj+jR+eSc6DLYYvVOdivT/jm7MtU6TebYuRmzEOI7w40UA==} + engines: {node: '>=10.0.0'} + '@xterm/xterm@5.5.0': resolution: {integrity: sha512-hqJHYaQb5OptNunnyAnkHyM8aCjZ1MEIDTQu1iIbbTD/xops91NB5yq1ZK/dC2JDbVWtF23zUtl9JE2NqwT87A==} @@ -3996,6 +4131,10 @@ packages: engines: {node: '>=0.4.0'} hasBin: true + adler-32@1.3.1: + resolution: {integrity: sha512-ynZ4w/nUUv5rrsR8UUGoe1VC9hZj6V5hU9Qw1HlMDJGEJw5S7TfTErWTjMys6M7vr0YWcPqs3qAr4ss0nDfP+A==} + engines: {node: '>=0.8'} + agent-base@6.0.2: resolution: {integrity: sha512-RZNwNclF7+MS/8bDg70amg32dyeZGZxiDuQmZxKLAlQjr3jGyLx+4Kkk58UO7D2QdgFIQCovuSuZESne6RG6XQ==} engines: {node: '>= 6.0.0'} @@ -4189,6 +4328,9 @@ packages: bl@4.1.0: resolution: {integrity: sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==} + bluebird@3.4.7: + resolution: {integrity: sha512-iD3898SR7sWVRHbiQv+sHUtHnMvC1o3nW5rAcqnq3uOn07DSAppZYUkIGslDz6gXC7HfunPe7YVBgoEJASPcHA==} + bn.js@4.12.5: resolution: {integrity: sha512-3aRg6/JxfffFD+OlOjOFR3Vo79l39ooBTFucxx+MT3dhCtzn3EmiUPQo+6/OZuI2jbXi3YKgmiTFBgChQMwIRQ==} @@ -4281,6 +4423,10 @@ packages: resolution: {integrity: sha512-Gmy6FhYlCY7uOElZUSbxo2UCDH8owEk996gkbrpsgGtrJLM3J7jGxl9Ic7Qwwj4ivOE5AWZWRMecDdF7hqGjFA==} engines: {node: '>=10'} + cfb@1.2.2: + resolution: {integrity: sha512-KfdUZsSOw19/ObEWasvBP/Ac4reZvAGauZhs6S/gqNhXhI7cKwvlH7ulj+dOEYnca4bm4SGo8C1bTAQvnTjgQA==} + engines: {node: '>=0.8'} + chai@5.3.3: resolution: {integrity: sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==} engines: {node: '>=18'} @@ -4377,6 +4523,10 @@ packages: resolution: {integrity: sha512-eYm0QWBtUrBWZWG0d386OGAw16Z995PiOVo2B7bjWSbHedGl5e0ZWaq65kOGgUSNesEIDkB9ISbTg/JK9dhCZA==} engines: {node: '>=6'} + codepage@1.15.0: + resolution: {integrity: sha512-3g6NUTPd/YtuuGrhMnOMRjFc+LJw/bnMp3+0r/Wcz3IXUuCosKRJvMphm5+Q+bvTVGcJJuRvVLuYba+WojaFaA==} + engines: {node: '>=0.8'} + color-convert@2.0.1: resolution: {integrity: sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==} engines: {node: '>=7.0.0'} @@ -4464,6 +4614,11 @@ packages: resolution: {integrity: sha512-tJtZBBHA6vjIAaF6EnIaq6laBBP9aq/Y3ouVJjEfoHbRBcHBAHYcMh/w8LDrk2PvIMMq8gmopa5D4V8RmbrxGw==} engines: {node: '>= 0.10'} + crc-32@1.2.2: + resolution: {integrity: sha512-ROmzCKrTnOwybPcJApAA6WBWij23HVfGVNKqqrZpuyZOHqK2CwHSvpGuyt/UNNvaIjEd8X5IFGp4Mh+Ie1IHJQ==} + engines: {node: '>=0.8'} + hasBin: true + croner@10.0.1: resolution: {integrity: sha512-ixNtAJndqh173VQ4KodSdJEI6nuioBWI0V1ITNKhZZsO0pEMoDxz539T4FTTbSZ/xIOSuDnzxLVRqBVSvPNE2g==} engines: {node: '>=18.0'} @@ -4622,6 +4777,9 @@ packages: dijkstrajs@1.0.3: resolution: {integrity: sha512-qiSlmBq9+BCdCA/L46dw8Uy93mloxsPSbwnm5yrKn2vMPiy8KyAskTF6zuV/j5BMsmOGZDPs7KjU+mjb670kfA==} + dingbat-to-unicode@1.0.1: + resolution: {integrity: sha512-98l0sW87ZT58pU4i61wa2OHwxbiYSbuxsCBozaVnYX2iCnr3bLM3fIes1/ej7h1YdOKuKt/MLs706TVnALA65w==} + dir-glob@3.0.1: resolution: {integrity: sha512-WkrWp9GR4KXfKGYzOLmTuGVi1UWFfws377n9cc55/tb6DuqyF6pcQ5AbiHEshaDpY9v6oaSr2XCDidGmMwdzIA==} engines: {node: '>=8'} @@ -4769,6 +4927,9 @@ packages: sqlite3: optional: true + duck@0.1.12: + resolution: {integrity: sha512-wkctla1O6VfP89gQ+J/yDesM0S7B7XLXjKGzXxMDVFg7uEn706niAtyYovKbyq1oT9YwDcly721/iUWoc8MVRg==} + dunder-proto@1.0.1: resolution: {integrity: sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==} engines: {node: '>= 0.4'} @@ -5098,6 +5259,10 @@ packages: resolution: {integrity: sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==} engines: {node: '>= 0.6'} + frac@1.1.2: + resolution: {integrity: sha512-w/XBfkibaTl3YDqASwfDUqkna4Z2p9cFSr1aHDt0WoMTECnRfBOv2WArlZILlqgWlmdIlALXGpM2AOhEk5W3IA==} + engines: {node: '>=0.8'} + fresh@2.0.0: resolution: {integrity: sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A==} engines: {node: '>= 0.8'} @@ -5670,6 +5835,9 @@ packages: resolution: {integrity: sha512-lyuxPGr/Wfhrlem2CL/UcnUc1zcqKAImBDzukY7Y5F/yQiNdko6+fRLevlw1HgMySw7f611UIY408EtxRSoK3Q==} hasBin: true + lop@0.4.2: + resolution: {integrity: sha512-RefILVDQ4DKoRZsJ4Pj22TxE3omDO47yFpkIBoDKzkqPRISs5U1cnAdg/5583YPkWPaLIYHOKRMQSvjFsO26cw==} + loupe@3.2.1: resolution: {integrity: sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==} @@ -5687,6 +5855,11 @@ packages: magic-string@0.30.21: resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==} + mammoth@1.12.2: + resolution: {integrity: sha512-MH2vkgafD/2MYUaEOtoXLKrHQZ7yLYTHGQgyLNtYZHiUxU1K1QQ+8qMFquDAfhBm06wSGSws3J+Q9QTsKtR44g==} + engines: {node: '>=12.0.0'} + hasBin: true + markdown-it@14.3.0: resolution: {integrity: sha512-RCEsPjR+sr0x+AuYp601tKTkgFG4YEPLCzHST3cQ/fhlJkqAkz1L2/Qbp1j9qw5SBwQHFBoW8+hoN5xssOF0Tw==} hasBin: true @@ -6002,6 +6175,9 @@ packages: engines: {node: '>=22.22.3 <23 || >=24.15.0 <25 || >=25.9.0'} hasBin: true + option@0.2.4: + resolution: {integrity: sha512-pkEqbDyl8ou5cpq+VsnQbe/WlEy5qS7xPzMS1U55OCG9KPvwFD46zDbxQIj3egJSFc3D+XhYOPUzz49zQAVy7A==} + os-paths@4.4.0: resolution: {integrity: sha512-wrAwOeXp1RRMFfQY8Sy7VaGVmPocaLwSFOYCGKSyo8qmJ+/yaafCl5BCA1IQZWqFSRBrKDYFeR9d/VyQzfH/jg==} engines: {node: '>= 6.0'} @@ -6086,6 +6262,10 @@ packages: resolution: {integrity: sha512-ak9Qy5Q7jYb2Wwcey5Fpvg2KoAc/ZIhLSLOSBmRmygPsGwkVVt0fZa0qrtMz+m6tJTAHfZQ8FnmB4MG4LWy7/w==} engines: {node: '>=8'} + path-is-absolute@1.0.1: + resolution: {integrity: sha512-AVbw3UJ2e9bq64vSaS9Am0fje1Pa8pbGqTTsmXfaIiMpnr5DlDhfJOuLj9Sf95ZPVDAUerDfEk88MPmPe7UCQg==} + engines: {node: '>=0.10.0'} + path-key@3.1.1: resolution: {integrity: sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==} engines: {node: '>=8'} @@ -6111,10 +6291,19 @@ packages: resolution: {integrity: sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==} engines: {node: '>= 14.16'} + pdf-parse@2.4.5: + resolution: {integrity: sha512-mHU89HGh7v+4u2ubfnevJ03lmPgQ5WU4CxAVmTSh/sxVTEDYd1er/dKS/A6vg77NX47KTEoihq8jZBLr8Cxuwg==} + engines: {node: '>=20.16.0 <21 || >=22.3.0'} + hasBin: true + pdfjs-dist@4.10.38: resolution: {integrity: sha512-/Y3fcFrXEAsMjJXeL9J8+ZG9U01LbuWaYypvDW2ycW1jL269L3js3DVBjDJ0Up9Np1uqDXsDrRihHANhZOlwdQ==} engines: {node: '>=20'} + pdfjs-dist@5.4.296: + resolution: {integrity: sha512-DlOzet0HO7OEnmUmB6wWGJrrdvbyJKftI1bhMitK7O2N8W2gc757yyYBbINy9IDafXAV9wmKr9t7xsTaNKRG5Q==} + engines: {node: '>=20.16.0 || >=22.3.0'} + pg-cloudflare@1.4.0: resolution: {integrity: sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==} @@ -6638,6 +6827,10 @@ packages: sqlite-vec@0.1.9: resolution: {integrity: sha512-L7XJWRIBNvR9O5+vh1FQ+IGkh/3D2AzVksW5gdtk28m78Hy8skFD0pqReKH1Yp0/BUKRGcffgKvyO/EON5JXpA==} + ssf@0.11.2: + resolution: {integrity: sha512-+idbmIXoYET47hH+d7dfm2epdOMUDjqcB4648sTZ+t2JwoyBFL/insLfB/racrDmsKB3diwsDA696pZMieAC5g==} + engines: {node: '>=0.8'} + sswr@2.2.0: resolution: {integrity: sha512-clTszLPZkmycALTHD1mXGU+mOtA/MIoLgS1KGTTzFNVm9rytQVykgRaP+z1zl572cz0bTqj4rFVoC2N+IGK4Sg==} peerDependencies: @@ -6957,6 +7150,9 @@ packages: resolution: {integrity: sha512-rvKSBiC5zqCCiDZ9kAOszZcDvdAHwwIKJG33Ykj43OKcWsnmcBRL09YTU4nOeHZ8Y2a7l1MgTd08SBe9A8Qj6A==} engines: {node: '>=18'} + underscore@1.13.8: + resolution: {integrity: sha512-DXtD3ZtEQzc7M8m4cXotyHR+FAS18C64asBYY5vqZexfYryNNnDc02W4hKg3rdQuqOYas1jkseX0+nZXjTXnvQ==} + undici-types@5.26.5: resolution: {integrity: sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA==} @@ -7229,6 +7425,14 @@ packages: resolution: {integrity: sha512-LZNJgPzfKR+/J3cHkxcpHKpKKvGfDZVPS4hfJCc4cCG0CgYzvlD6yE/S3CIL/Yt91ak327YCpiF/0MyeZHEHKA==} engines: {node: '>= 12.0.0'} + wmf@1.0.2: + resolution: {integrity: sha512-/p9K7bEh0Dj6WbXg4JG0xvLQmIadrner1bi45VMJTfnbVHsc7yIajZyoSoK60/dtVBs12Fm6WkUI5/3WAVsNMw==} + engines: {node: '>=0.8'} + + word@0.3.0: + resolution: {integrity: sha512-OELeY0Q61OXpdUfTp+oweA/vtLVg5VDOXh+3he3PNzLGG/y0oylSOC1xRVj0+l4vQ3tj/bB1HVHv1ocXkQceFA==} + engines: {node: '>=0.8'} + wordwrap@1.0.0: resolution: {integrity: sha512-gvVzJFlPycKc5dZN4yPkP8w7Dc37BtP1yczEneOb4uq34pXZcvrtRTmWV8W+Ume+XCxKgbjM+nevkyFPMybd4Q==} @@ -7293,10 +7497,19 @@ packages: resolution: {integrity: sha512-sqMMuL1rc0FmMBOzCpd0yuy9trqF2yTTVe+E9ogwCSWQCdDEtQUwrZPT6AxqtsFGRNxycgncbP/xmOOSPw5ZUw==} engines: {node: '>= 6.0'} + xlsx@0.18.5: + resolution: {integrity: sha512-dmg3LCjBPHZnQp5/F/+nnTa+miPJxUXB6vtk42YjBBKayDNagxGEeIdWApkYPOf3Z3pm3k62Knjzp7lMeTEtFQ==} + engines: {node: '>=0.8'} + hasBin: true + xml-name-validator@5.0.0: resolution: {integrity: sha512-EvGK8EJ3DhaHfbRlETOWAS5pO9MZITeauHKJyb8wyajUfQUenkIg2MvLDTZ4T/TgIcm3HU0TFBgWWboAZ30UHg==} engines: {node: '>=18'} + xmlbuilder@10.1.1: + resolution: {integrity: sha512-OyzrcFLL/nb6fMGHbiRDuPup9ljBycsdCypwuyg5AAHvyWzGfChJpCXMG88AGTIMFhGZ9RccFN1e6lhg3hkwKg==} + engines: {node: '>=4.0'} + xmlchars@2.2.0: resolution: {integrity: sha512-JZnDKK8B0RCDw84FNdDAIpZK+JuJw+s7Lz8nksI7SIuU3UXJJslUthsi+uWBUYOwPFwW7W7PRLRfUKpxjtjFCw==} @@ -8670,7 +8883,7 @@ snapshots: - openai - ws - '@langchain/community@1.1.29(382ed5f2a3e6bc1e3575eeff382cf5cb)': + '@langchain/community@1.1.29(473a380aa82688979edbb67c4e0acfa5)': dependencies: '@browserbasehq/stagehand': 1.14.0(@playwright/test@1.62.1)(bufferutil@4.1.0)(deepmerge@4.3.1)(dotenv@17.4.2)(encoding@0.1.13)(openai@4.104.0(encoding@0.1.13)(ws@8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(zod@4.4.3))(utf-8-validate@5.0.10)(zod@4.4.3) '@ibm-cloud/watsonx-ai': 1.7.16 @@ -8691,9 +8904,10 @@ snapshots: '@aws-sdk/credential-provider-node': 3.972.80 '@browserbasehq/sdk': 2.16.0(encoding@0.1.13) '@huggingface/transformers': 3.8.1 + '@lancedb/lancedb': 0.37.1(@types/node@26.2.0)(apache-arrow@18.1.0)(encoding@0.1.13) '@mozilla/readability': 0.6.0 '@smithy/signature-v4': 5.7.2 - '@zilliz/milvus2-sdk-node': 2.6.17(bufferutil@4.1.0)(utf-8-validate@5.0.10) + '@zilliz/milvus2-sdk-node': 2.5.13 better-sqlite3: 11.10.0 chromadb: 3.5.0 fast-xml-parser: 4.5.7 @@ -8702,6 +8916,8 @@ snapshots: ignore: 7.0.5 jsdom: 24.1.3(bufferutil@4.1.0)(utf-8-validate@5.0.10) jsonwebtoken: 9.0.3 + mammoth: 1.12.2 + pdf-parse: 2.4.5 pg: 8.23.0 playwright: 1.62.1 ws: 8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10) @@ -8712,7 +8928,7 @@ snapshots: - '@smithy/hash-node' - peggy - '@langchain/community@1.1.29(905166cb4cd0093f294357964e3103f4)': + '@langchain/community@1.1.29(8e07a13ecb0caa2defe560ab6fa2e6f8)': dependencies: '@browserbasehq/stagehand': 1.14.0(@playwright/test@1.62.1)(bufferutil@4.1.0)(deepmerge@4.3.1)(dotenv@17.4.2)(encoding@0.1.13)(openai@4.104.0(encoding@0.1.13)(ws@8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(zod@4.4.3))(utf-8-validate@5.0.10)(zod@4.4.3) '@ibm-cloud/watsonx-ai': 1.7.16 @@ -8733,10 +8949,9 @@ snapshots: '@aws-sdk/credential-provider-node': 3.972.80 '@browserbasehq/sdk': 2.16.0(encoding@0.1.13) '@huggingface/transformers': 3.8.1 - '@lancedb/lancedb': 0.37.1(@types/node@26.2.0)(apache-arrow@18.1.0)(encoding@0.1.13) '@mozilla/readability': 0.6.0 '@smithy/signature-v4': 5.7.2 - '@zilliz/milvus2-sdk-node': 2.5.13 + '@zilliz/milvus2-sdk-node': 2.6.17(bufferutil@4.1.0)(utf-8-validate@5.0.10) better-sqlite3: 11.10.0 chromadb: 3.5.0 fast-xml-parser: 4.5.7 @@ -8745,6 +8960,8 @@ snapshots: ignore: 7.0.5 jsdom: 24.1.3(bufferutil@4.1.0)(utf-8-validate@5.0.10) jsonwebtoken: 9.0.3 + mammoth: 1.12.2 + pdf-parse: 2.4.5 pg: 8.23.0 playwright: 1.62.1 ws: 8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10) @@ -8946,36 +9163,66 @@ snapshots: '@napi-rs/canvas-android-arm64@0.1.100': optional: true + '@napi-rs/canvas-android-arm64@0.1.80': + optional: true + '@napi-rs/canvas-darwin-arm64@0.1.100': optional: true + '@napi-rs/canvas-darwin-arm64@0.1.80': + optional: true + '@napi-rs/canvas-darwin-x64@0.1.100': optional: true + '@napi-rs/canvas-darwin-x64@0.1.80': + optional: true + '@napi-rs/canvas-linux-arm-gnueabihf@0.1.100': optional: true + '@napi-rs/canvas-linux-arm-gnueabihf@0.1.80': + optional: true + '@napi-rs/canvas-linux-arm64-gnu@0.1.100': optional: true + '@napi-rs/canvas-linux-arm64-gnu@0.1.80': + optional: true + '@napi-rs/canvas-linux-arm64-musl@0.1.100': optional: true + '@napi-rs/canvas-linux-arm64-musl@0.1.80': + optional: true + '@napi-rs/canvas-linux-riscv64-gnu@0.1.100': optional: true + '@napi-rs/canvas-linux-riscv64-gnu@0.1.80': + optional: true + '@napi-rs/canvas-linux-x64-gnu@0.1.100': optional: true + '@napi-rs/canvas-linux-x64-gnu@0.1.80': + optional: true + '@napi-rs/canvas-linux-x64-musl@0.1.100': optional: true + '@napi-rs/canvas-linux-x64-musl@0.1.80': + optional: true + '@napi-rs/canvas-win32-arm64-msvc@0.1.100': optional: true '@napi-rs/canvas-win32-x64-msvc@0.1.100': optional: true + '@napi-rs/canvas-win32-x64-msvc@0.1.80': + optional: true + '@napi-rs/canvas@0.1.100': optionalDependencies: '@napi-rs/canvas-android-arm64': 0.1.100 @@ -8991,6 +9238,19 @@ snapshots: '@napi-rs/canvas-win32-x64-msvc': 0.1.100 optional: true + '@napi-rs/canvas@0.1.80': + optionalDependencies: + '@napi-rs/canvas-android-arm64': 0.1.80 + '@napi-rs/canvas-darwin-arm64': 0.1.80 + '@napi-rs/canvas-darwin-x64': 0.1.80 + '@napi-rs/canvas-linux-arm-gnueabihf': 0.1.80 + '@napi-rs/canvas-linux-arm64-gnu': 0.1.80 + '@napi-rs/canvas-linux-arm64-musl': 0.1.80 + '@napi-rs/canvas-linux-riscv64-gnu': 0.1.80 + '@napi-rs/canvas-linux-x64-gnu': 0.1.80 + '@napi-rs/canvas-linux-x64-musl': 0.1.80 + '@napi-rs/canvas-win32-x64-msvc': 0.1.80 + '@napi-rs/lzma-linux-x64-gnu@1.5.1': optional: true @@ -9425,6 +9685,14 @@ snapshots: optionalDependencies: vite: 6.4.3(@types/node@26.2.0)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0) + '@vitest/mocker@4.1.10(vite@6.4.3(@types/node@22.20.1)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0))': + dependencies: + '@vitest/spy': 4.1.10 + estree-walker: 3.0.3 + magic-string: 0.30.21 + optionalDependencies: + vite: 6.4.3(@types/node@22.20.1)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0) + '@vitest/mocker@4.1.10(vite@6.4.3(@types/node@26.2.0)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0))': dependencies: '@vitest/spy': 4.1.10 @@ -9563,6 +9831,8 @@ snapshots: optionalDependencies: node-fetch: 3.3.2 + '@xmldom/xmldom@0.8.15': {} + '@xterm/xterm@5.5.0': optional: true @@ -9617,6 +9887,8 @@ snapshots: acorn@8.18.0: {} + adler-32@1.3.1: {} + agent-base@6.0.2: dependencies: debug: 4.4.3 @@ -9807,6 +10079,8 @@ snapshots: inherits: 2.0.4 readable-stream: 3.6.2 + bluebird@3.4.7: {} + bn.js@4.12.5: {} body-parser@2.3.0: @@ -9902,6 +10176,11 @@ snapshots: camelcase@6.3.0: {} + cfb@1.2.2: + dependencies: + adler-32: 1.3.1 + crc-32: 1.2.2 + chai@5.3.3: dependencies: assertion-error: 2.0.1 @@ -9983,6 +10262,8 @@ snapshots: clsx@2.1.1: {} + codepage@1.15.0: {} + color-convert@2.0.1: dependencies: color-name: 1.1.4 @@ -10061,6 +10342,8 @@ snapshots: object-assign: 4.1.1 vary: 1.1.2 + crc-32@1.2.2: {} + croner@10.0.1: {} croner@8.1.2: {} @@ -10185,6 +10468,8 @@ snapshots: dijkstrajs@1.0.3: {} + dingbat-to-unicode@1.0.1: {} + dir-glob@3.0.1: dependencies: path-type: 4.0.0 @@ -10256,6 +10541,10 @@ snapshots: kysely: 0.29.2 pg: 8.23.0 + duck@0.1.12: + dependencies: + underscore: 1.13.8 + dunder-proto@1.0.1: dependencies: call-bind-apply-helpers: 1.0.2 @@ -10679,6 +10968,8 @@ snapshots: forwarded@0.2.0: {} + frac@1.1.2: {} + fresh@2.0.0: {} fs-constants@1.0.0: {} @@ -11323,6 +11614,12 @@ snapshots: js-tokens: 4.0.0 optional: true + lop@0.4.2: + dependencies: + duck: 0.1.12 + option: 0.2.4 + underscore: 1.13.8 + loupe@3.2.1: {} lru-cache@10.4.3: @@ -11336,6 +11633,19 @@ snapshots: dependencies: '@jridgewell/sourcemap-codec': 1.5.5 + mammoth@1.12.2: + dependencies: + '@xmldom/xmldom': 0.8.15 + argparse: 1.0.10 + base64-js: 1.5.1 + bluebird: 3.4.7 + dingbat-to-unicode: 1.0.1 + jszip: 3.10.1 + lop: 0.4.2 + path-is-absolute: 1.0.1 + underscore: 1.13.8 + xmlbuilder: 10.1.1 + markdown-it@14.3.0: dependencies: argparse: 2.0.1 @@ -11697,6 +12007,8 @@ snapshots: - tree-sitter - utf-8-validate + option@0.2.4: {} + os-paths@4.4.0: {} outdent@0.5.0: {} @@ -11767,6 +12079,8 @@ snapshots: path-exists@4.0.0: {} + path-is-absolute@1.0.1: {} + path-key@3.1.1: {} path-scurry@2.0.2: @@ -11784,10 +12098,19 @@ snapshots: pathval@2.0.1: {} + pdf-parse@2.4.5: + dependencies: + '@napi-rs/canvas': 0.1.80 + pdfjs-dist: 5.4.296 + pdfjs-dist@4.10.38: optionalDependencies: '@napi-rs/canvas': 0.1.100 + pdfjs-dist@5.4.296: + optionalDependencies: + '@napi-rs/canvas': 0.1.100 + pg-cloudflare@1.4.0: optional: true @@ -12391,6 +12714,10 @@ snapshots: sqlite-vec-linux-x64: 0.1.9 sqlite-vec-windows-x64: 0.1.9 + ssf@0.11.2: + dependencies: + frac: 1.1.2 + sswr@2.2.0(svelte@5.56.8): dependencies: svelte: 5.56.8 @@ -12755,6 +13082,8 @@ snapshots: uint8array-extras@1.5.0: {} + underscore@1.13.8: {} + undici-types@5.26.5: {} undici-types@6.21.0: {} @@ -12964,6 +13293,35 @@ snapshots: - tsx - yaml + vitest@4.1.10(@opentelemetry/api@1.9.0)(@types/node@22.20.1)(jsdom@24.1.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(vite@6.4.3(@types/node@22.20.1)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0)): + dependencies: + '@vitest/expect': 4.1.10 + '@vitest/mocker': 4.1.10(vite@6.4.3(@types/node@22.20.1)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0)) + '@vitest/pretty-format': 4.1.10 + '@vitest/runner': 4.1.10 + '@vitest/snapshot': 4.1.10 + '@vitest/spy': 4.1.10 + '@vitest/utils': 4.1.10 + es-module-lexer: 2.3.1 + expect-type: 1.4.0 + magic-string: 0.30.21 + obug: 2.1.4 + pathe: 2.0.3 + picomatch: 4.0.5 + std-env: 4.2.0 + tinybench: 2.9.0 + tinyexec: 1.3.0 + tinyglobby: 0.2.17 + tinyrainbow: 3.1.1 + vite: 6.4.3(@types/node@22.20.1)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0) + why-is-node-running: 2.3.0 + optionalDependencies: + '@opentelemetry/api': 1.9.0 + '@types/node': 22.20.1 + jsdom: 24.1.3(bufferutil@4.1.0)(utf-8-validate@5.0.10) + transitivePeerDependencies: + - msw + vitest@4.1.10(@opentelemetry/api@1.9.0)(@types/node@26.2.0)(jsdom@24.1.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(vite@6.4.3(@types/node@26.2.0)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0)): dependencies: '@vitest/expect': 4.1.10 @@ -13094,6 +13452,10 @@ snapshots: triple-beam: 1.4.1 winston-transport: 4.9.0 + wmf@1.0.2: {} + + word@0.3.0: {} + wordwrap@1.0.0: {} wordwrapjs@5.1.1: {} @@ -13144,9 +13506,21 @@ snapshots: dependencies: os-paths: 4.4.0 + xlsx@0.18.5: + dependencies: + adler-32: 1.3.1 + cfb: 1.2.2 + codepage: 1.15.0 + crc-32: 1.2.2 + ssf: 0.11.2 + wmf: 1.0.2 + word: 0.3.0 + xml-name-validator@5.0.0: optional: true + xmlbuilder@10.1.1: {} + xmlchars@2.2.0: optional: true