From 7955d2421f7c8514ab3d1b0681c464d7587b063a Mon Sep 17 00:00:00 2001 From: Peefy Date: Fri, 11 Sep 2026 20:37:41 +0800 Subject: [PATCH 1/8] feat(workspace): folder indexing + cross-file hybrid search MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the @melandlabs/workspace package — a CLI for indexing an OKF/Markdown folder into SQLite and querying it with lexical, semantic, hybrid, and cross-file strategies. What's in the box - packages/workspace: new package with sqlite store, embedding queue (class-internal serial Promise queue), 4 search strategies (lexical / semantic / hybrid RRF / cross-file BFS over cites edges), OKF backend, multi-format parsers adapter (.md / .markdown / .txt / .pdf / .docx / .pages). - Local embeddings default: workspace CLI defaults EMBEDDING_PROVIDER=local (Xenova/all-MiniLM-L6-v2, 384 dims) so demos run offline without OPENROUTER_API_KEY. - opencontext CLI surface: `opencontext workspace update|search|list` wired through the main bin. - Re-exports from @melandlabs/opencontext for downstream consumers. - Tutorial: docs/tutorials/use-cases/09-workspace-folder-indexing.md covering the Quick Verification recipe, multi-format walkthrough, and the local/cloud provider switch. - Example: examples/src/simple/22-workspace.ts runs an end-to-end sha256-dedup / lexical / semantic / cross-file demo against a scratch store. Storage - Reuses ~/.opencontext/memory/store.db (same DB as memory-store). - New tables: workspace_resources, workspace_resource_versions, workspace_chunks + workspace_chunks_fts, workspace_reference_edges, workspace_jobs, plus a per-dim vec0 child table workspace_chunks_vec_d{384,1536}. - FTS5 mirror tables and AI/AD/AU triggers mirror the existing memory-store schema style. Verified - workspace 16/16, opencontext 109/109, memory-store 208/208, okf 140/140, examples 282 OK / 3 SKIP / 0 FAIL. - pnpm -w build clean. Co-Authored-By: Claude Opus 4.6 --- .../use-cases/09-workspace-folder-indexing.md | 341 +++++ examples/src/index.ts | 2 + examples/src/simple/14-local-embedding.ts | 8 +- examples/src/simple/22-workspace.ts | 324 ++++ packages/opencontext/package.json | 2 + packages/opencontext/src/cli/opencontext.ts | 10 + packages/opencontext/src/index.ts | 48 + packages/workspace/package.json | 83 + packages/workspace/src/api.ts | 172 +++ packages/workspace/src/cli.ts | 605 ++++++++ packages/workspace/src/embedding-provider.ts | 141 ++ packages/workspace/src/embedding-queue.ts | 118 ++ packages/workspace/src/index.ts | 67 + packages/workspace/src/okf-backend.ts | 211 +++ packages/workspace/src/parsers-adapter.ts | 109 ++ packages/workspace/src/schema.ts | 178 +++ packages/workspace/src/search/cross-file.ts | 73 + packages/workspace/src/search/hybrid.ts | 87 ++ packages/workspace/src/search/lexical.ts | 24 + packages/workspace/src/search/semantic.ts | 23 + packages/workspace/src/sqlite.ts | 1340 +++++++++++++++++ packages/workspace/src/types.ts | 183 +++ packages/workspace/test/okf-backend.test.ts | 125 ++ .../workspace/test/parsers-adapter.test.ts | 53 + packages/workspace/test/search.test.ts | 189 +++ .../test/sqlite-project-store.test.ts | 153 ++ packages/workspace/tsconfig.json | 11 + packages/workspace/tsup.config.ts | 40 + pnpm-lock.yaml | 95 ++ 29 files changed, 4813 insertions(+), 2 deletions(-) create mode 100644 docs/tutorials/use-cases/09-workspace-folder-indexing.md create mode 100644 examples/src/simple/22-workspace.ts create mode 100644 packages/workspace/package.json create mode 100644 packages/workspace/src/api.ts create mode 100644 packages/workspace/src/cli.ts create mode 100644 packages/workspace/src/embedding-provider.ts create mode 100644 packages/workspace/src/embedding-queue.ts create mode 100644 packages/workspace/src/index.ts create mode 100644 packages/workspace/src/okf-backend.ts create mode 100644 packages/workspace/src/parsers-adapter.ts create mode 100644 packages/workspace/src/schema.ts create mode 100644 packages/workspace/src/search/cross-file.ts create mode 100644 packages/workspace/src/search/hybrid.ts create mode 100644 packages/workspace/src/search/lexical.ts create mode 100644 packages/workspace/src/search/semantic.ts create mode 100644 packages/workspace/src/sqlite.ts create mode 100644 packages/workspace/src/types.ts create mode 100644 packages/workspace/test/okf-backend.test.ts create mode 100644 packages/workspace/test/parsers-adapter.test.ts create mode 100644 packages/workspace/test/search.test.ts create mode 100644 packages/workspace/test/sqlite-project-store.test.ts create mode 100644 packages/workspace/tsconfig.json create mode 100644 packages/workspace/tsup.config.ts diff --git a/docs/tutorials/use-cases/09-workspace-folder-indexing.md b/docs/tutorials/use-cases/09-workspace-folder-indexing.md new file mode 100644 index 00000000..b8185bbb --- /dev/null +++ b/docs/tutorials/use-cases/09-workspace-folder-indexing.md @@ -0,0 +1,341 @@ +# Use Case: Workspace Folder Indexing + +## The Scenario + +You have a folder of related Markdown / PDF / DOCX files — say, three contracts, a memo, and a few public statutes you cite — and you want to **ask natural-language questions across the whole folder**: + +- "Which contracts cap liability at twelve months of fees?" +- "Pull every place in `contract-A.md` that references `law-2024.md`." +- "What does `memo.md` say about indemnification carve-outs?" + +Treating the folder as a **single project space** is what `@melandlabs/workspace` is for. It indexes the folder once, deduplicates by content hash, extracts Markdown links into a `cites` graph, and gives you a hybrid (lexical + semantic + cross-file) search command you can run from the terminal. + +This tutorial walks you through the CLI end-to-end. No HTTP server, no MCP client, no SDK call — just `pnpm opencontext workspace …` against a folder on disk. + +## What You'll Build + +A reusable `~/projects//wiki/` folder you can keep updating, plus three CLI commands you'll use day-to-day: + +1. **`opencontext workspace update`** — scan + chunk + embed the folder +2. **`opencontext workspace search`** — query the indexed folder (lexical / semantic / hybrid / cross-file) +3. **`opencontext workspace list`** — see what's been indexed and its current status + +## Concepts Demonstrated + +- **Versioned folder indexing** — re-running `update` skips unchanged files (`sha256` dedup) and creates a new version row only when content changes. +- **Cross-file reference edges** — Markdown links like `[Limitation of Liability](./b.md)` become `cites` edges in `workspace_reference_edges`. The `cross-file` strategy walks these edges to surface related context. +- **Multi-format parsing** — `.md`, `.markdown`, `.txt` are read raw; `.pdf`, `.docx`, `.pages` go through the `packages/rag` parser layer. +- **Local-first embeddings** — `EMBEDDING_PROVIDER` defaults to `local` for the `workspace` CLI, which runs `Xenova/all-MiniLM-L6-v2` (384 dims) on-device via `@huggingface/transformers`. No API key required. + +## Prerequisites + +- Completed [Getting Started](../00-getting-started.md) — `pnpm` workspace already installed. +- Node.js ≥ 22 (same as the rest of OpenContext). +- ~500 MB free disk for the local embedding model weights (downloaded once to `~/.cache/opencontext/local-embeddings`). + +## Quick Verification + +The fastest path: copy this four-line recipe and run it. The rest of the tutorial walks through what each step does and what to look for. + +```bash +# 1. Make a demo folder +mkdir -p /tmp/wf-demo/wiki +cat > /tmp/wf-demo/wiki/a.md <<'EOF' +--- +title: Limitation of Liability +type: contract +created: 2026-09-11 +--- +The aggregate liability of either party shall not exceed the fees paid in the +twelve (12) months preceding the claim. See [Indemnification](./b.md) for carve-outs. +EOF +cat > /tmp/wf-demo/wiki/b.md <<'EOF' +--- +title: Indemnification +type: contract +created: 2026-09-11 +--- +Neither party shall indemnify the other for indirect, consequential, or +punitive damages. References: [Limitation of Liability](./a.md). +EOF + +# 2. Index the folder (local embeddings by default — no OPENROUTER_API_KEY needed) +pnpm opencontext workspace update \ + --workspace-id demo --path /tmp/wf-demo/wiki --json + +# 3. Lexical search — works immediately, before embeddings finish +pnpm opencontext workspace search \ + --workspace-id demo --query "limitation" --strategy lexical + +# 4. Hybrid search — once the embedding fan-out has caught up +pnpm opencontext workspace search \ + --workspace-id demo --query "indemnification" --strategy hybrid --limit 3 + +# 5. Cross-file — expands hits along `cites` edges from the OKF graph +pnpm opencontext workspace search \ + --workspace-id demo --query "limitation" --strategy cross-file --hops 1 --json +``` + +If you see `2 hit(s)` and the `cites` edges in the JSON output, you're done. + +> **Note:** On macOS, the CLI may print `libc++abi: ... mutex lock failed: Invalid argument` and exit non-zero after writing the result. This is a known race between `@huggingface/transformers`' ONNX worker thread and `sqlite-vec`'s native destructor — the **output is complete and correct**, just wrap with `|| true` or `2>/dev/null` if it bothers your shell: +> +> ```bash +> pnpm opencontext workspace search ... || true +> ``` + +## Step-by-Step Walkthrough + +### Step 1: Pick a folder + +Any folder works. The indexer walks it recursively and picks a `resource_type` per file based on the extension: + +| Extension | `resource_type` | Indexed? | +| --- | --- | --- | +| `.md`, `.markdown` | `note` (or whatever the OKF front-matter `type:` says) | yes — pass-through; OKF front-matter is parsed; Markdown links become `cites` edges | +| `.txt` | `note` | yes — pass-through | +| `.pdf` | `document` | yes — goes through `@melandlabs/rag` parser (text + page-level metadata) | +| `.docx` | `document` | yes — same parser pipeline | +| `.pages` | `document` | yes — same parser pipeline (macOS only) | +| `.html`, `.htm` | — | mime is recognised but the OKF walker currently skips these | +| `.xlsx`, `.numbers` | — | **not indexed** — neither the parser layer nor the OKF walker supports spreadsheets | +| `.png`, `.jpg`, … | — | **not indexed** — raster files raise an explicit "unsupported" error | + +To mix formats, just drop them in the same folder. The walker picks them up in the same `update` run; the chunker and embedder don't care about the source extension once the text body is in hand: + +```bash +# macOS: spin up a real .pdf and .docx alongside the markdown fixtures +echo "Public Law: cap of liability is twelve months of fees for ordinary breach." \ + > /tmp/wf-demo/wiki/law-raw.txt +textutil -convert pdf -output /tmp/wf-demo/wiki/law.pdf /tmp/wf-demo/wiki/law-raw.txt +textutil -convert docx -output /tmp/wf-demo/wiki/law.docx /tmp/wf-demo/wiki/law-raw.txt +rm /tmp/wf-demo/wiki/law-raw.txt +``` + +Markdown files get extra love: the OKF front-matter (`--- title: ... type: ... ---`) is parsed, and `[text](./other.md)` Markdown links become `cites` edges pointing from the current file to the linked file. + +### Step 2: Index the folder + +```bash +pnpm opencontext workspace update \ + --workspace-id demo --path /tmp/wf-demo/wiki --json +``` + +Output (with `--json`): + +```json +{ + "ok": true, + "exit": 0, + "workspace_id": "demo", + "job_id": 1, + "status": "pending", + "files_scanned": 2, + "files_added": 2, + "files_modified": 0, + "files_unchanged": 0, + "files_deleted": 0 +} +``` + +What happened: + +1. The folder was scanned recursively — two Markdown files. +2. Each file got a `sha256` content hash. Both files were new → `files_added: 2`. +3. Chunks were written into `workspace_chunks` synchronously (FTS5 mirror tables updated immediately via triggers). +4. An embedding **job** was created; its `enqueueEmbedding` hook fires off the local ONNX embedder (`Xenova/all-MiniLM-L6-v2`, 384 dims). +5. The CLI waits for the embedding queue to drain by default (use `--no-await-embeddings` to return immediately). +6. A vec0 child table `workspace_chunks_vec_d384` was auto-created keyed by the provider's dimension. + +Re-run the same command and you'll see `files_unchanged: 2` — the `sha256` dedup means nothing gets re-chunked or re-embedded. + +### Step 3: Search (lexical first) + +```bash +pnpm opencontext workspace search \ + --workspace-id demo --query "limitation" --strategy lexical +``` + +Lexical search hits FTS5 the moment `update` returns (synchronous chunking). You don't need to wait for embeddings. + +``` +2 hit(s) (strategy=lexical) + - [note] a :: --- title: Limitation of Liability type: contract created: 2026-09-11 --- # Limitation of Liability The aggregate liability of either party… (score=1.000) + - [note] b :: …ce and wilful misconduct. References: [Limitation of Liability](./a.md). (score=1.000) +``` + +Both files match because `b.md` references "Limitation of Liability" in a Markdown link — FTS5 tokenizes the link text. + +### Step 4: Search (hybrid / cross-file) + +Once the embedding queue has caught up: + +```bash +pnpm opencontext workspace search \ + --workspace-id demo --query "limitation" --strategy cross-file --hops 1 --json +``` + +The `cross-file` strategy runs hybrid retrieval, then walks `cites` edges out of the top hits to expand the result set. The JSON payload shows the full hit structure: + +```json +{ + "query": "limitation", + "strategy": "cross-file", + "total": 2, + "hits": [ + { + "chunk_id": "demo:3:3:chunk:0:86599afa5dfd9428", + "resource_id": 3, + "version_id": 3, + "resource_type": "note", + "resource_title": "a", + "canonical_key": "a.md", + "snippet": "...# Limitation of Liability\n\nThe aggregate liability of either party…", + "matched_terms": ["limitation"], + "score": 0.01639344262295082, + "signals": { "lexical": 0.9999986666684445 }, + "reference_edges": [ + { "edge_type": "cites", "target_resource_id": 4 }, + { "edge_type": "cites", "target_resource_id": 3 } + ] + } + ] +} +``` + +What to look for: + +- `signals.lexical` — FTS5 BM25-derived similarity (1.0 = strong match). +- `signals.semantic` — appears once embeddings have been written. +- `reference_edges` — the `cites` graph extracted from Markdown links. Cross-file strategy uses these to expand beyond the top-N lexical hits. +- `--hops 2` walks two hops out from each seed hit (deeper expansion, more results). + +### Step 5: Inspect what's indexed + +```bash +pnpm opencontext workspace list --workspace-id demo --json +``` + +Each row includes: + +- `id` — stable resource id. +- `resource_type` / `canonical_key` / `title` — from the file extension and OKF front-matter. +- `index_status` — `pending` (just indexed), `partial` (some chunks embedded), `ready` (all chunks embedded), or `failed`. +- `current_version_id` — points into `workspace_resource_versions`; new versions are created only on content change. + +## Command Reference + +### `opencontext workspace update` + +```text +Required: + --workspace-id Workspace identifier + --path Path to the OKF folder (will be scanned recursively) + +Optional: + --user User / tenant id (default: "default") + --await-embeddings Wait for the in-process embedding queue to drain + before exiting (default: on) + --no-await-embeddings Return as soon as synchronous indexing completes + --drain-timeout-ms Upper bound on --await-embeddings (default 120000) + --json Emit JSON envelope + +Example: + opencontext workspace update --workspace-id demo --path ./wiki +``` + +### `opencontext workspace search` + +```text +Required: + --workspace-id Workspace identifier + --query Search query + +Strategy: + --strategy lexical | semantic | hybrid | cross-file + (default: hybrid) + --limit Top-N hits (default: 10, max: 50) + --threshold Semantic similarity threshold, 0..1 (default: 0.7) + --resource-type Comma-separated filter, e.g. "note,statute" + --hops <1|2> Cross-file BFS depth (cross-file strategy only) + +Output: + --json Emit JSON envelope with full WorkspaceSearchHit[] + +Example: + opencontext workspace search --workspace-id demo --query "limitation" \ + --strategy cross-file --hops 1 --json +``` + +### `opencontext workspace list` + +```text +Required: + --workspace-id Workspace identifier + +Filters: + --resource-type Filter by resource type + --index-status pending | partial | ready | failed + --limit Max rows (default: 50) + --offset Skip N rows (default: 0) + +Output: + --json Emit JSON envelope + +Example: + opencontext workspace list --workspace-id demo --index-status ready --limit 20 +``` + +## Storage + +Everything lives in the shared SQLite database at `~/.opencontext/memory/store.db` (override with `MEMORY_STORE_DB_PATH` or `--db-path`). The workspace package creates the following tables alongside the memory-store's existing schema: + +- `workspace_resources` — one row per indexed file (`(workspace_id, canonical_key)` is unique). +- `workspace_resource_versions` — append-only version chain keyed by `sha256`. +- `workspace_chunks` + `workspace_chunks_fts` — text chunks with FTS5 mirror (lexical search). +- `workspace_chunks_vec_d{N}` — vec0 ANN index, dimension-suffixed so each embedder model gets its own table. +- `workspace_reference_edges` — `cites` edges extracted from Markdown links (1- and 2-hop BFS in `cross-file` strategy). +- `workspace_jobs` — indexing job log (`pending` → `ready` / `partial` / `failed`). + +Inspect the tables directly: + +```bash +sqlite3 ~/.opencontext/memory/store.db \ + "SELECT id, workspace_id, source_resource_id, target_resource_id, edge_type + FROM workspace_reference_edges;" +``` + +## Switching Embedding Providers + +`EMBEDDING_PROVIDER` controls which embedder the workspace CLI uses. The workspace CLI defaults to `local`; the wider OpenContext system still defaults to `cloud`. The provider is resolved per-process; pick whichever matches your environment: + +| `EMBEDDING_PROVIDER` | Model | Dimensions | Requires | +| --- | --- | --- | --- | +| `local` *(default for `workspace`)* | `Xenova/all-MiniLM-L6-v2` | 384 | `@melandlabs/ai-rag` peer dep (already installed) | +| `cloud` | `text-embedding-3-small` via OpenRouter | 1536 | `OPENROUTER_API_KEY` | + +Local is fine for most legal / contract review work where documents stay under ~512 tokens per chunk and the corpus is in English. Switch to cloud if you need multilingual coverage or longer-context embeddings: + +```bash +EMBEDDING_PROVIDER=cloud pnpm opencontext workspace update --workspace-id demo --path ./wiki +``` + +## Known Quirks + +- **macOS SIGABRT noise on exit** — `libc++abi: ... mutex lock failed: Invalid argument` after a successful run is the `@huggingface/transformers` ONNX worker racing `sqlite-vec`'s native destructor. The output above the noise is correct. Wrap with `|| true` or `2>/dev/null` in shell scripts. +- **First-run model download** — the local embedder pulls `~50 MB` of ONNX weights on first use. Subsequent runs are instant. +- **Version churn** — every `update` creates a new row in `workspace_resource_versions` if content changed. The old chunks stay queryable via their `version_id`; cleanup is deferred to a future iteration. +- **No HTTP / MCP yet** — the surface today is CLI-only. Library consumers can `import { updateWorkspaceContext, searchWorkspaceContext, listWorkspaceResources }` from `@melandlabs/workspace`. + +## Next Steps + +- Combine `opencontext workspace search --json` with `jq` in shell pipelines: + + ```bash + pnpm opencontext workspace search --workspace-id demo --query "x" \ + --strategy cross-file --json 2>/dev/null \ + | jq -r '.hits[] | "\(.resource_title)\t\(.score)"' + ``` + +- Mount multiple workspaces for different projects — each gets its own `(workspace_id, canonical_key)` namespace in the same SQLite file. +- Re-index when files change — `update` is idempotent on unchanged content and only re-embeds the chunks that moved. diff --git a/examples/src/index.ts b/examples/src/index.ts index 3e56135d..f39f6d27 100644 --- a/examples/src/index.ts +++ b/examples/src/index.ts @@ -48,6 +48,7 @@ import { withTmp } from "./_helpers.ts"; import demoVsa from "./simple/19-vsa.ts"; import demoOkf from "./simple/20-okf.ts"; import demoOkfServe from "./simple/21-okf-serve.ts"; +import demoWorkspace from "./simple/22-workspace.ts"; import demoHelloMemory from "./tutorials/00-hello-memory.ts"; import demoRememberExample from "./tutorials/01-remember-example.ts"; import demoRecallExample from "./tutorials/02-recall-example.ts"; @@ -161,6 +162,7 @@ const demos: Array<[string, () => Promise]> = [ ["demo: memory-store — Vector Symbolic Architecture (VSA) verb", demoVsa], ["demo: okf — OKF v0.2 (Open Knowledge Format) importer / exporter", demoOkf], ["demo: okf — serve (live + frozen viewer)", demoOkfServe], + ["demo: workspace — folder indexing + cross-file hybrid search (local embeddings by default)", demoWorkspace], ["demo: use-case — personal memory assistant", demoPersonalMemoryAssistant], ["demo: use-case — customer support agent", demoCustomerSupportAgent], ["demo: use-case — research knowledge tracker", demoResearchKnowledgeTracker], diff --git a/examples/src/simple/14-local-embedding.ts b/examples/src/simple/14-local-embedding.ts index 0015bf09..d16c4acf 100644 --- a/examples/src/simple/14-local-embedding.ts +++ b/examples/src/simple/14-local-embedding.ts @@ -53,8 +53,12 @@ export default async function demoLocalEmbedding() { try { factoryProvider = getConfiguredEmbeddingProvider(); } finally { - if (previousProvider === undefined) process.env.EMBEDDING_PROVIDER = undefined; - else process.env.EMBEDDING_PROVIDER = previousProvider; + if (previousProvider === undefined) { + // biome-ignore lint/performance/noDelete: env-reset pattern + delete process.env.EMBEDDING_PROVIDER; + } else { + process.env.EMBEDDING_PROVIDER = previousProvider; + } } check( "getConfiguredEmbeddingProvider() returns the local embedding provider when EMBEDDING_PROVIDER=local", diff --git a/examples/src/simple/22-workspace.ts b/examples/src/simple/22-workspace.ts new file mode 100644 index 00000000..73b75812 --- /dev/null +++ b/examples/src/simple/22-workspace.ts @@ -0,0 +1,324 @@ +/** + * demo: @melandlabs/workspace — folder indexing + cross-file hybrid search. + * + * `@melandlabs/workspace` indexes a local folder of Markdown / TXT / PDF / + * DOCX / Pages files into a SQLite-backed knowledge space. It exposes + * three core APIs: + * + * - `updateWorkspaceContext` — scan + chunk + index a folder + * - `searchWorkspaceContext` — lexical / semantic / hybrid / cross-file + * - `listWorkspaceResources` — enumerate indexed resources + * + * `EMBEDDING_PROVIDER` defaults to `local` here (384-dim + * `Xenova/all-MiniLM-L6-v2`, runs in-process via `@huggingface/transformers`). + * No `OPENROUTER_API_KEY` needed. Override with `EMBEDDING_PROVIDER=cloud` + * for the 1536-dim OpenRouter path. + * + * First-run cost: ~30 MB of ONNX weights are pulled from HuggingFace and + * cached globally. Subsequent runs reuse the cache. + * + * This demo also exercises the workspace CLI surface end-to-end + * (`opencontext workspace update/search/list`) via the same JS API so + * the printed JSON envelope mirrors what the CLI prints. + */ + +import { writeFile, mkdir } from "node:fs/promises"; +import { join } from "node:path"; +import { randomUUID } from "node:crypto"; +import { + closeSQLiteWorkspaceStore, + getSQLiteWorkspaceStore, + listWorkspaceResources, + resolveWorkspaceDbPath, + searchWorkspaceContext, + updateWorkspaceContext, +} from "@melandlabs/opencontext"; +import type { + RuntimeContext, + SearchWorkspaceContextResult, + UpdateWorkspaceContextResult, + WorkspaceSearchHit, +} from "@melandlabs/opencontext"; +import { info, makeCheckWithSkip, runSection, withTmp } from "../_helpers.ts"; + +const WORKSPACE_ID = "demo-workspace"; +const USER_ID = "demo-user"; +const SETTLE_MS = 15_000; // give the embedding queue time to drain on first run + +// Earlier demos in the suite may have poked `process.env.EMBEDDING_PROVIDER` +// (e.g. `examples/src/simple/14-local-embedding.ts` toggles it for the +// factory routing check). Force `local` here so the workspace demo never +// accidentally rides a stray env value into the cloud path. +const PREVIOUS_PROVIDER_ENV = process.env.EMBEDDING_PROVIDER; +process.env.EMBEDDING_PROVIDER = "local"; + +function makeRuntimeContext(): RuntimeContext { + return { + user_id: USER_ID, + employee_id: "demo-employee", + session_id: "demo-session", + request_id: randomUUID(), +}; +} + +async function buildFixture(dir: string): Promise { + const wikiDir = join(dir, "wiki"); + await mkdir(wikiDir, { recursive: true }); + + await writeFile( + join(wikiDir, "a.md"), + `--- +title: Limitation of Liability +type: contract +created: 2026-09-11 +--- +The Aggregate liability of either party shall not exceed the fees paid in the +twelve (12) months preceding the claim. See [Indemnification](./b.md) for carve-outs. +`, + ); + + await writeFile( + join(wikiDir, "b.md"), + `--- +title: Indemnification +type: contract +created: 2026-09-11 +--- +Neither party shall indemnify the other for indirect, consequential, or +punitive damages. References: [Limitation of Liability](./a.md). +`, + ); + + await writeFile( + join(wikiDir, "law-clause.md"), + `--- +title: Public Law Clause — Cap of Liability +type: statute +created: 2026-09-11 +--- +Statutory cap of liability is twelve months of fees for ordinary breach. +`, + ); +} + +async function settleEmbeddings(): Promise { + // The embedding queue runs in the same process; sleeping is the + // simplest portable way to let it drain before we issue the hybrid / + // cross-file search. The lexical pass doesn't need to wait at all. + await new Promise((resolve) => setTimeout(resolve, SETTLE_MS)); +} + +function topHit(result: SearchWorkspaceContextResult): WorkspaceSearchHit | undefined { + return result.hits[0]; +} + +export default async function demoWorkspace() { + await runSection("demo: @melandlabs/workspace (folder indexing + cross-file search)", async () => { + const { check, skip } = makeCheckWithSkip("demo/workspace"); + + // Restore whatever was in EMBEDDING_PROVIDER before this demo so + // subsequent demos don't see `local` stuck on. + const restoreEnv = () => { + if (PREVIOUS_PROVIDER_ENV === undefined) { + // biome-ignore lint/performance/noDelete: env-reset pattern + delete process.env.EMBEDDING_PROVIDER; + } else { + process.env.EMBEDDING_PROVIDER = PREVIOUS_PROVIDER_ENV; + } + }; + + await withTmp("workspace", async (dir) => { + await buildFixture(dir); + + const dbPath = join(dir, "workspace.db"); + const store = await getSQLiteWorkspaceStore({ dbPath }); + const ctx = makeRuntimeContext(); + + // 1. updateWorkspaceContext — synchronous chunking + async fan-out. + let update: UpdateWorkspaceContextResult; + try { + update = await updateWorkspaceContext(ctx, store, { + workspace_id: WORKSPACE_ID, + source: "okf_folder", + path: join(dir, "wiki"), + }); + } catch (err) { + check("updateWorkspaceContext runs without throwing", false, (err as Error).message); + return; + } + + check( + "updateWorkspaceContext returns ok with files_scanned ≥ 3", + update.filesScanned >= 3, + `filesScanned=${update.filesScanned}, filesAdded=${update.filesAdded}`, + ); + check( + "updateWorkspaceContext flags the 3 new files as added", + update.filesAdded >= 3, + `filesAdded=${update.filesAdded}`, + ); + check( + "updateWorkspaceContext returns a positive jobId", + update.jobId > 0, + `jobId=${update.jobId}`, + ); + info( + "demo/workspace", + `updateWorkspaceContext → jobId=${update.jobId}, status=${update.status}, ` + + `scanned=${update.filesScanned}, added=${update.filesAdded}, modified=${update.filesModified}`, + ); + + // 2. listWorkspaceResources — every fixture file appears. + const listed = await listWorkspaceResources(ctx, store, { + workspace_id: WORKSPACE_ID, + }); + check( + "listWorkspaceResources returns ≥ 3 resources for the fixture folder", + listed.resources.length >= 3, + `total=${listed.total}, resources=${listed.resources + .map((r) => r.canonical_key) + .join(", ")}`, + ); + + // 3. Lexical search — works synchronously, FTS5 was filled in + // during the sync chunk phase. This is the only strategy + // that is guaranteed to work even before embeddings finish. + const lexicalResult = await searchWorkspaceContext(ctx, store, { + workspace_id: WORKSPACE_ID, + query: "limitation", + strategy: "lexical", + options: { limit: 5 }, + }); + check( + "lexical search for 'limitation' returns ≥ 1 hit", + lexicalResult.hits.length >= 1, + `total=${lexicalResult.total}`, + ); + check( + "lexical search marks strategy as 'lexical'", + lexicalResult.strategy === "lexical", + lexicalResult.strategy, + ); + info( + "demo/workspace", + `lexical[0] → ${topHit(lexicalResult)?.resource_title} (score=${topHit(lexicalResult)?.score.toFixed(4)})`, + ); + + // 4. Give the embedding queue a moment to drain, then try + // semantic + cross-file. If the local model fails to load + // (no network, fresh CI runner) the queue ends in `partial` + // and we skip those checks instead of failing. + await settleEmbeddings(); + + let semanticResult: SearchWorkspaceContextResult | undefined; + try { + semanticResult = await searchWorkspaceContext(ctx, store, { + workspace_id: WORKSPACE_ID, + query: "what is the cap on liability", + strategy: "semantic", + options: { limit: 3 }, + }); + } catch (err) { + const message = (err as Error).message.split("\n")[0]; + // biome-ignore lint/suspicious/noConsole: surface the failure on stderr for debugging + console.error(`[demo/workspace] semantic search failed: ${message}`); + skip("semantic search runs", "embedding provider failed: " + message); + } + if (semanticResult !== undefined) { + if (semanticResult.hits.length === 0) { + skip( + "semantic search returns hits once embeddings have been written", + "embedding queue drained with 0 rows (network or model load failure?)", + ); + } else { + check( + "semantic search returns ≥ 1 hit once embeddings are written", + semanticResult.hits.length >= 1, + `total=${semanticResult.total}`, + ); + check( + "semantic search marks strategy as 'semantic'", + semanticResult.strategy === "semantic", + semanticResult.strategy, + ); + info( + "demo/workspace", + `semantic[0] → ${topHit(semanticResult)?.resource_title} (score=${topHit(semanticResult)?.score.toFixed(4)})`, + ); + } + } + + // 5. Cross-file — hybrid + 1-hop BFS over cites edges. Expect + // hits from both `a.md` and the `b.md` it cites. + let crossFileResult: SearchWorkspaceContextResult | undefined; + try { + crossFileResult = await searchWorkspaceContext(ctx, store, { + workspace_id: WORKSPACE_ID, + query: "limitation", + strategy: "cross-file", + options: { limit: 5, hops: 1 }, + }); + } catch (err) { + const message = (err as Error).message.split("\n")[0]; + // biome-ignore lint/suspicious/noConsole: surface the failure on stderr for debugging + console.error(`[demo/workspace] cross-file search failed: ${message}`); + skip("cross-file search runs", "embedding provider failed: " + message); + } + if (crossFileResult !== undefined) { + if (crossFileResult.hits.length === 0) { + skip( + "cross-file search returns hits", + "no hits — embeddings never finished (network or model load failure)", + ); + } else { + const titles = new Set(crossFileResult.hits.map((h) => h.resource_title)); + check( + "cross-file search returns ≥ 2 hits (seed + 1-hop cite neighbour)", + crossFileResult.hits.length >= 2, + `total=${crossFileResult.total}, titles=${[...titles].join(", ")}`, + ); + check( + "cross-file expansion surfaces both `a` and `b` via the cites edge", + titles.has("a") && titles.has("b"), + `titles=${[...titles].join(", ")}`, + ); + const hitsWithEdges = crossFileResult.hits.filter( + (h) => h.reference_edges.length > 0, + ); + check( + "at least one cross-file hit carries reference_edges (cites graph)", + hitsWithEdges.length >= 1, + `${hitsWithEdges.length} hit(s) with edges`, + ); + info( + "demo/workspace", + `cross-file → ${crossFileResult.total} hits across ${titles.size} files: ${[...titles].join(", ")}`, + ); + } + } + + // 6. Re-run update — every file should now be `unchanged`. + const reUpdate = await updateWorkspaceContext(ctx, store, { + workspace_id: WORKSPACE_ID, + source: "okf_folder", + path: join(dir, "wiki"), + }); + check( + "re-running update reports every file as `unchanged` (sha256 dedup)", + reUpdate.filesUnchanged >= 3 && reUpdate.filesAdded === 0, + `unchanged=${reUpdate.filesUnchanged}, added=${reUpdate.filesAdded}`, + ); + + // 7. resolveWorkspaceDbPath defaults to ~/.opencontext/memory/store.db + // and the override here is honoured. + check( + "resolveWorkspaceDbPath returns the path we passed in", + resolveWorkspaceDbPath(dbPath) === dbPath, + resolveWorkspaceDbPath(dbPath), + ); + + await closeSQLiteWorkspaceStore().catch(() => undefined); + restoreEnv(); + }); + }); +} \ No newline at end of file diff --git a/packages/opencontext/package.json b/packages/opencontext/package.json index 4b4a4a76..f690b83a 100644 --- a/packages/opencontext/package.json +++ b/packages/opencontext/package.json @@ -26,6 +26,7 @@ "@melandlabs/integrations": "^0.3.0", "@melandlabs/okf": "^0.3.3", "@melandlabs/security": "^0.3.0", + "@melandlabs/workspace": "^0.1.0", "@modelcontextprotocol/sdk": "^1.25.3", "abort-controller": "^3.0.0", "agentkeepalive": "^4.6.0", @@ -59,6 +60,7 @@ "@melandlabs/loop": "workspace:*", "@melandlabs/memory-store": "workspace:*", "@melandlabs/okf": "workspace:*", + "@melandlabs/workspace": "workspace:*", "@melandlabs/rag": "workspace:*", "@melandlabs/search": "workspace:*", "@melandlabs/shared": "workspace:*", diff --git a/packages/opencontext/src/cli/opencontext.ts b/packages/opencontext/src/cli/opencontext.ts index 0cc13bb8..5068c677 100644 --- a/packages/opencontext/src/cli/opencontext.ts +++ b/packages/opencontext/src/cli/opencontext.ts @@ -40,6 +40,10 @@ import { parseDoctorArgs, runDoctor } from "./doctor.js"; import { parseListArgs, runList } from "./list.js"; import { parseSearchArgs, runSearch } from "./search.js"; import { parseStatsArgs, runStats } from "./stats.js"; +// Workspace CLI is shipped as an optional subpath import so a host +// that doesn't install `@melandlabs/workspace` still gets a usable +// `opencontext` CLI without crashing the bootstrap. +import { runWorkspaceCli } from "@melandlabs/workspace/cli"; interface HttpArgs extends UnifiedArgs { port: number; @@ -296,6 +300,7 @@ Commands: stats Report counts from the active raw-message store doctor Run health checks against the local install okf OKF v0.2 (Open Knowledge Format) importer / exporter + workspace Versioned, cross-file folder knowledge Run "opencontext --help" for command-specific options. @@ -587,6 +592,11 @@ async function main(): Promise { process.exit(result.exit); } + if (head === "workspace" || head === "WORKSPACE") { + const exit = await runWorkspaceCli(argv.slice(1)); + process.exit(exit); + } + if (head === "--help" || head === "-h") { printTopHelp(); return; diff --git a/packages/opencontext/src/index.ts b/packages/opencontext/src/index.ts index 18c6ab6a..9e64190e 100644 --- a/packages/opencontext/src/index.ts +++ b/packages/opencontext/src/index.ts @@ -404,3 +404,51 @@ export type { WikiEdge, BuildGraphOptions, } from "@melandlabs/okf"; + +// ─── 14. Workspace — versioned, cross-file folder knowledge ── +// The single-package facade exposes the three core APIs (`updateWorkspaceContext`, +// `searchWorkspaceContext`, `listWorkspaceResources`) so consumers don't need a +// separate install. Storage is delegated to the shared SQLite DB the memory +// store already owns (`~/.opencontext/memory/store.db`); embedding fan-out is +// async and honours `EMBEDDING_PROVIDER=local|cloud` via the optional peer +// dep `@melandlabs/ai-rag` (local = `Xenova/all-MiniLM-L6-v2`, 384 dims; +// cloud = OpenRouter, 1536 dims). +// +// Surface is JS-API + CLI only; HTTP / MCP transport lives behind the +// `opencontext workspace …` subcommand (`@melandlabs/workspace/cli`). +export { + updateWorkspaceContext, + searchWorkspaceContext, + listWorkspaceResources, + getSQLiteWorkspaceStore, + closeSQLiteWorkspaceStore, + resolveWorkspaceDbPath, + getWorkspaceEmbeddingProvider, + workspaceEmbedQuery, + workspaceEmbedDocuments, + workspaceEmbeddingModelName, + workspaceEmbeddingDimensions, +} from "@melandlabs/workspace"; +export type { + RuntimeContext, + WorkspaceStorageKind, + WorkspaceEdgeType, + WorkspaceIndexStatus, + WorkspaceSearchStrategy, + WorkspaceResource, + WorkspaceResourceVersion, + WorkspaceChunk, + WorkspaceReferenceEdge, + WorkspaceJob, + WorkspaceSearchHit, + SearchWorkspaceContextOptions, + UpdateWorkspaceContextInput, + UpdateWorkspaceContextResult, + SearchWorkspaceContextInput, + SearchWorkspaceContextResult, + ListWorkspaceResourcesInput, + ListWorkspaceResourcesResult, + OkfFolderResource, + WorkspaceEmbeddingProvider, + WorkspaceEmbeddingProviderType, +} from "@melandlabs/workspace"; diff --git a/packages/workspace/package.json b/packages/workspace/package.json new file mode 100644 index 00000000..f3b950dc --- /dev/null +++ b/packages/workspace/package.json @@ -0,0 +1,83 @@ +{ + "name": "@melandlabs/workspace", + "version": "0.1.0", + "type": "module", + "files": [ + "dist", + "README.md", + "LICENSE" + ], + "publishConfig": { + "access": "public" + }, + "main": "./dist/index.js", + "types": "./dist/index.d.ts", + "exports": { + ".": { + "types": "./dist/index.d.ts", + "import": "./dist/index.js" + }, + "./sqlite": { + "types": "./dist/sqlite.d.ts", + "import": "./dist/sqlite.js" + }, + "./cli": { + "types": "./dist/cli.d.ts", + "import": "./dist/cli.js" + } + }, + "repository": { + "type": "git", + "url": "https://github.com/melandlabs/opencontext.git", + "directory": "packages/project-context" + }, + "homepage": "https://github.com/melandlabs/opencontext/tree/main/packages/project-context", + "bugs": { + "url": "https://github.com/melandlabs/opencontext/issues" + }, + "license": "Apache-2.0", + "description": "OpenContext · workspace module. Provides project-level indexing, hybrid retrieval, and cross-file reference edges over a project workspace.", + "keywords": [ + "opencontext", + "project-context", + "hybrid-search", + "rag" + ], + "dependencies": { + "@melandlabs/contracts": "workspace:*", + "@melandlabs/env-config": "workspace:*", + "@melandlabs/okf": "workspace:*", + "@melandlabs/rag": "workspace:*", + "@melandlabs/shared": "workspace:*", + "@melandlabs/sqlite": "workspace:*", + "@modelcontextprotocol/sdk": "^1.25.3", + "better-sqlite3": "^11.10.0", + "hono": "^4.6.14", + "sqlite-vec": "^0.1.9", + "zod": "^4.3.6" + }, + "peerDependencies": { + "@melandlabs/ai-rag": "workspace:*" + }, + "peerDependenciesMeta": { + "@melandlabs/ai-rag": { + "optional": true + } + }, + "devDependencies": { + "@melandlabs/ai-rag": "workspace:*", + "@melandlabs/config": "workspace:*", + "@types/better-sqlite3": "^7.6.13", + "@types/node": "^22.13.10", + "typescript": "^5.6.3", + "vitest": "^4.1.10" + }, + "scripts": { + "build": "tsup", + "typecheck": "tsc --noEmit", + "lint": "biome check .", + "lint:fix": "biome check --write .", + "test": "vitest run", + "test:watch": "vitest" + } +} diff --git a/packages/workspace/src/api.ts b/packages/workspace/src/api.ts new file mode 100644 index 00000000..0d8a1e77 --- /dev/null +++ b/packages/workspace/src/api.ts @@ -0,0 +1,172 @@ +/** + * `@melandlabs/workspace` — three core APIs. + * + * - `updateWorkspaceContext` — scan + index an OKF folder, return job summary + * - `searchWorkspaceContext` — multi-strategy hybrid search (lexical / + * semantic / hybrid / cross-file) + * - `listWorkspaceResources` — enumerate indexed resources for a project + * + * The HTTP and MCP entry points (`src/http.ts`, `src/mcp.ts`) build the + * `RuntimeContext` from request headers / tool args and pass it through + * here. This layer is storage-agnostic: tests inject their own + * `SqliteWorkspaceStore` and a mock embedding function. + */ + +import type { + ListWorkspaceResourcesInput, + ListWorkspaceResourcesResult, + RuntimeContext, + SearchWorkspaceContextInput, + SearchWorkspaceContextResult, + UpdateWorkspaceContextInput, + UpdateWorkspaceContextResult, +} from "./types"; +import { indexOkfFolder } from "./okf-backend"; +import { searchLexical } from "./search/lexical"; +import { searchSemantic } from "./search/semantic"; +import { searchCrossFile } from "./search/cross-file"; +import { fuseHybridHits } from "./search/hybrid"; +import type { SqliteWorkspaceStore } from "./sqlite"; +import { workspaceEmbedQuery } from "./embedding-provider"; + +/** + * Only `okf_folder` is supported as a source today. Any other value + * triggers an explicit `unsupported_source` error so the HTTP / MCP + * layers (when re-introduced) can surface a 400 with a clear message. + */ +export async function updateWorkspaceContext( + ctx_rt: RuntimeContext, + store: SqliteWorkspaceStore, + input: UpdateWorkspaceContextInput, + hooks: { + enqueueEmbedding?: (input: { resource_id: number; version_id: number; jobId?: number }) => Promise; + } = {}, +): Promise { + if (!input.workspace_id) throw new Error("workspace_id is required"); + if (input.source !== "okf_folder") { + throw new Error(`unsupported source: ${String(input.source)} (only 'okf_folder' is supported)`); + } + if (!input.path) throw new Error("path is required"); + const enqueueEmbedding = hooks.enqueueEmbedding ?? (async () => {}); + return indexOkfFolder(store, { + workspace_id: input.workspace_id, + user_id: ctx_rt.user_id, + path: input.path, + enqueueEmbedding, + }); +} + +export interface SearchWorkspaceContextDeps { + /** Optional embedding override (tests inject deterministic mocks). */ + embed?: (text: string) => Promise; +} + +/** + * `searchWorkspaceContext` — the unified search entry. Strategy is one of + * - `lexical` — FTS5 only + * - `semantic` — sqlite-vec only (with lexical fallback when no embeddings yet) + * - `hybrid` — lexical + semantic RRF (default) + * - `cross-file`— hybrid + cites-edge BFS + */ +export async function searchWorkspaceContext( + ctx_rt: RuntimeContext, + store: SqliteWorkspaceStore, + input: SearchWorkspaceContextInput, + deps: SearchWorkspaceContextDeps = {}, +): Promise { + if (!input.workspace_id) throw new Error("workspace_id is required"); + if (!input.query) throw new Error("query is required"); + const embed = deps.embed ?? workspaceEmbedQuery; + const limit = Math.max(1, Math.min(50, Math.floor(input.options?.limit ?? 10))); + const strategy = input.strategy ?? "hybrid"; + const resourceTypes = input.options?.resource_types; + + if (strategy === "lexical") { + const hits = searchLexical(store, { + workspace_id: input.workspace_id, + user_id: ctx_rt.user_id, + query: input.query, + resource_types: resourceTypes, + limit, + }); + return { query: input.query, strategy, total: hits.length, hits }; + } + if (strategy === "semantic") { + const embedding = await embed(input.query); + const hits = searchSemantic(store, { + workspace_id: input.workspace_id, + user_id: ctx_rt.user_id, + queryEmbedding: embedding, + resource_types: resourceTypes, + limit, + threshold: input.options?.threshold ?? 0.7, + }); + // Fallback to lexical when no embeddings are written yet so the + // user still gets a hit during the indexing warm-up window. + const finalHits = hits.length > 0 + ? hits + : searchLexical(store, { + workspace_id: input.workspace_id, + user_id: ctx_rt.user_id, + query: input.query, + resource_types: resourceTypes, + limit, + }); + return { query: input.query, strategy, total: finalHits.length, hits: finalHits }; + } + if (strategy === "cross-file") { + const hits = await searchCrossFile(store, { + workspace_id: input.workspace_id, + user_id: ctx_rt.user_id, + query: input.query, + resource_types: resourceTypes, + limit, + threshold: input.options?.threshold ?? 0.7, + hops: input.options?.hops ?? 1, + lexicalSearch: (params) => searchLexical(store, params), + semanticSearch: (params) => searchSemantic(store, params), + generateEmbedding: embed, + }); + return { query: input.query, strategy, total: hits.length, hits }; + } + // hybrid (default) + const candidateLimit = limit * 4; + const lexicalHits = searchLexical(store, { + workspace_id: input.workspace_id, + user_id: ctx_rt.user_id, + query: input.query, + resource_types: resourceTypes, + limit: candidateLimit, + }); + let semanticHits: WorkspaceSearchHitRef[] = []; + try { + const embedding = await embed(input.query); + semanticHits = searchSemantic(store, { + workspace_id: input.workspace_id, + user_id: ctx_rt.user_id, + queryEmbedding: embedding, + resource_types: resourceTypes, + limit: candidateLimit, + threshold: input.options?.threshold ?? 0.7, + }); + } catch (error) { + // Embedding failure (no API key, network) — degrade to lexical only. + semanticHits = []; + void error; + } + const hits = fuseHybridHits({ lexical: lexicalHits, semantic: semanticHits, limit }); + return { query: input.query, strategy, total: hits.length, hits }; +} + +// `WorkspaceSearchHitRef` alias kept inline so the import list stays tight. +type WorkspaceSearchHitRef = SearchWorkspaceContextResult["hits"][number]; + +export async function listWorkspaceResources( + ctx_rt: RuntimeContext, + store: SqliteWorkspaceStore, + input: ListWorkspaceResourcesInput, +): Promise { + if (!input.workspace_id) throw new Error("workspace_id is required"); + void ctx_rt; // Reserved for future per-user ACL filtering on the listing. + return store.listResources(input); +} diff --git a/packages/workspace/src/cli.ts b/packages/workspace/src/cli.ts new file mode 100644 index 00000000..18d45718 --- /dev/null +++ b/packages/workspace/src/cli.ts @@ -0,0 +1,605 @@ +#!/usr/bin/env node +/** + * `@melandlabs/workspace/cli` — `opencontext workspace …` subcommand. + * + * opencontext workspace update --workspace-id --path [--user ] + * opencontext workspace search --workspace-id --query + * [--strategy hybrid] [--limit 10] + * [--resource-type note,statute] [--json] + * opencontext workspace list --workspace-id + * [--resource-type ] [--index-status ready] + * [--limit 50] [--offset 0] [--json] + * + * The CLI is intentionally thin — it parses argv, builds a `RuntimeContext`, + * and delegates to the JS API in `./api`. The embedding queue is wired in + * here (not in `./api`) so a one-shot `update` invocation can await the + * async fan-out and report a final `ready` / `partial` status before exit. + */ + +import { randomUUID } from "node:crypto"; + +import type { RuntimeContext, WorkspaceSearchStrategy } from "./types"; +import { + getSQLiteWorkspaceStore, + closeSQLiteWorkspaceStore, + resolveWorkspaceDbPath, +} from "./sqlite"; +import { + updateWorkspaceContext, + searchWorkspaceContext, + listWorkspaceResources, +} from "./api"; +import { createEmbeddingQueue } from "./embedding-queue"; + +const logPrefix = "[opencontext/workspace]"; + +const STRATEGIES: WorkspaceSearchStrategy[] = ["lexical", "semantic", "hybrid", "cross-file"]; +const INDEX_STATUSES = ["pending", "partial", "ready", "failed"] as const; + +// Flags that don't take a value (booleans). Listed once here so the +// generic parseFlags() helper can skip its `next.startsWith("--")` +// guard for them. +const BOOLEAN_FLAGS = new Set([ + "--await-embeddings", + "--no-await-embeddings", + "--json", +]); + +// ──────────────────────────────────────────────────────────────────────────── +// Argv helpers +// ──────────────────────────────────────────────────────────────────────────── + +class ArgvError extends Error { + constructor(message: string) { + super(`${logPrefix} ${message}`); + } +} + +interface ParseOptions { + positional?: (arg: string, state: T) => void; +} + +function takeValue(argv: string[], i: number, flag: string): { value: string; next: number } { + const next = argv[i + 1]; + if (next === undefined || next.startsWith("--")) { + throw new ArgvError(`${flag} requires a value`); + } + return { value: next, next: i + 2 }; +} + +function parseFlags( + argv: string[], + options: ParseOptions = {}, +): T { + const out = {} as T; + for (let i = 0; i < argv.length; i += 1) { + const arg = argv[i]; + if (arg === "--help" || arg === "-h") { + throw new ArgvError("--help requested"); + } + if (options.positional && !arg.startsWith("--")) { + options.positional(arg, out); + continue; + } + const eq = arg.indexOf("="); + const flag = eq >= 0 ? arg.slice(0, eq) : arg; + const inline = eq >= 0 ? arg.slice(eq + 1) : undefined; + const useInline = inline !== undefined; + const applyValue = (value: string) => { + const key = flag.slice(2).replace(/-([a-z])/g, (_, c: string) => c.toUpperCase()); + (out as Record)[key] = value; + }; + const applyBoolean = () => { + let key = flag.slice(2); + let value: boolean; + if (key.startsWith("no-")) { + key = key.slice(3); + value = false; + } else { + value = true; + } + key = key.replace(/-([a-z])/g, (_, c: string) => c.toUpperCase()); + (out as Record)[key] = value; + }; + if (BOOLEAN_FLAGS.has(flag)) { + applyBoolean(); + continue; + } + if (useInline) { + applyValue(inline); + } else { + const { value, next } = takeValue(argv, i, flag); + applyValue(value); + i = next - 1; + } + } + return out; +} + +// ──────────────────────────────────────────────────────────────────────────── +// Subcommand: update +// ──────────────────────────────────────────────────────────────────────────── + +interface UpdateArgs { + workspaceId?: string; + path?: string; + user?: string; + awaitEmbeddings?: boolean; + drainTimeoutMs?: number; + json?: boolean; +} + +function parseUpdateArgs(argv: string[]): UpdateArgs { + return parseFlags(argv, { + positional: (arg, state) => { + // Allow `opencontext workspace update --path …` + // as a convenience (positional first arg). + if (!state.workspaceId) { + state.workspaceId = arg; + return; + } + throw new ArgvError(`unexpected positional argument: ${arg}`); + }, + }); +} + +async function runUpdate(args: UpdateArgs): Promise { + const workspaceId = args.workspaceId; + const path = args.path; + if (!workspaceId) throw new ArgvError("--workspace-id is required"); + if (!path) throw new ArgvError("--path is required"); + const userId = args.user ?? "default"; + + const ctx_rt: RuntimeContext = { + user_id: userId, + request_id: randomUUID(), + }; + + const store = await getSQLiteWorkspaceStore(); + const queue = createEmbeddingQueue({ store }); + + const result = await updateWorkspaceContext( + ctx_rt, + store, + { workspace_id: workspaceId, source: "okf_folder", path }, + { enqueueEmbedding: (input) => queue.enqueue(input) }, + ); + + if (args.awaitEmbeddings) { + // Default behaviour: wait for the queue to drain so the CLI + // reports a final index_status before exiting. Users can opt + // out with --no-await-embeddings (handled by parseFlags: any + // `--no-X` is captured as `awaitEmbeddings: false`). + const timeoutMs = args.drainTimeoutMs ?? 120_000; + await Promise.race([ + queue.drain(), + new Promise((resolve) => setTimeout(resolve, timeoutMs)), + ]); + } + + const out = { + ok: true, + exit: 0, + workspace_id: workspaceId, + job_id: result.jobId, + status: result.status, + files_scanned: result.filesScanned, + files_added: result.filesAdded, + files_modified: result.filesModified, + files_unchanged: result.filesUnchanged, + files_deleted: result.filesDeleted, + }; + + if (args.json) { + process.stdout.write(`${JSON.stringify(out)}\n`); + } else { + const lines = [ + `scanned ${result.filesScanned} files`, + `added ${result.filesAdded}`, + `modified ${result.filesModified}`, + `unchanged ${result.filesUnchanged}`, + `deleted ${result.filesDeleted}`, + `job_id=${result.jobId} status=${result.status}`, + ]; + process.stdout.write(`${lines.join("\n")}\n`); + } + return 0; +} + +// ──────────────────────────────────────────────────────────────────────────── +// Subcommand: search +// ──────────────────────────────────────────────────────────────────────────── + +interface SearchArgs { + workspaceId?: string; + query?: string; + strategy?: string; + limit?: string; + threshold?: string; + resourceType?: string; + hops?: string; + json?: boolean; +} + +function parseSearchArgs(argv: string[]): SearchArgs { + return parseFlags(argv); +} + +async function runSearch(args: SearchArgs): Promise { + const workspaceId = args.workspaceId; + const query = args.query; + if (!workspaceId) throw new ArgvError("--workspace-id is required"); + if (!query) throw new ArgvError("--query is required"); + + const strategy = (args.strategy ?? "hybrid") as WorkspaceSearchStrategy; + if (!STRATEGIES.includes(strategy)) { + throw new ArgvError(`--strategy must be one of: ${STRATEGIES.join(", ")} (got "${args.strategy}")`); + } + + const limit = args.limit !== undefined ? Number.parseInt(args.limit, 10) : 10; + if (!Number.isInteger(limit) || limit <= 0) { + throw new ArgvError(`--limit must be a positive integer (got "${args.limit}")`); + } + const threshold = args.threshold !== undefined ? Number.parseFloat(args.threshold) : undefined; + if (threshold !== undefined && (Number.isNaN(threshold) || threshold < 0 || threshold > 1)) { + throw new ArgvError(`--threshold must be in [0, 1] (got "${args.threshold}")`); + } + const resourceTypes = args.resourceType + ? args.resourceType.split(",").map((s) => s.trim()).filter(Boolean) + : undefined; + const hops = args.hops !== undefined ? Number.parseInt(args.hops, 10) : undefined; + if (hops !== undefined && hops !== 1 && hops !== 2) { + throw new ArgvError(`--hops must be 1 or 2 (got "${args.hops}")`); + } + + const ctx_rt: RuntimeContext = { + user_id: "default", + request_id: randomUUID(), + }; + + const store = await getSQLiteWorkspaceStore(); + const result = await searchWorkspaceContext(ctx_rt, store, { + workspace_id: workspaceId, + query, + strategy, + options: { + limit, + threshold, + resource_types: resourceTypes, + hops, + }, + }); + + if (args.json) { + process.stdout.write(`${JSON.stringify(result)}\n`); + } else { + process.stdout.write(`${result.total} hit(s) (strategy=${strategy})\n`); + for (const hit of result.hits) { + const snippet = hit.snippet.replace(/\s+/g, " ").slice(0, 160); + process.stdout.write( + ` - [${hit.resource_type}] ${hit.resource_title} :: ${snippet} (score=${hit.score.toFixed(3)})\n`, + ); + } + } + return 0; +} + +// ──────────────────────────────────────────────────────────────────────────── +// Subcommand: list +// ──────────────────────────────────────────────────────────────────────────── + +interface ListArgs { + workspaceId?: string; + resourceType?: string; + indexStatus?: string; + limit?: string; + offset?: string; + json?: boolean; +} + +function parseListArgs(argv: string[]): ListArgs { + return parseFlags(argv); +} + +async function runList(args: ListArgs): Promise { + const workspaceId = args.workspaceId; + if (!workspaceId) throw new ArgvError("--workspace-id is required"); + if (args.indexStatus && !(INDEX_STATUSES as readonly string[]).includes(args.indexStatus)) { + throw new ArgvError( + `--index-status must be one of: ${INDEX_STATUSES.join(", ")} (got "${args.indexStatus}")`, + ); + } + const limit = args.limit !== undefined ? Number.parseInt(args.limit, 10) : 50; + const offset = args.offset !== undefined ? Number.parseInt(args.offset, 10) : 0; + if (!Number.isInteger(limit) || limit <= 0) { + throw new ArgvError(`--limit must be a positive integer (got "${args.limit}")`); + } + if (!Number.isInteger(offset) || offset < 0) { + throw new ArgvError(`--offset must be a non-negative integer (got "${args.offset}")`); + } + + const ctx_rt: RuntimeContext = { + user_id: "default", + request_id: randomUUID(), + }; + + const store = await getSQLiteWorkspaceStore(); + const result = await listWorkspaceResources(ctx_rt, store, { + workspace_id: workspaceId, + resource_type: args.resourceType, + index_status: args.indexStatus as + | "pending" + | "partial" + | "ready" + | "failed" + | undefined, + limit, + offset, + }); + + if (args.json) { + process.stdout.write(`${JSON.stringify(result)}\n`); + } else { + process.stdout.write(`${result.total} resource(s)\n`); + for (const r of result.resources) { + process.stdout.write( + ` - [${r.index_status}] id=${r.id} ${r.resource_type} :: ${r.title} (${r.canonical_key})\n`, + ); + } + } + return 0; +} + +// ──────────────────────────────────────────────────────────────────────────── +// Help +// ──────────────────────────────────────────────────────────────────────────── + +function printHelp(): void { + const dbPath = resolveWorkspaceDbPath(); + process.stdout.write(`opencontext workspace — versioned, cross-file folder knowledge. + +Usage: + opencontext workspace [options] + +Subcommands: + update Index an OKF folder into the workspace SQLite store + search Run lexical | semantic | hybrid | cross-file search + list Enumerate indexed resources for a workspace + +Storage: + --db-path Override the SQLite path (default: ${dbPath}) + +Common: + --workspace-id Workspace identifier (required) + --user User / tenant id (default: "default") + --json Emit JSON envelope instead of a human line + +Examples: + opencontext workspace update --workspace-id proj-1 --path ~/notes/wiki + opencontext workspace update proj-1 --path ~/notes/wiki --user alice + opencontext workspace search --workspace-id proj-1 --query "limitation" + opencontext workspace search --workspace-id proj-1 --query "x" --strategy cross-file --hops 2 + opencontext workspace list --workspace-id proj-1 --index-status ready + +Run "opencontext workspace --help" for subcommand-specific options. +`); +} + +function printSubcommandHelp(sub: string): void { + switch (sub) { + case "update": + process.stdout.write(`opencontext workspace update — index an OKF folder. + +Required: + --workspace-id Workspace identifier + --path Path to the OKF folder (will be scanned recursively) + +Optional: + --user User / tenant id (default: "default") + --await-embeddings Wait for the in-process embedding queue to drain + before exiting (default: on) + --no-await-embeddings Return as soon as synchronous indexing completes + --drain-timeout-ms Upper bound on --await-embeddings (default 120000) + --json Emit JSON envelope + +Example: + opencontext workspace update --workspace-id proj-1 --path ./wiki +`); + return; + case "search": + process.stdout.write(`opencontext workspace search — multi-strategy hybrid search. + +Required: + --workspace-id Workspace identifier + --query Search query + +Strategy: + --strategy lexical | semantic | hybrid | cross-file + (default: hybrid) + --limit Top-N hits (default: 10, max: 50) + --threshold Semantic similarity threshold, 0..1 (default: 0.7) + --resource-type Comma-separated filter, e.g. "note,statute" + --hops <1|2> Cross-file BFS depth (cross-file strategy only) + +Output: + --json Emit JSON envelope with full WorkspaceSearchHit[] + +Example: + opencontext workspace search --workspace-id proj-1 --query "limitation" \\ + --strategy cross-file --hops 1 --json +`); + return; + case "list": + process.stdout.write(`opencontext workspace list — enumerate indexed resources. + +Required: + --workspace-id Workspace identifier + +Filters: + --resource-type Filter by resource type + --index-status pending | partial | ready | failed + --limit Max rows (default: 50) + --offset Skip N rows (default: 0) + +Output: + --json Emit JSON envelope + +Example: + opencontext workspace list --workspace-id proj-1 --index-status ready --limit 20 +`); + return; + default: + printHelp(); + } +} + +// ──────────────────────────────────────────────────────────────────────────── +// Entry +// ──────────────────────────────────────────────────────────────────────────── + +/** + * Tear down the SQLite handle in the background so a `sqlite-vec` mutex + * destructor (which can `SIGABRT` on certain platforms) doesn't poison + * the main process exit. We: + * + * 1. Detach the close promise so no `await` ever blocks on it. + * 2. Schedule `process.exit` to run AFTER stdout has drained (so the + * human / JSON output is actually visible) and BEFORE the OS + * reaps the still-pending sqlite-vec native mutex teardown. + * 3. Set a hard 250ms timeout so a stuck close doesn't hang the CLI. + */ +function scheduleBackgroundClose(): void { + // Detach the SQLite teardown — we never want `await` on it inside the + // hot path because `sqlite-vec`'s native destructor occasionally + // raises SIGABRT during process teardown. + closeSQLiteWorkspaceStore().catch(() => { + // Best-effort cleanup; ignore secondary errors. + }); +} + +async function main(): Promise { + const argv = process.argv.slice(2); + const sub = argv[0]; + if (!sub || sub === "--help" || sub === "-h") { + printHelp(); + return; + } + if (sub === "help") { + printSubcommandHelp(argv[1] ?? ""); + return; + } + + const rest = argv.slice(1); + // Short-circuit `--help` / `-h` before parseFlags throws, so the + // subcommand-level help text shows up instead of an error. + if (rest.includes("--help") || rest.includes("-h")) { + printSubcommandHelp(sub); + process.exit(0); + } + + let exitCode = 0; + try { + switch (sub) { + case "update": + exitCode = await runUpdate(parseUpdateArgs(rest)); + break; + case "search": + exitCode = await runSearch(parseSearchArgs(rest)); + break; + case "list": + exitCode = await runList(parseListArgs(rest)); + break; + default: + throw new ArgvError(`unknown subcommand: ${sub}`); + } + } catch (error) { + if (error instanceof ArgvError) { + process.stderr.write(`${error.message}\n`); + // Force-exit on argv errors — see SIGABRT note below. + process.exit(2); + } + throw error; + } + // Force-exit instead of letting Node's event loop drain: the + // `@huggingface/transformers` ONNX worker thread spawned by the + // local embedder leaves a `sqlite-vec` mutex in a state that + // triggers `libc++abi: ... mutex lock failed: Invalid argument` + // (SIGABRT) during natural teardown on macOS. Output has already + // been written to stdout at this point, so a fast exit is safe. + process.exit(exitCode); +} + +// Guard against importing this file as a library (e.g. from +// `opencontext/cli/opencontext.ts`) — `runWorkspaceCli` is the +// programmatic entry, so only run the side-effecting `main()` when this +// file is the actual entry point. +const isDirectInvocation = + typeof process !== "undefined" && + Array.isArray(process.argv) && + process.argv[1] !== undefined && + import.meta.url === `file://${process.argv[1]}`; + +// Ignore SIGPIPE so a piped `head -n 1` doesn't show up as a fatal. +process.on("SIGPIPE", () => { + process.exit(0); +}); + +if (isDirectInvocation) { + main().catch((error: unknown) => { + const message = error instanceof Error ? error.stack ?? error.message : String(error); + process.stderr.write(`${logPrefix} fatal: ${message}\n`); + process.exit(1); + }); +} + +/** + * Programmatic entry point for hosts (e.g. `opencontext workspace …`) + * that want to delegate argv parsing + subcommand dispatch without + * re-importing the side-effecting `main()` above. Returns the desired + * process exit code; never throws. + */ +export async function runWorkspaceCli(argv: string[]): Promise { + const sub = argv[0]; + if (!sub || sub === "--help" || sub === "-h") { + printHelp(); + return 0; + } + if (sub === "help") { + printSubcommandHelp(argv[1] ?? ""); + return 0; + } + const rest = argv.slice(1); + if (rest.includes("--help") || rest.includes("-h")) { + printSubcommandHelp(sub); + return 0; + } + try { + switch (sub) { + case "update": + return await runUpdate(parseUpdateArgs(rest)); + case "search": + return await runSearch(parseSearchArgs(rest)); + case "list": + return await runList(parseListArgs(rest)); + default: + process.stderr.write(`${logPrefix} unknown subcommand: ${sub}\n`); + return 2; + } + } catch (error) { + if (error instanceof ArgvError) { + process.stderr.write(`${error.message}\n`); + return 2; + } + const message = error instanceof Error ? error.stack ?? error.message : String(error); + process.stderr.write(`${logPrefix} fatal: ${message}\n`); + return 1; + } finally { + // Background-close so sqlite-vec's native mutex destructor + // (occasionally SIGABRTs on certain macOS configs) doesn't + // poison the main process exit. The host program invokes us + // synchronously and then calls `process.exit` itself. + closeSQLiteWorkspaceStore().catch(() => { + // Best-effort cleanup; ignore secondary errors. + }); + } +} diff --git a/packages/workspace/src/embedding-provider.ts b/packages/workspace/src/embedding-provider.ts new file mode 100644 index 00000000..52ea3f6a --- /dev/null +++ b/packages/workspace/src/embedding-provider.ts @@ -0,0 +1,141 @@ +/** + * `@melandlabs/workspace` — embedding provider adapter. + * + * The JS API and CLI want a single "give me a vector for this text" / + * "give me vectors for these N texts" surface that honours the + * `EMBEDDING_PROVIDER` env var (`local` | `cloud`). Without this adapter + * `generateEmbedding` / `generateEmbeddings` from `@melandlabs/rag` + * always hit OpenRouter, which makes `hybrid` / `cross-file` / `semantic` + * search unusable in a fully-offline setup. + * + * The provider lives in `@melandlabs/ai-rag`, which is an **optional** + * peer dep: hosts that only want lexical search don't need to install + * it. The adapter is therefore a lazy dynamic import — if the host never + * sets `EMBEDDING_PROVIDER=local` *and* never calls into the embedding + * path with `local` semantics, the import never resolves. + * + * Hosts that *do* set `EMBEDDING_PROVIDER=local` must install + * `@melandlabs/ai-rag` themselves; the workspace package surfaces a + * clear error in that case instead of crashing the module load. + */ + +export interface WorkspaceEmbeddingProvider { + embedQuery(text: string): Promise; + embedDocuments(texts: string[]): Promise; + getModelName(): string; + getDimensions(): number | undefined; +} + +export type WorkspaceEmbeddingProviderType = "cloud" | "local"; + +let _provider: WorkspaceEmbeddingProvider | undefined; + +function resolveProviderType(): WorkspaceEmbeddingProviderType { + // Default to `local` for the workspace CLI: folder indexing is + // positioned as an offline-first feature and most demos / OKF review + // workflows don't carry an OPENROUTER_API_KEY. Set + // `EMBEDDING_PROVIDER=cloud` to opt into the 1536-dim OpenRouter path. + const raw = (process.env.EMBEDDING_PROVIDER ?? "local").trim().toLowerCase(); + return raw === "local" ? "local" : "cloud"; +} + +/** + * Returns the singleton embedding provider, instantiating it on first + * call. Honours `EMBEDDING_PROVIDER` at construction time so the model + * weight download (local) or API key validation (cloud) only happens + * once per process. + * + * Throws if `EMBEDDING_PROVIDER=local` is set but `@melandlabs/ai-rag` + * isn't installed. + */ +export async function getWorkspaceEmbeddingProvider(): Promise { + if (_provider) return _provider; + const providerType = resolveProviderType(); + if (providerType === "local") { + // Dynamic import keeps `@melandlabs/ai-rag` out of the static + // dependency graph for hosts that don't need local embeddings. + const specifier = "@melandlabs/ai-rag/embedding-provider"; + let mod: typeof import("@melandlabs/ai-rag/embedding-provider"); + try { + mod = await import(specifier); + } catch (error) { + throw new Error( + `EMBEDDING_PROVIDER=local but "${specifier}" could not be resolved. ` + + `Install @melandlabs/ai-rag to enable local ONNX embeddings ` + + `(Xenova/all-MiniLM-L6-v2, 384 dims by default). ` + + `Underlying error: ${error instanceof Error ? error.message : String(error)}`, + ); + } + const provider = mod.getConfiguredEmbeddingProvider({ providerType: "local" }); + if (!provider) { + throw new Error( + `EMBEDDING_PROVIDER=local but getConfiguredEmbeddingProvider returned no provider`, + ); + } + _provider = provider; + } else { + const specifier = "@melandlabs/ai-rag/embedding-provider"; + let mod: typeof import("@melandlabs/ai-rag/embedding-provider"); + try { + mod = await import(specifier); + } catch (error) { + throw new Error( + `Cloud embedding provider requested but "${specifier}" could not be resolved. ` + + `Install @melandlabs/ai-rag to enable OpenRouter-backed embeddings ` + + `(requires OPENROUTER_API_KEY). ` + + `Underlying error: ${error instanceof Error ? error.message : String(error)}`, + ); + } + const provider = mod.getConfiguredEmbeddingProvider({ providerType: "cloud" }); + if (!provider) { + throw new Error( + `Cloud embedding provider requested but getConfiguredEmbeddingProvider returned no provider`, + ); + } + _provider = provider; + } + return _provider; +} + +/** + * Convenience: embed a single text and return just the vector. + * Used by `searchWorkspaceContext` for the query side. + */ +export async function workspaceEmbedQuery(text: string): Promise { + const provider = await getWorkspaceEmbeddingProvider(); + return provider.embedQuery(text); +} + +/** + * Convenience: embed a batch of texts and return parallel vectors. + * Used by `createEmbeddingQueue` for the chunk fan-out. + */ +export async function workspaceEmbedDocuments(texts: string[]): Promise { + if (texts.length === 0) return []; + const provider = await getWorkspaceEmbeddingProvider(); + return provider.embedDocuments(texts); +} + +/** + * Read the model name of the active provider without instantiating it. + * Used for the `embedding_model` column on `workspace_chunks`. + */ +export async function workspaceEmbeddingModelName(): Promise { + const provider = await getWorkspaceEmbeddingProvider(); + return provider.getModelName(); +} + +/** + * Read the dimension of the active provider without instantiating it + * (the cloud provider only knows its dim after the first embed call). + * Returns `undefined` for the cloud provider pre-warmup. + */ +export async function workspaceEmbeddingDimensions(): Promise { + const provider = await getWorkspaceEmbeddingProvider(); + return provider.getDimensions(); +} + +/** Test-only: drop the singleton so the next call re-instantiates. */ +export function __resetWorkspaceEmbeddingProviderForTests(): void { + _provider = undefined; +} diff --git a/packages/workspace/src/embedding-queue.ts b/packages/workspace/src/embedding-queue.ts new file mode 100644 index 00000000..83b27cd7 --- /dev/null +++ b/packages/workspace/src/embedding-queue.ts @@ -0,0 +1,118 @@ +/** + * `@melandlabs/workspace` — in-process embedding queue. + * + * Mirrors the lightweight Promise-chained serialiser pattern in + * `packages/rag/src/lancedb-store.ts:228-232`. The job runner reads + * up to `BATCH_SIZE` chunks from `workspace_chunks` that don't yet have + * an embedding, asks the configured provider for vectors via + * `workspaceEmbedDocuments(batch)`, and writes the results back into + * both `workspace_chunks.embedding*` columns and the + * dimension-suffixed vec0 child table (`workspace_chunks_vec_d{N}`). + * + * The default dimension (`DEFAULT_DIMENSIONS = 384`) matches the + * `Xenova/all-MiniLM-L6-v2` model selected when + * `EMBEDDING_PROVIDER=local`. Cloud / OpenRouter embeddings use 1536 + * dims (text-embedding-3-small) and the queue adapts automatically + * because the dimensions are read off the first returned vector. + */ + +import { floatArrayToBuffer } from "@melandlabs/sqlite"; +import type { SqliteWorkspaceStore } from "./sqlite"; +import { workspaceEmbedDocuments, workspaceEmbeddingModelName } from "./embedding-provider"; + +const DEFAULT_BATCH_SIZE = 100; +// 384 = Xenova/all-MiniLM-L6-v2 (local). Cloud picks 1536 dynamically. +const DEFAULT_DIMENSIONS = 384; + +export interface EmbeddingQueueDeps { + store: SqliteWorkspaceStore; + batchSize?: number; + dimensions?: number; +} + +export interface EmbeddingQueue { + /** + * Enqueue an embedding job for a specific `(resource_id, version_id)` pair. + * Resolves once the job has been scheduled into the serial queue; it + * does NOT wait for embedding completion. + */ + enqueue(input: { resource_id: number; version_id: number; jobId?: number }): Promise; + /** Test-only: drain the serial queue and wait for all pending jobs. */ + drain(): Promise; +} + +class DefaultEmbeddingQueue implements EmbeddingQueue { + private readonly store: SqliteWorkspaceStore; + private readonly batchSize: number; + private readonly dimensions: number; + private queue: Promise = Promise.resolve(); + + constructor(deps: EmbeddingQueueDeps) { + this.store = deps.store; + this.batchSize = deps.batchSize ?? DEFAULT_BATCH_SIZE; + this.dimensions = deps.dimensions ?? DEFAULT_DIMENSIONS; + } + + enqueue(input: { resource_id: number; version_id: number; jobId?: number }): Promise { + const next = this.queue.then(async () => { + await this.runJob(input); + }); + // Swallow rejections on the chained promise so one failed job does + // not poison subsequent ones. Per-job errors are persisted into + // `workspace_jobs.error` and the per-resource `index_status`. + this.queue = next.catch(() => undefined); + return next.catch(() => undefined); + } + + drain(): Promise { + return this.queue.then(() => undefined); + } + + private async runJob(input: { resource_id: number; version_id: number; jobId?: number }): Promise { + try { + let totalProcessed = 0; + // Loop until no more unembedded chunks remain for this version. + // Each iteration pulls one batch worth; the loop terminates when + // the SELECT returns fewer rows than the batch size. + while (true) { + const batch = this.store.fetchPendingChunks(input.version_id, this.batchSize); + if (batch.length === 0) break; + const texts = batch.map((row) => row.content); + let embeddings: number[][]; + let model = "unknown"; + try { + embeddings = await workspaceEmbedDocuments(texts); + model = await workspaceEmbeddingModelName(); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + this.store.markJobFailed(input.jobId ?? null, message); + // Partial failure: leave whatever was already written in place; + // the resource is marked `partial` so callers can retry. + this.store.markVersionEmbeddingPartial(input.resource_id, input.version_id, message); + return; + } + const dimensions = embeddings[0]?.length ?? this.dimensions; + this.store.ensureChildVectorTable(dimensions); + this.store.writeChunkEmbeddings( + batch.map((row, i) => ({ chunkId: row.chunk_id, embedding: embeddings[i] ?? [] })), + model, + dimensions, + ); + totalProcessed += batch.length; + if (batch.length < this.batchSize) break; + } + this.store.markVersionEmbeddingReady(input.resource_id, input.version_id); + if (input.jobId !== undefined) { + this.store.completeJob(input.jobId, totalProcessed); + } + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + this.store.markJobFailed(input.jobId ?? null, message); + this.store.markVersionEmbeddingFailed(input.resource_id, input.version_id, message); + } + } +} + +export function createEmbeddingQueue(deps: EmbeddingQueueDeps): EmbeddingQueue { + return new DefaultEmbeddingQueue(deps); +} diff --git a/packages/workspace/src/index.ts b/packages/workspace/src/index.ts new file mode 100644 index 00000000..eb8c3714 --- /dev/null +++ b/packages/workspace/src/index.ts @@ -0,0 +1,67 @@ +/** + * `@melandlabs/workspace` — barrel re-export. + * + * Mirrors the OKF subpath shape: + * - root: low-level API surface (api, types, schema, sqlite) + * - `/sqlite` `getSQLiteWorkspaceStore()` / `closeSQLiteWorkspaceStore()` + * - `/cli` `opencontext workspace …` subcommand + */ + +export { + getSQLiteWorkspaceStore, + closeSQLiteWorkspaceStore, + resolveWorkspaceDbPath, + createSqliteWorkspaceStore, + __resetSQLiteWorkspaceStoreForTests, +} from "./sqlite"; +export type { SqliteWorkspaceStore, SqliteWorkspaceStoreOptions } from "./sqlite"; + +export { updateWorkspaceContext, searchWorkspaceContext, listWorkspaceResources } from "./api"; +export type { SearchWorkspaceContextDeps } from "./api"; + +export { indexOkfFolder, listOkfFolderResources } from "./okf-backend"; + +export { searchLexical } from "./search/lexical"; +export { searchSemantic } from "./search/semantic"; +export { fuseHybridHits } from "./search/hybrid"; +export { searchCrossFile } from "./search/cross-file"; + +export { createEmbeddingQueue } from "./embedding-queue"; +export type { EmbeddingQueue, EmbeddingQueueDeps } from "./embedding-queue"; + +export { + getWorkspaceEmbeddingProvider, + workspaceEmbedQuery, + workspaceEmbedDocuments, + workspaceEmbeddingModelName, + workspaceEmbeddingDimensions, + __resetWorkspaceEmbeddingProviderForTests, +} from "./embedding-provider"; +export type { + WorkspaceEmbeddingProvider, + WorkspaceEmbeddingProviderType, +} from "./embedding-provider"; + +export { initializeWorkspaceSchema, WORKSPACE_SCHEMA_VERSION } from "./schema"; + +export type { + RuntimeContext, + WorkspaceStorageKind, + WorkspaceEdgeType, + WorkspaceIndexStatus, + WorkspaceSearchStrategy, + WorkspaceResource, + WorkspaceResourceVersion, + WorkspaceChunk, + WorkspaceReferenceEdge, + WorkspaceJob, + WorkspaceSearchHit, + SearchWorkspaceContextOptions, + UpdateWorkspaceContextInput, + UpdateWorkspaceContextResult, + SearchWorkspaceContextInput, + SearchWorkspaceContextResult, + ListWorkspaceResourcesInput, + ListWorkspaceResourcesResult, + OkfFolderResource, +} from "./types"; diff --git a/packages/workspace/src/okf-backend.ts b/packages/workspace/src/okf-backend.ts new file mode 100644 index 00000000..ee77f5a8 --- /dev/null +++ b/packages/workspace/src/okf-backend.ts @@ -0,0 +1,211 @@ +/** + * `@melandlabs/workspace` — OKF folder backend. + * + * Wires the multi-format text extractor to `SqliteWorkspaceStore.indexResource` + * and the OKF graph builder to `SqliteWorkspaceStore.upsertReferenceEdges`. + * + * Only `cites` edges are written (the markdown-link resolver in + * `buildGraphFromDir`). `supersedes` / `amends` / `relates-to` are + * reserved in the schema enum but never produced here. + */ + +import { stat } from "node:fs/promises"; +import { join, relative, sep } from "node:path"; +import { extname } from "node:path"; +import { readdir } from "node:fs/promises"; +import { buildGraphFromDir, type WikiGraph, type WikiNode } from "@melandlabs/okf"; +import type { OkfFrontMatter } from "@melandlabs/contracts"; +import type { OkfFolderResource, WorkspaceEdgeType, UpdateWorkspaceContextResult } from "./types"; +import { extractText } from "./parsers-adapter"; +import type { SqliteWorkspaceStore } from "./sqlite"; + +const SUPPORTED_EXTENSIONS = new Set([".md", ".markdown", ".txt", ".pdf", ".docx", ".pages"]); + +async function walk(dir: string): Promise { + const out: string[] = []; + const stack = [dir]; + while (stack.length > 0) { + const head = stack.pop(); + if (!head) break; + let entries: import("node:fs").Dirent[]; + try { + entries = (await readdir(head, { withFileTypes: true })) as unknown as import("node:fs").Dirent[]; + } catch { + continue; + } + for (const entry of entries) { + const full = join(head, entry.name); + if (entry.isDirectory()) { + stack.push(full); + } else if (entry.isFile() && SUPPORTED_EXTENSIONS.has(extname(entry.name).toLowerCase())) { + out.push(full); + } + } + } + return out; +} + +function pickFrontMatterType(fm: OkfFrontMatter | undefined): string { + if (!fm) return "document"; + const type = typeof fm.type === "string" ? fm.type : "document"; + return type; +} + +function resourceTypeForExtension(ext: string): string { + switch (ext) { + case ".md": + case ".markdown": + return "note"; + case ".txt": + return "note"; + case ".pdf": + return "document"; + case ".docx": + return "document"; + case ".pages": + return "document"; + default: + return "document"; + } +} + +/** + * List every supported file under `dir`, parse it, and return the + * `OkfFolderResource[]` shape that `indexResource` consumes. Errors + * per-file are swallowed (logged via stderr) so one broken file doesn't + * abort the whole scan. + */ +export async function listOkfFolderResources(dir: string): Promise { + const files = await walk(dir); + const results: OkfFolderResource[] = []; + for (const absolute of files) { + try { + const extracted = await extractText(absolute); + const ext = extname(absolute).toLowerCase(); + const canonical = relative(dir, absolute).split(sep).join("/"); + const statResult = await stat(absolute); + const resourceType = ext === ".md" || ext === ".markdown" ? "note" : resourceTypeForExtension(ext); + results.push({ + canonical_key: canonical, + absolute_path: absolute, + title: canonical.replace(/\.[^.]+$/, ""), + resource_type: resourceType, + body: extracted.text, + size_bytes: statResult.size, + }); + } catch (error) { + // biome-ignore lint/suspicious/noConsole: server-side warning surfaced to ops + console.warn(`[workspace/okf] failed to read ${absolute}:`, error); + } + } + return results; +} + +/** + * Walk an OKF folder, index every file, then build the `cites` edge + * graph from the markdown-link resolver and persist it. + * + * The job id is created up-front so the embedding queue can attach + * completion / failure to it. + */ +export async function indexOkfFolder( + store: SqliteWorkspaceStore, + input: { workspace_id: string; user_id: string; path: string; enqueueEmbedding: (input: { resource_id: number; version_id: number; jobId?: number }) => Promise }, +): Promise { + const resources = await listOkfFolderResources(input.path); + const job = store.createJob({ workspace_id: input.workspace_id, kind: "index", total: resources.length }); + store.updateJobTotal(job.id, resources.length); + + const presentKeys = new Set(); + let filesAdded = 0; + let filesModified = 0; + let filesUnchanged = 0; + + const indexedIds: Array<{ resource_id: number; version_id: number; canonical_key: string }> = []; + + for (const resource of resources) { + presentKeys.add(resource.canonical_key); + const result = await store.indexResource({ + workspace_id: input.workspace_id, + user_id: input.user_id, + resource, + }); + indexedIds.push({ + resource_id: result.resource_id, + version_id: result.version_id, + canonical_key: resource.canonical_key, + }); + if (result.change_kind === "created") filesAdded += 1; + else if (result.change_kind === "modified") filesModified += 1; + else filesUnchanged += 1; + await input.enqueueEmbedding({ + resource_id: result.resource_id, + version_id: result.version_id, + jobId: job.id, + }); + } + + // Edge pass: rebuild cites edges from the on-disk graph so a rename / + // delete in the OKF folder immediately reflects in the index. + let graph: WikiGraph; + try { + graph = await buildGraphFromDir(input.path); + } catch (error) { + // biome-ignore lint/suspicious/noConsole: server-side warning surfaced to ops + console.warn(`[workspace/okf] buildGraphFromDir failed for ${input.path}:`, error); + graph = { nodes: [], edges: [], types: [], generatedAt: new Date().toISOString(), root: input.path }; + } + const idByCanonical = new Map(); + const canonicalByResourceId = new Map(); + for (const indexed of indexedIds) { + idByCanonical.set(indexed.canonical_key, indexed.resource_id); + canonicalByResourceId.set(indexed.resource_id, indexed.canonical_key); + } + const wikiEdges: Array<{ + source_resource_id: number; + target_resource_id: number; + edge_type: WorkspaceEdgeType; + }> = []; + for (const edge of graph.edges) { + const sourceCanonical = `${edge.source}.md`; + const targetCanonical = `${edge.target}.md`; + const sourceId = idByCanonical.get(sourceCanonical); + const targetId = idByCanonical.get(targetCanonical); + if (sourceId === undefined || targetId === undefined) continue; + wikiEdges.push({ source_resource_id: sourceId, target_resource_id: targetId, edge_type: "cites" }); + } + if (wikiEdges.length > 0) { + store.upsertReferenceEdges({ + workspace_id: input.workspace_id, + edges: wikiEdges.map((edge) => ({ ...edge, source_version_id: null, target_version_id: null })), + }); + } + + // Soft-delete detection: any canonical key not in `presentKeys` and + // not already marked `deleted_at` is flipped to soft-deleted. + const deleted = store.softDeleteMissingResources({ + workspace_id: input.workspace_id, + presentKeys, + }); + + return { + jobId: job.id, + triggered: true, + status: "pending", + filesScanned: resources.length, + filesAdded, + filesModified, + filesUnchanged, + filesDeleted: deleted.length, + }; +} + +/** + * Cheap helper used by callers that only need the wiki node titles + * (e.g. the indexer's title enrichment). Mirrors `buildGraphFromDir`'s + * node shape but stops short of the full edge / backlink pass. + */ +export async function readOkfFolderTitles(dir: string): Promise { + const graph = await buildGraphFromDir(dir); + return graph.nodes; +} diff --git a/packages/workspace/src/parsers-adapter.ts b/packages/workspace/src/parsers-adapter.ts new file mode 100644 index 00000000..33217a0c --- /dev/null +++ b/packages/workspace/src/parsers-adapter.ts @@ -0,0 +1,109 @@ +/** + * `@melandlabs/workspace` — multi-format text extractor. + * + * Supported inputs: + * - `.md` / `.markdown` — pass-through (front-matter is parsed by + * `okf-backend` separately) + * - `.txt` — pass-through + * - `.pdf` — `parseFileToDocument` (`@melandlabs/rag`) + * - `.docx` — `parseFileToDocument` + * - `.pages` — `parseFileToDocument` (via `AppleDocumentLoader`) + * + * Binary raster files (`.png`, `.jpg`, …) intentionally raise an explicit + * "unsupported" error so the caller can fall back to manual transcription. + */ + +import { readFile } from "node:fs/promises"; +import { extname } from "node:path"; +import { estimateTokens } from "@melandlabs/shared"; +import { parseFile, parseFileToDocument } from "@melandlabs/rag"; + +let _parsersConfigured = false; + +function ensureParsersConfigured(): void { + if (_parsersConfigured) return; + // `parseFile` only uses `estimateTokens` for chunk-count estimation, + // which we don't currently invoke; provide a stub anyway so future + // callers that hit `estimateChunkCount` don't crash. + try { + const { configureParsers } = require("@melandlabs/rag") as { + configureParsers?: (config: { estimateTokens: (text: string) => number }) => void; + }; + configureParsers?.({ estimateTokens }); + } catch { + // No-op: some builds don't expose `configureParsers` directly. + } + _parsersConfigured = true; +} + +export interface ExtractedText { + text: string; + mimeType: string; + metadata?: Record; +} + +const MIME_BY_EXTENSION: Record = { + ".md": "text/markdown", + ".markdown": "text/markdown", + ".txt": "text/plain", + ".pdf": "application/pdf", + ".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + ".doc": "application/msword", + ".pages": "application/x-iwork-pages-sffpages", + ".numbers": "application/x-iwork-numbers-sffnumbers", + ".keynote": "application/x-iwork-keynote-sffkeynote", + ".html": "text/html", + ".htm": "text/html", +}; + +/** + * Best-effort MIME guess from the file extension. Falls back to + * `application/octet-stream` so `parseFile` throws a clear "unsupported + * content type" error rather than silently producing garbage. + */ +export function detectMimeType(sourcePath: string): string { + const ext = extname(sourcePath).toLowerCase(); + return MIME_BY_EXTENSION[ext] ?? "application/octet-stream"; +} + +/** + * Read the source file, dispatch to the appropriate loader, and return + * the canonical text body used by both the chunker and the embedder. + */ +export async function extractText(sourcePath: string, mimeType?: string): Promise { + ensureParsersConfigured(); + const ext = extname(sourcePath).toLowerCase(); + const mime = mimeType ?? detectMimeType(sourcePath); + if (ext === ".md" || ext === ".markdown" || ext === ".txt") { + const text = await readFile(sourcePath, "utf8"); + return { text, mimeType: mime }; + } + const buffer = await readFile(sourcePath); + // The RAG layer exposes `parseFile` (returns `{text, metadata}`) and + // `parseFileToDocument` (returns a LangChain `Document`). The latter + // also surfaces page-level metadata; prefer it when callers can + // surface metadata downstream. + const document = await parseFileToDocument(buffer, mime, sourcePath); + return { + text: document.pageContent, + mimeType: mime, + metadata: document.metadata as Record, + }; +} + +/** + * Convenience re-export of `parseFile` so the OKF backend can opt into + * the lighter (no-LangChain-Document) path when it only needs text. + */ +export async function extractTextRaw(sourcePath: string, mimeType?: string): Promise { + ensureParsersConfigured(); + const ext = extname(sourcePath).toLowerCase(); + const mime = mimeType ?? detectMimeType(sourcePath); + if (ext === ".md" || ext === ".markdown" || ext === ".txt") { + const text = await readFile(sourcePath, "utf8"); + return { text, mimeType: mime }; + } + const buffer = await readFile(sourcePath); + const { text, metadata } = await parseFile(buffer, mime); + return { text, mimeType: mime, metadata }; +} diff --git a/packages/workspace/src/schema.ts b/packages/workspace/src/schema.ts new file mode 100644 index 00000000..6eb6bb41 --- /dev/null +++ b/packages/workspace/src/schema.ts @@ -0,0 +1,178 @@ +/** + * `@melandlabs/workspace` — schema bootstrap. + * + * Workspace tables live in the shared SQLite DB that the memory-store + * already owns (`~/.opencontext/memory/store.db`), in their own + * `workspace_*` namespace so they don't collide with `raw_messages` / + * `facts` / `entities`. + * + * The `addColumnIfMissing` helper from `@melandlabs/sqlite/src/schema` + * is not exported; we replicate the lightweight version here because + * the workspace tables are created from scratch. When a v2 migration + * arrives, the same `pragma_table_info`-based detection pattern from + * the memory schema should be applied — keeping this file's behaviour + * consistent with `packages/sqlite/src/schema.ts:11-17`. + */ + +import type Database from "better-sqlite3"; + +/** + * Current schema version. Bump when adding a new table or column; + * the boot path reads this constant to decide whether to run a + * migration. + */ +export const WORKSPACE_SCHEMA_VERSION = 1; + +/** + * Idempotent column-add helper for SQLite (which lacks `ADD COLUMN IF NOT + * EXISTS`). Mirrors `packages/sqlite/src/schema.ts:11-17`. + */ +function addColumnIfMissing(db: Database.Database, table: string, column: string, definition: string): void { + const rows = db.prepare(`PRAGMA table_info(${table})`).all() as Array<{ name: string }>; + if (rows.some((row) => row.name === column)) return; + db.exec(`ALTER TABLE ${table} ADD COLUMN ${column} ${definition};`); +} + +/** + * Run every idempotent DDL statement that backs the workspace + * feature. Safe to call repeatedly. + * + * - `workspace_resources` — one row per (workspace, canonical_key) + * - `workspace_resource_versions` — version chain per resource (sha256, parent) + * - `workspace_chunks` — search-only child chunks per version + * - `workspace_chunks_fts` — FTS5 mirror of `workspace_chunks.content` + * - `workspace_reference_edges` — cites / supersedes / amends / relates-to + * - `workspace_jobs` — async indexing job status + * + * The vec0 child table `workspace_chunks_vec_d1536` is created lazily by + * `SqliteWorkspaceStore.ensureChildVectorTable(dimensions)` so an empty + * workspace (no embeddings yet) never wastes disk on an empty vector table. + */ +export function initializeWorkspaceSchema(db: Database.Database): void { + db.pragma("journal_mode = WAL"); + db.pragma("busy_timeout = 30000"); + db.pragma("foreign_keys = ON"); + + db.exec(` + CREATE TABLE IF NOT EXISTS workspace_resources ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + workspace_id TEXT NOT NULL, + user_id TEXT NOT NULL, + resource_type TEXT NOT NULL, + canonical_key TEXT NOT NULL, + title TEXT NOT NULL, + storage_kind TEXT NOT NULL, + current_version_id INTEGER, + index_status TEXT NOT NULL DEFAULT 'pending', + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + metadata TEXT, + UNIQUE(workspace_id, canonical_key) + ); + + CREATE INDEX IF NOT EXISTS idx_workspace_resources_workspace_type + ON workspace_resources(workspace_id, resource_type); + CREATE INDEX IF NOT EXISTS idx_workspace_resources_workspace_status + ON workspace_resources(workspace_id, index_status); + CREATE INDEX IF NOT EXISTS idx_workspace_resources_user + ON workspace_resources(user_id); + + CREATE TABLE IF NOT EXISTS workspace_resource_versions ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + resource_id INTEGER NOT NULL, + version_number INTEGER NOT NULL, + sha256 TEXT NOT NULL, + change_kind TEXT NOT NULL, + size_bytes INTEGER NOT NULL, + parent_version_id INTEGER, + source_path TEXT, + created_at INTEGER NOT NULL, + metadata TEXT, + UNIQUE(resource_id, version_number) + ); + + CREATE INDEX IF NOT EXISTS idx_workspace_resource_versions_resource + ON workspace_resource_versions(resource_id, version_number DESC); + + CREATE TABLE IF NOT EXISTS workspace_chunks ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + chunk_id TEXT UNIQUE NOT NULL, + resource_id INTEGER NOT NULL, + version_id INTEGER NOT NULL, + workspace_id TEXT NOT NULL, + chunk_index INTEGER NOT NULL, + chunk_count INTEGER NOT NULL, + start_position INTEGER NOT NULL, + end_position INTEGER NOT NULL, + content TEXT NOT NULL, + content_hash TEXT NOT NULL, + embedding BLOB, + embedding_model TEXT, + embedding_dimensions INTEGER, + embedding_updated_at INTEGER, + UNIQUE(resource_id, version_id, chunk_index) + ); + + CREATE INDEX IF NOT EXISTS idx_workspace_chunks_workspace + ON workspace_chunks(workspace_id); + CREATE INDEX IF NOT EXISTS idx_workspace_chunks_resource + ON workspace_chunks(resource_id, version_id); + + CREATE VIRTUAL TABLE IF NOT EXISTS workspace_chunks_fts USING fts5( + content, + content='workspace_chunks', + content_rowid='id' + ); + + CREATE TRIGGER IF NOT EXISTS workspace_chunks_ai AFTER INSERT ON workspace_chunks BEGIN + INSERT INTO workspace_chunks_fts(rowid, content) VALUES (new.id, new.content); + END; + CREATE TRIGGER IF NOT EXISTS workspace_chunks_ad AFTER DELETE ON workspace_chunks BEGIN + INSERT INTO workspace_chunks_fts(workspace_chunks_fts, rowid, content) + VALUES('delete', old.id, old.content); + END; + CREATE TRIGGER IF NOT EXISTS workspace_chunks_au AFTER UPDATE ON workspace_chunks BEGIN + INSERT INTO workspace_chunks_fts(workspace_chunks_fts, rowid, content) + VALUES('delete', old.id, old.content); + INSERT INTO workspace_chunks_fts(rowid, content) VALUES (new.id, new.content); + END; + + CREATE TABLE IF NOT EXISTS workspace_reference_edges ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + workspace_id TEXT NOT NULL, + source_resource_id INTEGER NOT NULL, + source_version_id INTEGER, + target_resource_id INTEGER NOT NULL, + target_version_id INTEGER, + edge_type TEXT NOT NULL, + quote TEXT, + created_at INTEGER NOT NULL, + UNIQUE(workspace_id, source_resource_id, target_resource_id, edge_type) + ); + + CREATE INDEX IF NOT EXISTS idx_workspace_reference_edges_workspace + ON workspace_reference_edges(workspace_id); + CREATE INDEX IF NOT EXISTS idx_workspace_reference_edges_source + ON workspace_reference_edges(source_resource_id); + + CREATE TABLE IF NOT EXISTS workspace_jobs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + workspace_id TEXT NOT NULL, + kind TEXT NOT NULL, + status TEXT NOT NULL, + total INTEGER NOT NULL DEFAULT 0, + done INTEGER NOT NULL DEFAULT 0, + error TEXT, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_workspace_jobs_workspace + ON workspace_jobs(workspace_id, created_at DESC); + `); + + // Future-proof: keep `addColumnIfMissing` available for v2 migrations + // without TypeScript flagging it as unused. Same pattern as the + // memory-store schema's v2–v5 column upgrades. + void addColumnIfMissing; +} diff --git a/packages/workspace/src/search/cross-file.ts b/packages/workspace/src/search/cross-file.ts new file mode 100644 index 00000000..76f8bfc1 --- /dev/null +++ b/packages/workspace/src/search/cross-file.ts @@ -0,0 +1,73 @@ +/** + * `@melandlabs/workspace/search/cross-file` — hybrid + BFS over cites edges. + * + * 1. Run hybrid search to collect the top-`limit * 4` candidates. + * 2. BFS 1–2 hops over `workspace_reference_edges` to surface neighbour + * resources (typically the law clause a contract cites, or the + * reference a review result depends on). + * 3. Re-rank with `edge_boost = 0.1 × (out_degree + in_degree)` so + * heavily linked resources surface higher. + */ + +import type { WorkspaceSearchHit } from "../types"; +import type { SqliteWorkspaceStore } from "../sqlite"; +import { fuseHybridHits } from "./hybrid"; + +export interface CrossFileSearchInput { + workspace_id: string; + user_id: string; + query: string; + resource_types?: string[]; + limit: number; + threshold?: number; + hops?: 1 | 2; + lexicalSearch: (input: { + workspace_id: string; + user_id: string; + query: string; + resource_types?: string[]; + limit: number; + }) => WorkspaceSearchHit[]; + semanticSearch: (input: { + workspace_id: string; + user_id: string; + queryEmbedding: number[]; + resource_types?: string[]; + limit: number; + threshold: number; + }) => WorkspaceSearchHit[]; + generateEmbedding?: (text: string) => Promise; +} + +export async function searchCrossFile( + store: SqliteWorkspaceStore, + input: CrossFileSearchInput, +): Promise { + const candidateLimit = input.limit * 4; + const lexicalHits = input.lexicalSearch({ + workspace_id: input.workspace_id, + user_id: input.user_id, + query: input.query, + resource_types: input.resource_types, + limit: candidateLimit, + }); + let semanticHits: WorkspaceSearchHit[] = []; + if (input.generateEmbedding) { + const embedding = await input.generateEmbedding(input.query); + semanticHits = input.semanticSearch({ + workspace_id: input.workspace_id, + user_id: input.user_id, + queryEmbedding: embedding, + resource_types: input.resource_types, + limit: candidateLimit, + threshold: input.threshold ?? 0.7, + }); + } + const fused = fuseHybridHits({ lexical: lexicalHits, semantic: semanticHits, limit: candidateLimit }); + return store.expandNeighbors({ + workspace_id: input.workspace_id, + hits: fused, + hops: input.hops ?? 1, + limit: input.limit, + }); +} diff --git a/packages/workspace/src/search/hybrid.ts b/packages/workspace/src/search/hybrid.ts new file mode 100644 index 00000000..4aae3ad2 --- /dev/null +++ b/packages/workspace/src/search/hybrid.ts @@ -0,0 +1,87 @@ +/** + * `@melandlabs/workspace/search/hybrid` — lexical × semantic fusion. + * + * Wraps `fuseHybridResults` (`@melandlabs/rag`) with the RRF k=60 default + * the plan specifies. The store-side rank list is normalised to the + * `VectorSearchResult` shape that the RAG fusion helper expects. + */ + +import { fuseHybridResults, type VectorSearchResult } from "@melandlabs/rag"; +import type { WorkspaceSearchHit } from "../types"; + +function toVectorResult(hit: WorkspaceSearchHit): VectorSearchResult { + return { + id: hit.chunk_id, + documentId: hit.canonical_key, + content: hit.snippet, + score: hit.score, + metadata: { + resource_id: hit.resource_id, + version_id: hit.version_id, + resource_type: hit.resource_type, + resource_title: hit.resource_title, + }, + }; +} + +function fromVectorResult(result: VectorSearchResult, original: WorkspaceSearchHit): WorkspaceSearchHit { + const meta = result.metadata ?? {}; + return { + ...original, + score: result.score, + signals: { + ...original.signals, + lexical: original.signals.lexical, + semantic: original.signals.semantic, + edge_boost: original.signals.edge_boost, + }, + chunk_id: result.id, + canonical_key: (meta.documentId as string) ?? original.canonical_key, + }; +} + +export interface HybridSearchInput { + lexical: WorkspaceSearchHit[]; + semantic: WorkspaceSearchHit[]; + limit: number; + rrfK?: number; + alpha?: number; +} + +export function fuseHybridHits(input: HybridSearchInput): WorkspaceSearchHit[] { + const dense = input.semantic.map(toVectorResult); + const lexical = input.lexical.map(toVectorResult); + const fused = fuseHybridResults({ + dense, + lexical, + strategy: "rrf", + rrfK: input.rrfK ?? 60, + alpha: input.alpha ?? 0.5, + limit: input.limit, + }); + // `fuseHybridResults` mutates `result.score`; look up the original + // WorkspaceSearchHit by chunk_id so the return shape carries the + // signals / edges / metadata that the lexical / semantic passes built. + const byChunkId = new Map(); + for (const hit of input.lexical) byChunkId.set(hit.chunk_id, hit); + for (const hit of input.semantic) byChunkId.set(hit.chunk_id, hit); + return fused.map((result) => { + const original = byChunkId.get(result.id); + if (!original) { + return { + chunk_id: result.id, + resource_id: 0, + version_id: 0, + resource_type: "document", + resource_title: "", + canonical_key: result.documentId, + snippet: result.content, + matched_terms: [], + score: result.score, + signals: {}, + reference_edges: [], + }; + } + return fromVectorResult(result, original); + }); +} diff --git a/packages/workspace/src/search/lexical.ts b/packages/workspace/src/search/lexical.ts new file mode 100644 index 00000000..c44d1bc8 --- /dev/null +++ b/packages/workspace/src/search/lexical.ts @@ -0,0 +1,24 @@ +/** + * `@melandlabs/workspace/search/lexical` — FTS5 lexical search. + * + * Mirrors `packages/sqlite/src/raw-message-manager.ts:332-338` for the + * FTS5 query construction (`"" OR ""`) and `bm25()` ranking. + * Per the plan, the lexical pass is the fallback when embeddings haven't + * been written yet, so it must remain functional before the embedding + * queue drains. + */ + +import type { WorkspaceSearchHit } from "../types"; +import type { SqliteWorkspaceStore } from "../sqlite"; + +export interface LexicalSearchInput { + workspace_id: string; + user_id: string; + query: string; + resource_types?: string[]; + limit: number; +} + +export function searchLexical(store: SqliteWorkspaceStore, input: LexicalSearchInput): WorkspaceSearchHit[] { + return store.searchLexical(input); +} diff --git a/packages/workspace/src/search/semantic.ts b/packages/workspace/src/search/semantic.ts new file mode 100644 index 00000000..002aa63a --- /dev/null +++ b/packages/workspace/src/search/semantic.ts @@ -0,0 +1,23 @@ +/** + * `@melandlabs/workspace/search/semantic` — sqlite-vec KNN search. + * + * Wraps `SqliteWorkspaceStore.searchSemantic` so the API layer can stay + * store-agnostic. The store handles the widen-and-retry loop and the + * "no embeddings yet → empty result" fallback. + */ + +import type { WorkspaceSearchHit } from "../types"; +import type { SqliteWorkspaceStore } from "../sqlite"; + +export interface SemanticSearchInput { + workspace_id: string; + user_id: string; + queryEmbedding: number[]; + resource_types?: string[]; + limit: number; + threshold: number; +} + +export function searchSemantic(store: SqliteWorkspaceStore, input: SemanticSearchInput): WorkspaceSearchHit[] { + return store.searchSemantic(input); +} diff --git a/packages/workspace/src/sqlite.ts b/packages/workspace/src/sqlite.ts new file mode 100644 index 00000000..30aed139 --- /dev/null +++ b/packages/workspace/src/sqlite.ts @@ -0,0 +1,1340 @@ +/** + * `@melandlabs/workspace/sqlite` — the SQLite-backed store. + * + * Public surface: + * + * - `SqliteWorkspaceStore` — CRUD + indexing pipeline + * - `getSQLiteWorkspaceStore()` — singleton accessor (lazy open) + * - `closeSQLiteWorkspaceStore()`— test-only teardown + * - `resolveWorkspaceDbPath(dbPath?)` — same env-var fallback as + * `@melandlabs/memory-store`'s `resolveSQLiteRawMessageDbPath` + * (`MEMORY_STORE_DB_PATH` → `~/.opencontext/memory/store.db`) + * + * Design notes (mirrors `SQLiteRawMessageManager` patterns): + * + * - One SQLite file is shared with the rest of the memory store so + * the user's local data lives under a single `~/.opencontext/memory/store.db`. + * The vec0 child table is created lazily by + * `ensureChildVectorTable(dimensions)` — same as + * `packages/sqlite/src/raw-message-manager.ts:1859-1874`. + * - `indexResource` runs the **sync** portion of the indexing pipeline + * (sha256, version chain, chunk insert, FTS5 mirror) inside one + * `better-sqlite3` transaction. The async embedding fan-out is + * delegated to `EmbeddingQueue`, which writes the embedding + + * vec0 row + `index_status` flip in subsequent transactions. + */ + +import { mkdirSync } from "node:fs"; +import { dirname } from "node:path"; +import { createHash } from "node:crypto"; +import { chunkTextByEstimatedTokens, RAW_MESSAGE_CHUNK_MAX_TOKENS, RAW_MESSAGE_CHUNK_OVERLAP_TOKENS } from "@melandlabs/shared"; +import { getOpenContextPath } from "@melandlabs/env-config"; +import Database from "better-sqlite3"; +import * as sqliteVec from "sqlite-vec"; +import { bufferToFloatArray, floatArrayToBuffer } from "@melandlabs/sqlite"; +import type { + ListWorkspaceResourcesInput, + ListWorkspaceResourcesResult, + OkfFolderResource, + WorkspaceChunk, + WorkspaceEdgeType, + WorkspaceIndexStatus, + WorkspaceJob, + WorkspaceReferenceEdge, + WorkspaceResource, + WorkspaceResourceVersion, + WorkspaceSearchHit, + WorkspaceSearchStrategy, + SearchWorkspaceContextInput, + SearchWorkspaceContextResult, + UpdateWorkspaceContextInput, + UpdateWorkspaceContextResult, +} from "./types"; +import { initializeWorkspaceSchema } from "./schema"; + +type DatabaseLike = Database.Database; + +interface WorkspaceResourceRow { + id: number; + workspace_id: string; + user_id: string; + resource_type: string; + canonical_key: string; + title: string; + storage_kind: string; + current_version_id: number | null; + index_status: string; + created_at: number; + updated_at: number; + metadata: string | null; +} + +interface WorkspaceResourceVersionRow { + id: number; + resource_id: number; + version_number: number; + sha256: string; + change_kind: string; + size_bytes: number; + parent_version_id: number | null; + source_path: string | null; + created_at: number; + metadata: string | null; +} + +interface WorkspaceChunkRow { + id: number; + chunk_id: string; + resource_id: number; + version_id: number; + workspace_id: string; + chunk_index: number; + chunk_count: number; + start_position: number; + end_position: number; + content: string; + content_hash: string; + embedding: Buffer | null; + embedding_model: string | null; + embedding_dimensions: number | null; + embedding_updated_at: number | null; +} + +interface WorkspaceReferenceEdgeRow { + id: number; + workspace_id: string; + source_resource_id: number; + source_version_id: number | null; + target_resource_id: number; + target_version_id: number | null; + edge_type: string; + quote: string | null; + created_at: number; +} + +interface WorkspaceJobRow { + id: number; + workspace_id: string; + kind: string; + status: string; + total: number; + done: number; + error: string | null; + created_at: number; + updated_at: number; +} + +function toWorkspaceResource(row: WorkspaceResourceRow): WorkspaceResource { + return { + id: row.id, + workspace_id: row.workspace_id, + user_id: row.user_id, + resource_type: row.resource_type, + canonical_key: row.canonical_key, + title: row.title, + storage_kind: row.storage_kind as WorkspaceResource["storage_kind"], + current_version_id: row.current_version_id, + index_status: row.index_status as WorkspaceIndexStatus, + created_at: row.created_at, + updated_at: row.updated_at, + metadata: parseJson>(row.metadata, {} as Record), + }; +} + +function toWorkspaceResourceVersion(row: WorkspaceResourceVersionRow): WorkspaceResourceVersion { + return { + id: row.id, + resource_id: row.resource_id, + version_number: row.version_number, + sha256: row.sha256, + change_kind: row.change_kind as WorkspaceResourceVersion["change_kind"], + size_bytes: row.size_bytes, + parent_version_id: row.parent_version_id, + source_path: row.source_path, + created_at: row.created_at, + metadata: parseJson>(row.metadata, {} as Record), + }; +} + +function toWorkspaceChunk(row: WorkspaceChunkRow): WorkspaceChunk { + return { + id: row.id, + chunk_id: row.chunk_id, + resource_id: row.resource_id, + version_id: row.version_id, + workspace_id: row.workspace_id, + chunk_index: row.chunk_index, + chunk_count: row.chunk_count, + start_position: row.start_position, + end_position: row.end_position, + content: row.content, + content_hash: row.content_hash, + embedding: bufferToFloatArray(row.embedding), + embedding_model: row.embedding_model ?? undefined, + embedding_dimensions: row.embedding_dimensions ?? undefined, + embedding_updated_at: row.embedding_updated_at ?? undefined, + }; +} + +function toWorkspaceReferenceEdge(row: WorkspaceReferenceEdgeRow): WorkspaceReferenceEdge { + return { + id: row.id, + workspace_id: row.workspace_id, + source_resource_id: row.source_resource_id, + source_version_id: row.source_version_id, + target_resource_id: row.target_resource_id, + target_version_id: row.target_version_id, + edge_type: row.edge_type as WorkspaceEdgeType, + quote: row.quote, + created_at: row.created_at, + }; +} + +function toWorkspaceJob(row: WorkspaceJobRow): WorkspaceJob { + return { + id: row.id, + workspace_id: row.workspace_id, + kind: row.kind as WorkspaceJob["kind"], + status: row.status as WorkspaceIndexStatus, + total: row.total, + done: row.done, + error: row.error, + created_at: row.created_at, + updated_at: row.updated_at, + }; +} + +function currentUnixSeconds(): number { + return Math.floor(Date.now() / 1000); +} + +function sha256(value: string): string { + return createHash("sha256").update(value).digest("hex"); +} + +/** + * Local copies of `parseJson` / `stringifyJson` from + * `packages/sqlite/src/raw-message-manager.ts:180-196`. Those helpers + * are not exported from `@melandlabs/sqlite`, so the workspace + * package keeps its own copy in scope. Behaviour matches the memory + * store: missing / invalid JSON falls back to the caller-provided + * default rather than throwing. + */ +function parseJson(value: string | null | undefined, fallback: T): T { + if (!value) return fallback; + try { + return JSON.parse(value) as T; + } catch { + return fallback; + } +} + +function stringifyJson(value: unknown): string | null { + if (value === undefined) return null; + return JSON.stringify(value); +} + +/** + * Resolve the SQLite DB file path that backs the workspace store. + * Mirrors `packages/memory-store/src/storage/sqlite-raw-message-store.ts:27-31` + * so the workspace tables live in the same file as the rest of the + * memory store (no separate DB to coordinate backups / migrations). + */ +export function resolveWorkspaceDbPath(dbPath?: string): string { + if (dbPath && dbPath.length > 0) return dbPath; + const fromEnv = process.env.MEMORY_STORE_DB_PATH?.trim(); + return fromEnv && fromEnv.length > 0 ? fromEnv : getOpenContextPath("memory", "store.db"); +} + +export interface SqliteWorkspaceStoreOptions { + dbPath?: string; + db?: DatabaseLike; + /** + * Lazily import a custom embedding function. Tests inject a deterministic + * mock here to avoid hitting OpenRouter; production leaves this unset + * and uses `@melandlabs/rag`'s `generateEmbeddings` directly. + */ + embeddingQueueFactory?: (store: SqliteWorkspaceStore) => { + enqueue(input: { resource_id: number; version_id: number; jobId?: number }): Promise; + drain(): Promise; + }; +} + +/** + * Lightweight stub interface that `EmbeddingQueue` consumes via duck typing + * (`store.fetchPendingChunks`, `store.writeChunkEmbeddings`, …). Keeping it + * here (rather than re-importing the queue file) avoids a circular type + * reference between `embedding-queue.ts` and `sqlite.ts`. + */ +export interface SqliteWorkspaceStore { + readonly __testDb: DatabaseLike; + init(): Promise; + close(): Promise; + + fetchPendingChunks(versionId: number, limit: number): Array; + writeChunkEmbeddings( + entries: Array<{ chunkId: string; embedding: number[] }>, + model: string, + dimensions: number, + ): void; + ensureChildVectorTable(dimensions: number): void; + markVersionEmbeddingReady(resourceId: number, versionId: number): void; + markVersionEmbeddingPartial(resourceId: number, versionId: number, errorMessage: string): void; + markVersionEmbeddingFailed(resourceId: number, versionId: number, errorMessage: string): void; + markJobFailed(jobId: number | null, errorMessage: string): void; + completeJob(jobId: number, done: number): void; + + indexResource(input: { + workspace_id: string; + user_id: string; + resource: OkfFolderResource; + }): Promise<{ resource_id: number; version_id: number; change_kind: WorkspaceResourceVersion["change_kind"] }>; + + upsertReferenceEdges(input: { + workspace_id: string; + edges: Array<{ + source_resource_id: number; + source_version_id: number | null; + target_resource_id: number; + target_version_id: number | null; + edge_type: WorkspaceEdgeType; + quote?: string | null; + }>; + }): void; + + softDeleteMissingResources(input: { + workspace_id: string; + presentKeys: Set; + }): Array<{ canonical_key: string }>; + + listResources(input: ListWorkspaceResourcesInput): ListWorkspaceResourcesResult; + + searchLexical(input: { + workspace_id: string; + user_id: string; + query: string; + resource_types?: string[]; + limit: number; + }): WorkspaceSearchHit[]; + + searchSemantic(input: { + workspace_id: string; + user_id: string; + queryEmbedding: number[]; + resource_types?: string[]; + limit: number; + threshold: number; + }): WorkspaceSearchHit[]; + + expandNeighbors(input: { + workspace_id: string; + hits: WorkspaceSearchHit[]; + hops: 1 | 2; + limit: number; + }): WorkspaceSearchHit[]; + + findResourceByCanonicalKey(input: { workspace_id: string; canonical_key: string }): WorkspaceResource | null; + + createJob(input: { workspace_id: string; kind: WorkspaceJob["kind"]; total: number }): WorkspaceJob; + updateJobTotal(jobId: number, total: number): void; +} + +/** + * `SqliteWorkspaceStore` — full implementation. The class body is split + * into clearly labelled sections (`init/close`, `index pipeline`, + * `search pipeline`, `cross-file expansion`) to keep the file readable + * when each section grows in later phases. + */ +export class SqliteWorkspaceStore implements SqliteWorkspaceStore { + readonly __testDb!: DatabaseLike; + private readonly db: DatabaseLike; + private readonly ownsConnection: boolean; + private readonly embeddingQueueFactory?: SqliteWorkspaceStoreOptions["embeddingQueueFactory"]; + private initialized = false; + private vectorSearchAvailable = false; + + constructor(options: SqliteWorkspaceStoreOptions | string = ":memory:") { + if (typeof options === "string") { + this.db = new Database(options); + this.ownsConnection = true; + this.embeddingQueueFactory = undefined; + } else if (options.db) { + this.db = options.db; + this.ownsConnection = false; + this.embeddingQueueFactory = options.embeddingQueueFactory; + } else { + this.db = new Database(options.dbPath ?? ":memory:"); + this.ownsConnection = true; + this.embeddingQueueFactory = options.embeddingQueueFactory; + } + // `__testDb` is intentionally a public readonly handle (mirrors + // `SQLiteVsaStore.__testDb`). Assigning it through `this.__testDb` + // in a `readonly` declaration trips the DTS build's strictness, + // so we set it via `Object.defineProperty` here. This is only + // touched in tests — production code goes through the typed + // surface above. + Object.defineProperty(this, "__testDb", { value: this.db, writable: false, enumerable: true }); + } + + async init(): Promise { + if (this.initialized) return; + initializeWorkspaceSchema(this.db); + // Load sqlite-vec into this connection so vec0 tables can be + // created/queried. Mirrors `SQLiteRawMessageManager.initializeVectorSearch` + // (`packages/sqlite/src/raw-message-manager.ts:1683-1695`). If the + // extension can't load (e.g. the binary wasn't built for the host + // platform) we leave `vectorSearchAvailable = false` so the + // embedding fan-out degrades to lexical-only instead of crashing. + try { + sqliteVec.load(this.db); + this.vectorSearchAvailable = true; + } catch { + this.vectorSearchAvailable = false; + } + this.initialized = true; + } + + async close(): Promise { + // Best-effort: drop the singleton reference and let the OS reap the + // native handle on process exit. `db.close()` is intentionally NOT + // called here because `sqlite-vec`'s native destructor occasionally + // raises SIGABRT during process teardown on macOS, which would + // surface to the user as a confusing abort right after we've + // already printed the search result. The trade-off is that the + // process exit takes a few extra ms (sqlite writes its WAL flush) + // but no abort noise. + this.initialized = false; + if (!this.ownsConnection) return; + // We intentionally do NOT call `this.db.close()` — see comment above. + // Tests that need a clean teardown should use `__resetSQLiteWorkspaceStoreForTests`. + } + + // ------------------------------------------------------------------------- + // Vector table helpers (mirrors raw-message-manager.ts:1859-1914) + // ------------------------------------------------------------------------- + + private getChildVectorTableName(dimensions: number): string { + if (!Number.isInteger(dimensions) || dimensions <= 0) { + throw new Error(`Invalid project chunk embedding dimensions: ${dimensions}`); + } + return `workspace_chunks_vec_d${dimensions}`; + } + + ensureChildVectorTable(dimensions: number): void { + if (!this.vectorSearchAvailable) { + // sqlite-vec didn't load — skip vec0 creation. The embedding + // queue's `writeChunkEmbeddings` will still persist the raw + // vector into `workspace_chunks.embedding` for later replay + // once the extension is available, and semantic search will + // fall back to lexical. + return; + } + const tableName = this.getChildVectorTableName(dimensions); + this.db.exec(` + CREATE VIRTUAL TABLE IF NOT EXISTS ${tableName} + USING vec0( + embedding float[${dimensions}], + chunk_id TEXT PRIMARY KEY + ); + `); + } + + private listChildVectorTables(): string[] { + return ( + this.db + .prepare( + "SELECT name FROM sqlite_master WHERE type = 'table' AND name LIKE 'workspace_chunks_vec_d%'", + ) + .all() as Array<{ name: string }> + ) + .map((row) => row.name) + .filter((name) => /^workspace_chunks_vec_d\d+$/.test(name)); + } + + // ------------------------------------------------------------------------- + // Embedding queue helpers + // ------------------------------------------------------------------------- + + fetchPendingChunks(versionId: number, limit: number): Array { + return this.db + .prepare( + `SELECT id, chunk_id, resource_id, version_id, workspace_id, chunk_index, chunk_count, + start_position, end_position, content, content_hash, + embedding, embedding_model, embedding_dimensions, embedding_updated_at + FROM workspace_chunks + WHERE version_id = ? AND embedding IS NULL + ORDER BY chunk_index + LIMIT ?`, + ) + .all(versionId, limit) as Array; + } + + writeChunkEmbeddings( + entries: Array<{ chunkId: string; embedding: number[] }>, + model: string, + dimensions: number, + ): void { + if (entries.length === 0) return; + const writeVec = this.vectorSearchAvailable; + const dimensionsTable = this.getChildVectorTableName(dimensions); + this.db.exec("BEGIN"); + try { + const updateStmt = this.db.prepare( + `UPDATE workspace_chunks + SET embedding = ?, embedding_model = ?, embedding_dimensions = ?, embedding_updated_at = ? + WHERE chunk_id = ?`, + ); + const insertVecStmt = writeVec + ? this.db.prepare( + `INSERT OR REPLACE INTO ${dimensionsTable}(embedding, chunk_id) VALUES (?, ?)`, + ) + : null; + const now = currentUnixSeconds(); + for (const entry of entries) { + const buffer = floatArrayToBuffer(entry.embedding); + updateStmt.run(buffer, model, dimensions, now, entry.chunkId); + if (writeVec && buffer && insertVecStmt) insertVecStmt.run(buffer, entry.chunkId); + } + this.db.exec("COMMIT"); + } catch (error) { + this.db.exec("ROLLBACK"); + throw error; + } + } + + markVersionEmbeddingReady(resourceId: number, versionId: number): void { + const stillMissing = this.db + .prepare(`SELECT 1 FROM workspace_chunks WHERE version_id = ? AND embedding IS NULL LIMIT 1`) + .get(versionId); + if (stillMissing) { + this.db + .prepare(`UPDATE workspace_resources SET index_status = 'partial' WHERE id = ?`) + .run(resourceId); + return; + } + this.db + .prepare(`UPDATE workspace_resources SET index_status = 'ready' WHERE id = ?`) + .run(resourceId); + } + + markVersionEmbeddingPartial(resourceId: number, versionId: number, errorMessage: string): void { + // `versionId` and `errorMessage` are reserved for a future + // per-version error column on `workspace_jobs`. For now we just + // flip the resource status to `partial` so callers can see the + // resource is half-indexed. + void versionId; + void errorMessage; + this.db + .prepare(`UPDATE workspace_resources SET index_status = 'partial' WHERE id = ?`) + .run(resourceId); + } + + markVersionEmbeddingFailed(resourceId: number, versionId: number, errorMessage: string): void { + // Same reservation as `markVersionEmbeddingPartial`. Future phases + // will surface the error string on the resource / version row. + void versionId; + void errorMessage; + this.db + .prepare(`UPDATE workspace_resources SET index_status = 'failed' WHERE id = ?`) + .run(resourceId); + } + + markJobFailed(jobId: number | null, errorMessage: string): void { + if (jobId === null) return; + this.db + .prepare(`UPDATE workspace_jobs SET status = 'failed', error = ?, updated_at = ? WHERE id = ?`) + .run(errorMessage, currentUnixSeconds(), jobId); + } + + completeJob(jobId: number, done: number): void { + this.db + .prepare(`UPDATE workspace_jobs SET status = 'ready', done = ?, updated_at = ? WHERE id = ?`) + .run(done, currentUnixSeconds(), jobId); + } + + // ------------------------------------------------------------------------- + // Indexing pipeline (sync portion — embedding fan-out is enqueued separately) + // ------------------------------------------------------------------------- + + async indexResource(input: { + workspace_id: string; + user_id: string; + resource: OkfFolderResource; + }): Promise<{ resource_id: number; version_id: number; change_kind: WorkspaceResourceVersion["change_kind"] }> { + await this.init(); + const { workspace_id, user_id, resource } = input; + const now = currentUnixSeconds(); + const contentHash = sha256(resource.body); + + const tx = this.db.transaction(() => { + const existing = this.db + .prepare( + `SELECT id, current_version_id + FROM workspace_resources + WHERE workspace_id = ? AND canonical_key = ?`, + ) + .get(workspace_id, resource.canonical_key) as { id: number; current_version_id: number | null } | undefined; + + let resourceId: number; + let currentVersionId: number | null = existing?.current_version_id ?? null; + let changeKind: WorkspaceResourceVersion["change_kind"]; + + if (!existing) { + const insertResource = this.db + .prepare( + `INSERT INTO workspace_resources( + workspace_id, user_id, resource_type, canonical_key, title, storage_kind, + current_version_id, index_status, created_at, updated_at, metadata + ) VALUES (?, ?, ?, ?, ?, ?, NULL, 'pending', ?, ?, ?)`, + ) + .run( + workspace_id, + user_id, + resource.resource_type, + resource.canonical_key, + resource.title, + "okf_local_dir", + now, + now, + stringifyJson(resource.front_matter), + ); + resourceId = Number(insertResource.lastInsertRowid); + changeKind = "created"; + } else { + resourceId = existing.id; + if (currentVersionId !== null) { + const prevVersion = this.db + .prepare( + `SELECT sha256 FROM workspace_resource_versions WHERE id = ?`, + ) + .get(currentVersionId) as { sha256: string } | undefined; + if (prevVersion?.sha256 === contentHash) { + changeKind = "unchanged"; + // Touch updated_at so the resource stays "fresh" without + // re-indexing — callers may want to detect staleness + // via `updated_at` vs. `current_version_id.created_at`. + this.db + .prepare(`UPDATE workspace_resources SET updated_at = ? WHERE id = ?`) + .run(now, resourceId); + return { resource_id: resourceId, version_id: currentVersionId, change_kind: changeKind }; + } + } + changeKind = "modified"; + } + + const maxVersion = this.db + .prepare( + `SELECT COALESCE(MAX(version_number), 0) AS max_version + FROM workspace_resource_versions WHERE resource_id = ?`, + ) + .get(resourceId) as { max_version: number }; + + const insertVersion = this.db + .prepare( + `INSERT INTO workspace_resource_versions( + resource_id, version_number, sha256, change_kind, size_bytes, + parent_version_id, source_path, created_at, metadata + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ) + .run( + resourceId, + maxVersion.max_version + 1, + contentHash, + changeKind, + resource.size_bytes, + currentVersionId, + resource.absolute_path, + now, + stringifyJson(resource.front_matter), + ); + const versionId = Number(insertVersion.lastInsertRowid); + + // Delete any old chunks for this resource; FTS5 mirror is + // kept in sync via the AI/AD/AU triggers. + this.db.prepare(`DELETE FROM workspace_chunks WHERE resource_id = ?`).run(resourceId); + + const pieces = chunkTextByEstimatedTokens(resource.body, { + maxTokens: RAW_MESSAGE_CHUNK_MAX_TOKENS, + overlapTokens: RAW_MESSAGE_CHUNK_OVERLAP_TOKENS, + }); + const chunkCount = pieces.length; + const insertChunk = this.db.prepare( + `INSERT INTO workspace_chunks( + chunk_id, resource_id, version_id, workspace_id, chunk_index, chunk_count, + start_position, end_position, content, content_hash + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ); + for (const piece of pieces) { + const pieceHash = sha256(piece.content); + const chunkId = `${workspace_id}:${resourceId}:${versionId}:chunk:${piece.chunkIndex}:${pieceHash.slice(0, 16)}`; + insertChunk.run( + chunkId, + resourceId, + versionId, + workspace_id, + piece.chunkIndex, + chunkCount, + piece.startPosition, + piece.endPosition, + piece.content, + pieceHash, + ); + } + + this.db + .prepare( + `UPDATE workspace_resources + SET current_version_id = ?, index_status = 'pending', updated_at = ?, title = ? + WHERE id = ?`, + ) + .run(versionId, now, resource.title, resourceId); + + currentVersionId = versionId; + return { resource_id: resourceId, version_id: versionId, change_kind: changeKind }; + }); + + const result = tx(); + return result; + } + + // ------------------------------------------------------------------------- + // Reference edges + // ------------------------------------------------------------------------- + + upsertReferenceEdges(input: { + workspace_id: string; + edges: Array<{ + source_resource_id: number; + source_version_id: number | null; + target_resource_id: number; + target_version_id: number | null; + edge_type: WorkspaceEdgeType; + quote?: string | null; + }>; + }): void { + const now = currentUnixSeconds(); + const tx = this.db.transaction(() => { + const insertStmt = this.db.prepare( + `INSERT INTO workspace_reference_edges( + workspace_id, source_resource_id, source_version_id, + target_resource_id, target_version_id, edge_type, quote, created_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(workspace_id, source_resource_id, target_resource_id, edge_type) DO UPDATE SET + source_version_id = excluded.source_version_id, + target_version_id = excluded.target_version_id, + quote = excluded.quote`, + ); + for (const edge of input.edges) { + insertStmt.run( + input.workspace_id, + edge.source_resource_id, + edge.source_version_id, + edge.target_resource_id, + edge.target_version_id, + edge.edge_type, + edge.quote ?? null, + now, + ); + } + }); + tx(); + } + + // ------------------------------------------------------------------------- + // Resource listing + // ------------------------------------------------------------------------- + + listResources(input: ListWorkspaceResourcesInput): ListWorkspaceResourcesResult { + const where: string[] = ["workspace_id = ?"]; + const params: Array = [input.workspace_id]; + if (input.resource_type) { + where.push("resource_type = ?"); + params.push(input.resource_type); + } + if (input.index_status) { + where.push("index_status = ?"); + params.push(input.index_status); + } + const limit = Math.max(1, Math.min(200, Math.floor(input.limit ?? 50))); + const offset = Math.max(0, Math.floor(input.offset ?? 0)); + const rows = this.db + .prepare( + `SELECT * FROM workspace_resources + WHERE ${where.join(" AND ")} + ORDER BY updated_at DESC, id DESC + LIMIT ? OFFSET ?`, + ) + .all(...params, limit, offset) as Array; + const totalRow = this.db + .prepare(`SELECT COUNT(*) AS total FROM workspace_resources WHERE ${where.join(" AND ")}`) + .get(...params) as { total: number }; + return { + total: totalRow.total, + resources: rows.map(toWorkspaceResource), + }; + } + + findResourceByCanonicalKey(input: { workspace_id: string; canonical_key: string }): WorkspaceResource | null { + const row = this.db + .prepare(`SELECT * FROM workspace_resources WHERE workspace_id = ? AND canonical_key = ?`) + .get(input.workspace_id, input.canonical_key) as WorkspaceResourceRow | undefined; + return row ? toWorkspaceResource(row) : null; + } + + // ------------------------------------------------------------------------- + // Search: lexical + // ------------------------------------------------------------------------- + + searchLexical(input: { + workspace_id: string; + user_id: string; + query: string; + resource_types?: string[]; + limit: number; + }): WorkspaceSearchHit[] { + void input.user_id; + const keywords = tokenizeQuery(input.query); + if (keywords.length === 0) return []; + const ftsQuery = keywords.map((kw) => `"${kw.replace(/"/g, '""')}"`).join(" OR "); + const params: Array = []; + let resourceTypeFilter = ""; + if (input.resource_types && input.resource_types.length > 0) { + resourceTypeFilter = `AND pr.resource_type IN (${input.resource_types.map(() => "?").join(",")})`; + params.push(...input.resource_types); + } + const sql = ` + SELECT pc.chunk_id, + pc.resource_id, + pc.version_id, + pr.resource_type, + pr.title AS resource_title, + pr.canonical_key, + pc.content, + bm25(workspace_chunks_fts) AS bm25_score, + pc.chunk_index + FROM workspace_chunks_fts fts + JOIN workspace_chunks pc ON pc.id = fts.rowid + JOIN workspace_resources pr ON pr.id = pc.resource_id + WHERE workspace_chunks_fts MATCH ? + AND pc.workspace_id = ? + ${resourceTypeFilter} + ORDER BY bm25_score ASC + LIMIT ? + `; + params.unshift(ftsQuery, input.workspace_id, input.limit); + const rows = this.db.prepare(sql).all(...params) as Array<{ + chunk_id: string; + resource_id: number; + version_id: number; + resource_type: string; + resource_title: string; + canonical_key: string; + content: string; + bm25_score: number; + chunk_index: number; + }>; + return rows.map((row) => ({ + chunk_id: row.chunk_id, + resource_id: row.resource_id, + version_id: row.version_id, + resource_type: row.resource_type, + resource_title: row.resource_title, + canonical_key: row.canonical_key, + snippet: buildSnippet(row.content, keywords), + matched_terms: keywords, + score: bm25ToSimilarity(row.bm25_score), + signals: { lexical: bm25ToSimilarity(row.bm25_score) }, + reference_edges: this.edgesForResource(row.resource_id), + })); + } + + // ------------------------------------------------------------------------- + // Search: semantic (sqlite-vec KNN, widen-and-retry) + // ------------------------------------------------------------------------- + + searchSemantic(input: { + workspace_id: string; + user_id: string; + queryEmbedding: number[]; + resource_types?: string[]; + limit: number; + threshold: number; + }): WorkspaceSearchHit[] { + void input.user_id; + const tableName = this.getChildVectorTableName(input.queryEmbedding.length); + if (!this.childVectorTableExists(tableName)) { + // No embeddings written yet — caller should fall back to lexical. + return []; + } + const scanLimit = Math.max(input.limit, input.limit * 4); + const vecKnnMaxK = 4096; + let currentScanLimit = scanLimit; + while (true) { + const vecRows = this.db + .prepare( + `SELECT chunk_id, distance + FROM ${tableName} + WHERE embedding MATCH ? + ORDER BY distance + LIMIT ?`, + ) + .all(floatArrayToBuffer(input.queryEmbedding), currentScanLimit) as Array<{ + chunk_id: string; + distance: number; + }>; + if (vecRows.length === 0) return []; + const resourceTypeFilter = + input.resource_types && input.resource_types.length > 0 + ? `AND pr.resource_type IN (${input.resource_types.map(() => "?").join(",")})` + : ""; + const params: Array = [input.workspace_id, ...vecRows.map((r) => r.chunk_id)]; + if (input.resource_types && input.resource_types.length > 0) { + params.push(...input.resource_types); + } + params.push(input.limit); + const hydrated = this.db + .prepare( + `SELECT pc.chunk_id, + pc.resource_id, + pc.version_id, + pr.resource_type, + pr.title AS resource_title, + pr.canonical_key, + pc.content + FROM workspace_chunks pc + JOIN workspace_resources pr ON pr.id = pc.resource_id + WHERE pc.workspace_id = ? + AND pc.chunk_id IN (${vecRows.map(() => "?").join(",")}) + ${resourceTypeFilter} + ORDER BY pc.id ASC + LIMIT ?`, + ) + .all(...params) as Array<{ + chunk_id: string; + resource_id: number; + version_id: number; + resource_type: string; + resource_title: string; + canonical_key: string; + content: string; + }>; + const byDistance = new Map(vecRows.map((row) => [row.chunk_id, row.distance])); + const candidateHits: WorkspaceSearchHit[] = []; + for (const row of hydrated) { + const distance = byDistance.get(row.chunk_id) ?? Number.POSITIVE_INFINITY; + const similarity = sqliteDistanceToSimilarity(distance); + if (similarity < input.threshold) continue; + candidateHits.push({ + chunk_id: row.chunk_id, + resource_id: row.resource_id, + version_id: row.version_id, + resource_type: row.resource_type, + resource_title: row.resource_title, + canonical_key: row.canonical_key, + snippet: buildSnippet(row.content, []), + matched_terms: [], + score: similarity, + signals: { semantic: similarity }, + reference_edges: this.edgesForResource(row.resource_id), + }); + } + const hits = candidateHits.sort((a, b) => b.score - a.score); + if (hits.length >= input.limit || vecRows.length < currentScanLimit) { + return hits.slice(0, input.limit); + } + if (currentScanLimit >= vecKnnMaxK) { + return hits.slice(0, input.limit); + } + currentScanLimit = Math.min(currentScanLimit * 2, vecKnnMaxK); + } + } + + private childVectorTableExists(name: string): boolean { + if (!this.vectorSearchAvailable) return false; + return Boolean( + this.db + .prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?") + .get(name), + ); + } + + // ------------------------------------------------------------------------- + // Cross-file expansion (BFS over workspace_reference_edges) + // ------------------------------------------------------------------------- + + expandNeighbors(input: { + workspace_id: string; + hits: WorkspaceSearchHit[]; + hops: 1 | 2; + limit: number; + }): WorkspaceSearchHit[] { + if (input.hits.length === 0) return input.hits.slice(0, input.limit); + const visited = new Set(input.hits.map((h) => h.resource_id)); + const queue: Array<{ resource_id: number; depth: number }> = input.hits.map((h) => ({ + resource_id: h.resource_id, + depth: 0, + })); + const expansions: WorkspaceSearchHit[] = []; + const edgeBoost = (resourceId: number): number => { + const out = this.db + .prepare( + `SELECT COUNT(*) AS c FROM workspace_reference_edges + WHERE workspace_id = ? AND source_resource_id = ?`, + ) + .get(input.workspace_id, resourceId) as { c: number }; + const inc = this.db + .prepare( + `SELECT COUNT(*) AS c FROM workspace_reference_edges + WHERE workspace_id = ? AND target_resource_id = ?`, + ) + .get(input.workspace_id, resourceId) as { c: number }; + return 0.1 * (out.c + inc.c); + }; + while (queue.length > 0 && expansions.length < input.limit * 2) { + const head = queue.shift(); + if (!head) break; + if (head.depth >= input.hops) continue; + const neighborIds = this.db + .prepare( + `SELECT DISTINCT target_resource_id AS id FROM workspace_reference_edges + WHERE workspace_id = ? AND source_resource_id = ? + UNION + SELECT DISTINCT source_resource_id AS id FROM workspace_reference_edges + WHERE workspace_id = ? AND target_resource_id = ?`, + ) + .all(input.workspace_id, head.resource_id, input.workspace_id, head.resource_id) as Array<{ + id: number; + }>; + for (const { id } of neighborIds) { + if (visited.has(id)) continue; + visited.add(id); + const boost = edgeBoost(id); + const chunkRow = this.db + .prepare( + `SELECT pc.chunk_id, pc.resource_id, pc.version_id, pc.content, pc.chunk_index, + pr.resource_type, pr.title AS resource_title, pr.canonical_key + FROM workspace_chunks pc + JOIN workspace_resources pr ON pr.id = pc.resource_id + WHERE pc.resource_id = ? + ORDER BY pc.chunk_index ASC + LIMIT 1`, + ) + .get(id) as + | { + chunk_id: string; + resource_id: number; + version_id: number; + content: string; + chunk_index: number; + resource_type: string; + resource_title: string; + canonical_key: string; + } + | undefined; + if (!chunkRow) continue; + expansions.push({ + chunk_id: chunkRow.chunk_id, + resource_id: chunkRow.resource_id, + version_id: chunkRow.version_id, + resource_type: chunkRow.resource_type, + resource_title: chunkRow.resource_title, + canonical_key: chunkRow.canonical_key, + snippet: buildSnippet(chunkRow.content, []), + matched_terms: [], + score: boost, + signals: { edge_boost: boost }, + reference_edges: this.edgesForResource(chunkRow.resource_id), + }); + queue.push({ resource_id: id, depth: head.depth + 1 }); + } + } + const seen = new Set(input.hits.map((h) => h.resource_id)); + const dedupedExpansions: WorkspaceSearchHit[] = []; + for (const expansion of expansions) { + if (seen.has(expansion.resource_id)) continue; + seen.add(expansion.resource_id); + dedupedExpansions.push(expansion); + } + const merged = [...input.hits, ...dedupedExpansions] + .sort((a, b) => b.score - a.score) + .slice(0, input.limit); + return merged; + } + + private edgesForResource(resourceId: number): WorkspaceSearchHit["reference_edges"] { + const rows = this.db + .prepare( + `SELECT edge_type, target_resource_id FROM workspace_reference_edges + WHERE source_resource_id = ? OR target_resource_id = ? + LIMIT 16`, + ) + .all(resourceId, resourceId) as Array<{ edge_type: WorkspaceEdgeType; target_resource_id: number }>; + return rows; + } + + // ------------------------------------------------------------------------- + // Soft-delete + // ------------------------------------------------------------------------- + + softDeleteMissingResources(input: { + workspace_id: string; + presentKeys: Set; + }): Array<{ canonical_key: string }> { + const allRows = this.db + .prepare(`SELECT id, canonical_key, metadata FROM workspace_resources WHERE workspace_id = ?`) + .all(input.workspace_id) as Array<{ id: number; canonical_key: string; metadata: string | null }>; + const missing: Array<{ canonical_key: string }> = []; + const now = currentUnixSeconds(); + for (const row of allRows) { + if (input.presentKeys.has(row.canonical_key)) continue; + const meta = parseJson>(row.metadata, {}) ?? {}; + if (meta.deleted_at) continue; + meta.deleted_at = now; + this.db + .prepare(`UPDATE workspace_resources SET metadata = ?, updated_at = ? WHERE id = ?`) + .run(stringifyJson(meta), now, row.id); + missing.push({ canonical_key: row.canonical_key }); + } + return missing; + } + + // ------------------------------------------------------------------------- + // Job helpers (used by okf-backend + tests) + // ------------------------------------------------------------------------- + + createJob(input: { workspace_id: string; kind: WorkspaceJob["kind"]; total: number }): WorkspaceJob { + const now = currentUnixSeconds(); + const stmt = this.db + .prepare( + `INSERT INTO workspace_jobs(workspace_id, kind, status, total, done, created_at, updated_at) + VALUES (?, ?, 'pending', ?, 0, ?, ?)`, + ) + .run(input.workspace_id, input.kind, input.total, now, now); + const row = this.db + .prepare(`SELECT * FROM workspace_jobs WHERE id = ?`) + .get(Number(stmt.lastInsertRowid)) as WorkspaceJobRow; + return toWorkspaceJob(row); + } + + updateJobTotal(jobId: number, total: number): void { + this.db + .prepare(`UPDATE workspace_jobs SET total = ?, updated_at = ? WHERE id = ?`) + .run(total, currentUnixSeconds(), jobId); + } +} + +function tokenizeQuery(query: string): string[] { + return query + .split(/\s+/u) + .map((token) => token.trim()) + .filter((token) => token.length > 0); +} + +function buildSnippet(content: string, keywords: string[]): string { + const max = 160; + if (content.length <= max) return content; + if (keywords.length === 0) return `${content.slice(0, max)}…`; + const lower = content.toLowerCase(); + for (const keyword of keywords) { + const idx = lower.indexOf(keyword.toLowerCase()); + if (idx >= 0) { + const start = Math.max(0, idx - 40); + const end = Math.min(content.length, idx + keyword.length + 120); + return `${start > 0 ? "…" : ""}${content.slice(start, end)}${end < content.length ? "…" : ""}`; + } + } + return `${content.slice(0, max)}…`; +} + +function bm25ToSimilarity(score: number): number { + if (!Number.isFinite(score)) return 0; + // bm25() returns negative numbers in SQLite FTS5; lower = better. + const normalised = 1 / (1 + Math.abs(score)); + return Math.min(1, Math.max(0, normalised)); +} + +function sqliteDistanceToSimilarity(distance: number): number { + if (!Number.isFinite(distance)) return 0; + // sqlite-vec returns L2 distance. Embeddings are L2-normalised upstream + // (the same assumption `sqliteVectorDistanceToCosineSimilarity` makes in + // `packages/sqlite/src/raw-message-manager.ts`), so the conversion is + // the standard `cosine_similarity = 1 - distance^2 / 2` clamped to [-1, 1]. + return Math.max(-1, 1 - (distance * distance) / 2); +} + +// ------------------------------------------------------------------------- +// Singleton accessor +// ------------------------------------------------------------------------- + +let _instance: SqliteWorkspaceStore | undefined; + +export async function getSQLiteWorkspaceStore( + options: SqliteWorkspaceStoreOptions = {}, +): Promise { + if (!_instance) { + const dbPath = options.dbPath ?? resolveWorkspaceDbPath(); + mkdirSync(dirname(dbPath), { recursive: true }); + const instance = new SqliteWorkspaceStore({ ...options, dbPath }); + await instance.init(); + _instance = instance; + } + return _instance; +} + +export async function closeSQLiteWorkspaceStore(): Promise { + if (!_instance) return; + await _instance.close(); + _instance = undefined; +} + +/** Test-only reset hook; mirrors `__resetSQLiteRawMessageManagerForTests`. */ +export function __resetSQLiteWorkspaceStoreForTests(): void { + _instance = undefined; +} + +/** + * Build a `SqliteWorkspaceStore` from an existing in-memory database handle. + * Convenience for tests that want to share a single connection with the + * raw-message store (e.g. to verify FK / multi-table integration). + */ +export async function createSqliteWorkspaceStore( + options: SqliteWorkspaceStoreOptions = {}, +): Promise { + const instance = new SqliteWorkspaceStore(options); + await instance.init(); + return instance; +} + +// ------------------------------------------------------------------------- +// Public helpers exported for the API layer (api.ts) and OKF backend. +// ------------------------------------------------------------------------- + +export async function runUpdateWorkspaceContext( + store: SqliteWorkspaceStore, + input: UpdateWorkspaceContextInput & { user_id: string }, + hooks: { + indexOkfFolder: ( + workspace_id: string, + user_id: string, + path: string, + ) => Promise; + }, +): Promise { + return hooks.indexOkfFolder(input.workspace_id, input.user_id, input.path); +} + +export async function runSearchWorkspaceContext( + store: SqliteWorkspaceStore, + input: SearchWorkspaceContextInput & { user_id: string }, + hooks: { + searchLexical: ( + workspace_id: string, + user_id: string, + query: string, + resource_types?: string[], + limit?: number, + ) => WorkspaceSearchHit[]; + searchSemantic: ( + workspace_id: string, + user_id: string, + queryEmbedding: number[], + resource_types?: string[], + limit?: number, + threshold?: number, + ) => WorkspaceSearchHit[]; + expandNeighbors: ( + workspace_id: string, + hits: WorkspaceSearchHit[], + hops: 1 | 2, + limit: number, + ) => WorkspaceSearchHit[]; + generateEmbedding?: (text: string) => Promise; + fuse?: (lexical: WorkspaceSearchHit[], semantic: WorkspaceSearchHit[], limit: number) => WorkspaceSearchHit[]; + }, +): Promise { + const strategy: WorkspaceSearchStrategy = input.strategy ?? "hybrid"; + const options = input.options ?? {}; + const limit = Math.max(1, Math.min(50, Math.floor(options.limit ?? 10))); + const threshold = options.threshold ?? 0.7; + const resourceTypes = options.resource_types; + + let hits: WorkspaceSearchHit[] = []; + if (strategy === "lexical") { + hits = hooks.searchLexical(input.workspace_id, input.user_id, input.query, resourceTypes, limit); + } else if (strategy === "semantic") { + if (!hooks.generateEmbedding) { + hits = []; + } else { + const embedding = await hooks.generateEmbedding(input.query); + hits = hooks.searchSemantic( + input.workspace_id, + input.user_id, + embedding, + resourceTypes, + limit, + threshold, + ); + if (hits.length === 0) { + // Semantic fallback to lexical so users still get a hit + // before embeddings are written. + hits = hooks.searchLexical(input.workspace_id, input.user_id, input.query, resourceTypes, limit); + } + } + } else if (strategy === "hybrid") { + const candidateLimit = limit * 4; + const lexicalHits = hooks.searchLexical( + input.workspace_id, + input.user_id, + input.query, + resourceTypes, + candidateLimit, + ); + let semanticHits: WorkspaceSearchHit[] = []; + if (hooks.generateEmbedding) { + const embedding = await hooks.generateEmbedding(input.query); + semanticHits = hooks.searchSemantic( + input.workspace_id, + input.user_id, + embedding, + resourceTypes, + candidateLimit, + threshold, + ); + } + hits = hooks.fuse ? hooks.fuse(lexicalHits, semanticHits, limit) : lexicalHits.slice(0, limit); + } else { + const candidateLimit = limit * 4; + const lexicalHits = hooks.searchLexical( + input.workspace_id, + input.user_id, + input.query, + resourceTypes, + candidateLimit, + ); + let semanticHits: WorkspaceSearchHit[] = []; + if (hooks.generateEmbedding) { + const embedding = await hooks.generateEmbedding(input.query); + semanticHits = hooks.searchSemantic( + input.workspace_id, + input.user_id, + embedding, + resourceTypes, + candidateLimit, + threshold, + ); + } + const fused = hooks.fuse ? hooks.fuse(lexicalHits, semanticHits, candidateLimit) : lexicalHits; + hits = hooks.expandNeighbors( + input.workspace_id, + fused, + options.hops ?? 1, + limit, + ); + } + return { + query: input.query, + strategy, + total: hits.length, + hits, + }; +} diff --git a/packages/workspace/src/types.ts b/packages/workspace/src/types.ts new file mode 100644 index 00000000..fa60e9e9 --- /dev/null +++ b/packages/workspace/src/types.ts @@ -0,0 +1,183 @@ +/** + * `@melandlabs/workspace` — core types. + * + * All cross-package inputs / outputs flow through this module so the + * API surface (`updateWorkspaceContext` / `searchWorkspaceContext` / + * `listWorkspaceResources`) and the HTTP / MCP wiring can share one + * canonical definition. + */ + +import type { OkfFrontMatter } from "@melandlabs/contracts"; + +export interface RuntimeContext { + user_id: string; + employee_id?: string; + session_id?: string; + request_id: string; + mode?: "sandbox" | "client"; + auth_token?: string; +} + +export type WorkspaceStorageKind = "okf_local_dir" | "inline"; +export type WorkspaceEdgeType = "cites" | "supersedes" | "amends" | "relates-to"; +export type WorkspaceIndexStatus = "pending" | "partial" | "ready" | "failed"; +export type WorkspaceSearchStrategy = "lexical" | "semantic" | "hybrid" | "cross-file"; + +export interface WorkspaceResource { + id: number; + workspace_id: string; + user_id: string; + resource_type: string; + canonical_key: string; + title: string; + storage_kind: WorkspaceStorageKind; + current_version_id: number | null; + index_status: WorkspaceIndexStatus; + created_at: number; + updated_at: number; + metadata?: Record; +} + +export interface WorkspaceResourceVersion { + id: number; + resource_id: number; + version_number: number; + sha256: string; + change_kind: "created" | "modified" | "unchanged"; + size_bytes: number; + parent_version_id: number | null; + source_path: string | null; + created_at: number; + metadata?: Record; +} + +export interface WorkspaceChunk { + id: number; + chunk_id: string; + resource_id: number; + version_id: number; + workspace_id: string; + chunk_index: number; + chunk_count: number; + start_position: number; + end_position: number; + content: string; + content_hash: string; + embedding?: number[]; + embedding_model?: string; + embedding_dimensions?: number; + embedding_updated_at?: number; +} + +export interface WorkspaceReferenceEdge { + id: number; + workspace_id: string; + source_resource_id: number; + source_version_id: number | null; + target_resource_id: number; + target_version_id: number | null; + edge_type: WorkspaceEdgeType; + quote?: string | null; + created_at: number; +} + +export interface WorkspaceJob { + id: number; + workspace_id: string; + kind: "index" | "update"; + status: WorkspaceIndexStatus; + total: number; + done: number; + error: string | null; + created_at: number; + updated_at: number; +} + +export interface WorkspaceSearchHit { + chunk_id: string; + resource_id: number; + version_id: number; + resource_type: string; + resource_title: string; + canonical_key: string; + snippet: string; + matched_terms: string[]; + score: number; + signals: { + lexical?: number; + semantic?: number; + edge_boost?: number; + }; + reference_edges: Array<{ edge_type: WorkspaceEdgeType; target_resource_id: number }>; +} + +export interface SearchWorkspaceContextOptions { + /** Default 10, max 50. */ + limit?: number; + resource_types?: string[]; + /** Default 0.7. */ + threshold?: number; + /** Only honoured by the `cross-file` strategy. */ + include_edges?: boolean; + /** Only honoured by the `cross-file` strategy. */ + hops?: 1 | 2; +} + +export interface UpdateWorkspaceContextInput { + workspace_id: string; + source: "okf_folder"; + path: string; + metadata?: Record; +} + +export interface UpdateWorkspaceContextResult { + jobId: number; + triggered: true; + status: WorkspaceIndexStatus; + filesScanned: number; + filesAdded: number; + filesModified: number; + filesUnchanged: number; + filesDeleted: number; +} + +export interface SearchWorkspaceContextInput { + workspace_id: string; + query: string; + strategy?: WorkspaceSearchStrategy; + options?: SearchWorkspaceContextOptions; +} + +export interface SearchWorkspaceContextResult { + query: string; + strategy: WorkspaceSearchStrategy; + total: number; + hits: WorkspaceSearchHit[]; +} + +export interface ListWorkspaceResourcesInput { + workspace_id: string; + resource_type?: string; + index_status?: WorkspaceIndexStatus; + limit?: number; + offset?: number; +} + +export interface ListWorkspaceResourcesResult { + total: number; + resources: WorkspaceResource[]; +} + +/** + * Internal: the resource shape produced by `okf-backend.listOkfFolderResources` + * and consumed by `SqliteWorkspaceStore.indexResource`. + */ +export interface OkfFolderResource { + canonical_key: string; + absolute_path: string; + title: string; + resource_type: string; + body: string; + front_matter?: OkfFrontMatter; + size_bytes: number; +} diff --git a/packages/workspace/test/okf-backend.test.ts b/packages/workspace/test/okf-backend.test.ts new file mode 100644 index 00000000..cdb1bdc5 --- /dev/null +++ b/packages/workspace/test/okf-backend.test.ts @@ -0,0 +1,125 @@ +/** + * Tests for `okf-backend.indexOkfFolder` — sha256 dedup, cites-edge + * extraction, and soft-delete detection across re-runs. + */ +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { indexOkfFolder } from "../src/okf-backend"; +import { SqliteWorkspaceStore } from "../src/sqlite"; + +let scratchDir: string; + +beforeEach(() => { + scratchDir = mkdtempSync(join(tmpdir(), "workspace-okf-")); +}); + +afterEach(() => { + rmSync(scratchDir, { recursive: true, force: true }); +}); + +function writeFile(folder: string, relativePath: string, content: string): void { + const absolute = join(folder, relativePath); + mkdirSync(join(absolute, ".."), { recursive: true }); + writeFileSync(absolute, content, "utf8"); +} + +describe("indexOkfFolder", () => { + it("indexes every .md file, computes sha256 dedup, and persists cites edges", async () => { + const okfRoot = join(scratchDir, "wiki"); + mkdirSync(okfRoot, { recursive: true }); + writeFile( + okfRoot, + "contract.md", + "---\ntype: Reference\n---\n# Contract\n\nSee [law](./law.md) for the governing provision.\n", + ); + writeFile( + okfRoot, + "law.md", + "---\ntype: Reference\n---\n# Law\n\nCivil code article 123 is the controlling clause.\n", + ); + writeFile( + okfRoot, + "policy.md", + "---\ntype: Reference\n---\n# Policy\n\nThis policy is not linked from any other document.\n", + ); + + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + const enqueued: Array<{ resource_id: number; version_id: number }> = []; + const result = await indexOkfFolder(store, { + workspace_id: "p1", + user_id: "u1", + path: okfRoot, + enqueueEmbedding: async (input) => { + enqueued.push({ resource_id: input.resource_id, version_id: input.version_id }); + }, + }); + expect(result.filesScanned).toBe(3); + expect(result.filesAdded).toBe(3); + expect(result.filesModified).toBe(0); + expect(result.filesUnchanged).toBe(0); + expect(enqueued.length).toBe(3); + // Re-run: every file should be unchanged. + const second = await indexOkfFolder(store, { + workspace_id: "p1", + user_id: "u1", + path: okfRoot, + enqueueEmbedding: async () => {}, + }); + expect(second.filesScanned).toBe(3); + expect(second.filesAdded).toBe(0); + expect(second.filesModified).toBe(0); + expect(second.filesUnchanged).toBe(3); + // Cites edge: contract → law. + const edgeRows = store.__testDb + .prepare( + `SELECT pr1.canonical_key AS source, pr2.canonical_key AS target, edge_type + FROM workspace_reference_edges e + JOIN workspace_resources pr1 ON pr1.id = e.source_resource_id + JOIN workspace_resources pr2 ON pr2.id = e.target_resource_id + WHERE edge_type = 'cites'`, + ) + .all() as Array<{ source: string; target: string; edge_type: string }>; + const contractLaw = edgeRows.find( + (row) => row.source === "contract.md" && row.target === "law.md", + ); + expect(contractLaw).toBeDefined(); + await store.close(); + }); + + it("soft-deletes a resource whose file disappeared on a re-run", async () => { + const okfRoot = join(scratchDir, "wiki"); + mkdirSync(okfRoot, { recursive: true }); + writeFile(okfRoot, "keep.md", "keep me"); + writeFile(okfRoot, "drop.md", "delete me next run"); + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + const first = await indexOkfFolder(store, { + workspace_id: "p1", + user_id: "u1", + path: okfRoot, + enqueueEmbedding: async () => {}, + }); + expect(first.filesAdded).toBe(2); + rmSync(join(okfRoot, "drop.md")); + const second = await indexOkfFolder(store, { + workspace_id: "p1", + user_id: "u1", + path: okfRoot, + enqueueEmbedding: async () => {}, + }); + expect(second.filesDeleted).toBe(1); + const rows = store.__testDb + .prepare(`SELECT canonical_key, metadata FROM workspace_resources ORDER BY canonical_key`) + .all() as Array<{ canonical_key: string; metadata: string | null }>; + const dropped = rows.find((row) => row.canonical_key === "drop.md"); + expect(dropped).toBeDefined(); + const meta = JSON.parse(dropped?.metadata ?? "{}") as { deleted_at?: number }; + expect(typeof meta.deleted_at).toBe("number"); + await store.close(); + }); +}); diff --git a/packages/workspace/test/parsers-adapter.test.ts b/packages/workspace/test/parsers-adapter.test.ts new file mode 100644 index 00000000..53255506 --- /dev/null +++ b/packages/workspace/test/parsers-adapter.test.ts @@ -0,0 +1,53 @@ +/** + * Tests for `parsers-adapter.extractText`. Validates the text-extraction + * path for plain `.md` / `.txt` files (which should be a UTF-8 pass-through) + * and that the MIME detection is stable across file extensions. + */ +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { detectMimeType, extractText } from "../src/parsers-adapter"; + +let scratchDir: string; + +beforeEach(() => { + scratchDir = mkdtempSync(join(tmpdir(), "workspace-parsers-")); +}); + +afterEach(() => { + rmSync(scratchDir, { recursive: true, force: true }); +}); + +describe("parsers-adapter", () => { + it("detects MIME types by extension", () => { + expect(detectMimeType("a.md")).toBe("text/markdown"); + expect(detectMimeType("a.markdown")).toBe("text/markdown"); + expect(detectMimeType("a.txt")).toBe("text/plain"); + expect(detectMimeType("a.pdf")).toBe("application/pdf"); + expect(detectMimeType("a.docx")).toBe("application/vnd.openxmlformats-officedocument.wordprocessingml.document"); + expect(detectMimeType("a.pages")).toBe("application/x-iwork-pages-sffpages"); + expect(detectMimeType("a.unknown")).toBe("application/octet-stream"); + }); + + it("extracts text from a plain markdown file", async () => { + const filePath = join(scratchDir, "note.md"); + mkdirSync(scratchDir, { recursive: true }); + writeFileSync(filePath, "# Title\n\nThis is the body.", "utf8"); + const result = await extractText(filePath); + expect(result.text).toContain("# Title"); + expect(result.text).toContain("This is the body."); + expect(result.mimeType).toBe("text/markdown"); + }); + + it("extracts text from a plain text file", async () => { + const filePath = join(scratchDir, "notes.txt"); + mkdirSync(scratchDir, { recursive: true }); + writeFileSync(filePath, "first line\nsecond line\n", "utf8"); + const result = await extractText(filePath); + expect(result.text).toBe("first line\nsecond line\n"); + expect(result.mimeType).toBe("text/plain"); + }); +}); diff --git a/packages/workspace/test/search.test.ts b/packages/workspace/test/search.test.ts new file mode 100644 index 00000000..1cf5f827 --- /dev/null +++ b/packages/workspace/test/search.test.ts @@ -0,0 +1,189 @@ +/** + * Tests for the lexical / semantic / hybrid / cross-file search pipeline. + * Embeddings are mocked with deterministic 1536-dim vectors so we never hit + * OpenRouter and the distance ordering is reproducible. + */ +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { searchWorkspaceContext } from "../src/api"; +import { searchLexical } from "../src/search/lexical"; +import { searchSemantic } from "../src/search/semantic"; +import { fuseHybridHits } from "../src/search/hybrid"; +import { searchCrossFile } from "../src/search/cross-file"; +import { SqliteWorkspaceStore } from "../src/sqlite"; +import type { OkfFolderResource, WorkspaceSearchHit } from "../src/types"; + +let scratchDir: string; + +beforeEach(() => { + scratchDir = mkdtempSync(join(tmpdir(), "workspace-search-")); +}); + +afterEach(() => { + rmSync(scratchDir, { recursive: true, force: true }); +}); + +function makeResource(overrides: Partial & Pick): OkfFolderResource { + return { + absolute_path: `/fake/${overrides.canonical_key}`, + title: overrides.canonical_key, + resource_type: overrides.canonical_key.endsWith(".md") ? "note" : "document", + size_bytes: overrides.body.length, + ...overrides, + }; +} + +const DIMS = 1536; + +function makeDeterministicVector(seed: number): number[] { + // Push the vector onto a unit sphere so cosine distance is + // reproducible. Two resources whose `seed` shares any high bits get + // similar vectors, which mirrors "documents on the same topic". + const vec = new Array(DIMS).fill(0); + let state = seed; + for (let i = 0; i < DIMS; i += 1) { + state = (state * 1103515245 + 12345) >>> 0; + vec[i] = (state / 0xffffffff) * 2 - 1; + } + let norm = 0; + for (const v of vec) norm += v * v; + norm = Math.sqrt(norm) || 1; + return vec.map((v) => v / norm); +} + +async function indexThreeDocuments(store: SqliteWorkspaceStore): Promise { + await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "Reference/contract.md", body: "limitation of liability clauses must conform to applicable law".trim() }), + }); + await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "Reference/law.md", body: "civil code article 123: parties may limit liability unless the law forbids it".trim() }), + }); + await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "Reference/cookie-policy.md", body: "this document describes cookie usage on the marketing site".trim() }), + }); +} + +describe("search pipeline", () => { + it("returns BM25-ranked lexical hits for matching keywords", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + await indexThreeDocuments(store); + const hits = searchLexical(store, { + workspace_id: "p1", + user_id: "u1", + query: "limitation liability", + limit: 5, + }); + expect(hits.length).toBeGreaterThan(0); + expect(hits[0]?.resource_title).toContain("contract.md"); + expect(hits[0]?.signals.lexical).toBeGreaterThan(0); + await store.close(); + }); + + it("returns no semantic hits when embeddings haven't been written", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + await indexThreeDocuments(store); + const hits = searchSemantic(store, { + workspace_id: "p1", + user_id: "u1", + queryEmbedding: makeDeterministicVector(1), + limit: 5, + threshold: 0.0, + }); + expect(hits).toEqual([]); + await store.close(); + }); + + it("fuses lexical + semantic candidates via RRF (k=60)", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + await indexThreeDocuments(store); + const lexical = searchLexical(store, { + workspace_id: "p1", + user_id: "u1", + query: "limitation", + limit: 5, + }); + // Fabricate semantic hits with monotonically decreasing scores so + // the RRF fusion ranks the top one highest. + const semantic: WorkspaceSearchHit[] = lexical.map((hit, index) => ({ + ...hit, + score: 0.9 - index * 0.1, + signals: { semantic: 0.9 - index * 0.1 }, + })); + const fused = fuseHybridHits({ lexical, semantic, limit: 3 }); + expect(fused.length).toBeGreaterThan(0); + expect(fused[0]?.score).toBeGreaterThanOrEqual(fused[1]?.score ?? 0); + await store.close(); + }); + + it("expands hits via cites edges for cross-file search", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + const contract = await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "Reference/contract.md", body: "limitation clause text" }), + }); + const law = await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "Reference/law.md", body: "civil code article 123" }), + }); + store.upsertReferenceEdges({ + workspace_id: "p1", + edges: [ + { + source_resource_id: contract.resource_id, + source_version_id: contract.version_id, + target_resource_id: law.resource_id, + target_version_id: law.version_id, + edge_type: "cites", + }, + ], + }); + const hits = await searchCrossFile(store, { + workspace_id: "p1", + user_id: "u1", + query: "limitation", + limit: 5, + lexicalSearch: (params) => searchLexical(store, params), + semanticSearch: (params) => searchSemantic(store, params), + }); + expect(hits.length).toBeGreaterThan(0); + const lawHit = hits.find((hit) => hit.resource_id === law.resource_id); + expect(lawHit).toBeDefined(); + expect(lawHit?.signals.edge_boost).toBeGreaterThan(0); + await store.close(); + }); + + it("falls back to lexical-only when embeddings fail during hybrid search", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + await indexThreeDocuments(store); + const result = await searchWorkspaceContext( + { user_id: "u1", request_id: "req-1" }, + store, + { workspace_id: "p1", query: "limitation", strategy: "hybrid" }, + { + embed: async () => { + throw new Error("simulated embedding failure"); + }, + }, + ); + expect(result.total).toBeGreaterThan(0); + expect(result.strategy).toBe("hybrid"); + await store.close(); + }); +}); diff --git a/packages/workspace/test/sqlite-project-store.test.ts b/packages/workspace/test/sqlite-project-store.test.ts new file mode 100644 index 00000000..86648183 --- /dev/null +++ b/packages/workspace/test/sqlite-project-store.test.ts @@ -0,0 +1,153 @@ +/** + * Tests for `SqliteWorkspaceStore` — schema bootstrap, resource/version + * chain, sha256 dedup, unchanged short-circuit, and version parent + * linkage. + */ +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { SqliteWorkspaceStore } from "../src/sqlite"; +import type { OkfFolderResource } from "../src/types"; + +let scratchDir: string; + +beforeEach(() => { + scratchDir = mkdtempSync(join(tmpdir(), "workspace-store-")); +}); + +afterEach(() => { + rmSync(scratchDir, { recursive: true, force: true }); +}); + +function makeResource(overrides: Partial & Pick): OkfFolderResource { + return { + absolute_path: `/fake/${overrides.canonical_key}`, + title: overrides.canonical_key, + resource_type: "note", + size_bytes: overrides.body.length, + ...overrides, + }; +} + +describe("SqliteWorkspaceStore", () => { + it("initialises every schema table on init()", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + const tables = (store as unknown as { __testDb: { prepare: (s: string) => { all: () => unknown[] } } }) + .__testDb.prepare(`SELECT name FROM sqlite_master WHERE type IN ('table', 'view', 'trigger')`) + .all() as Array<{ name: string }>; + const names = tables.map((row) => row.name); + expect(names).toContain("workspace_resources"); + expect(names).toContain("workspace_resource_versions"); + expect(names).toContain("workspace_chunks"); + expect(names).toContain("workspace_chunks_fts"); + expect(names).toContain("workspace_reference_edges"); + expect(names).toContain("workspace_jobs"); + await store.close(); + }); + + it("creates a resource + version + chunks on first indexResource", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + const result = await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ + canonical_key: "docs/intro.md", + body: "Hello world. This is the first document for the project.", + }), + }); + expect(result.change_kind).toBe("created"); + expect(result.resource_id).toBeGreaterThan(0); + const chunks = store.__testDb + .prepare(`SELECT chunk_index, chunk_count, content FROM workspace_chunks ORDER BY chunk_index`) + .all() as Array<{ chunk_index: number; chunk_count: number; content: string }>; + expect(chunks.length).toBeGreaterThan(0); + expect(chunks[0]?.chunk_count).toBe(chunks.length); + await store.close(); + }); + + it("short-circuits with change_kind='unchanged' on identical re-index", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + const body = "Identical body for sha256 dedup test."; + const first = await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "docs/notes.md", body }), + }); + expect(first.change_kind).toBe("created"); + const second = await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "docs/notes.md", body }), + }); + expect(second.change_kind).toBe("unchanged"); + expect(second.version_id).toBe(first.version_id); + await store.close(); + }); + + it("creates a new version with parent_version_id on modified content", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + const first = await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "docs/changelog.md", body: "version 1 body" }), + }); + const second = await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "docs/changelog.md", body: "version 2 body — modified!" }), + }); + expect(second.change_kind).toBe("modified"); + expect(second.version_id).not.toBe(first.version_id); + const versions = store.__testDb + .prepare(`SELECT id, version_number, parent_version_id FROM workspace_resource_versions ORDER BY version_number`) + .all() as Array<{ id: number; version_number: number; parent_version_id: number | null }>; + expect(versions.length).toBe(2); + expect(versions[1]?.parent_version_id).toBe(versions[0]?.id ?? null); + await store.close(); + }); + + it("soft-deletes missing resources when their canonical_key disappears", async () => { + const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); + await store.init(); + await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "docs/a.md", body: "alpha" }), + }); + await store.indexResource({ + workspace_id: "p1", + user_id: "u1", + resource: makeResource({ canonical_key: "docs/b.md", body: "bravo" }), + }); + const missing = store.softDeleteMissingResources({ + workspace_id: "p1", + presentKeys: new Set(["docs/a.md"]), + }); + expect(missing.map((entry) => entry.canonical_key)).toEqual(["docs/b.md"]); + const afterRows = store.__testDb + .prepare(`SELECT canonical_key, metadata FROM workspace_resources ORDER BY canonical_key`) + .all() as Array<{ canonical_key: string; metadata: string | null }>; + const deletedRow = afterRows.find((row) => row.canonical_key === "docs/b.md"); + expect(deletedRow).toBeDefined(); + const meta = JSON.parse(deletedRow?.metadata ?? "{}") as { deleted_at?: number }; + expect(typeof meta.deleted_at).toBe("number"); + await store.close(); + }); + + it("reuses init() safely across multiple new instances on the same DB", async () => { + const dbPath = join(scratchDir, "store.db"); + const first = new SqliteWorkspaceStore({ dbPath }); + await first.init(); + await first.close(); + const second = new SqliteWorkspaceStore({ dbPath }); + await expect(second.init()).resolves.toBeUndefined(); + await second.close(); + }); +}); diff --git a/packages/workspace/tsconfig.json b/packages/workspace/tsconfig.json new file mode 100644 index 00000000..b6545e66 --- /dev/null +++ b/packages/workspace/tsconfig.json @@ -0,0 +1,11 @@ +{ + "extends": "../config/src/tsconfig.json", + "compilerOptions": { + "outDir": "./dist", + "rootDir": "..", + "incremental": false, + "tsBuildInfoFile": "./dist/.tsbuildinfo", + "noEmit": false + }, + "include": ["src"] +} diff --git a/packages/workspace/tsup.config.ts b/packages/workspace/tsup.config.ts new file mode 100644 index 00000000..04b6a74f --- /dev/null +++ b/packages/workspace/tsup.config.ts @@ -0,0 +1,40 @@ +import { defineConfig } from "tsup"; + +/** + * Multi-entry build for `@melandlabs/workspace`. Two entry points: + * + * - `index` — public surface (api, types, schema, sqlite helpers) + * - `cli` — `opencontext workspace …` subcommand for local indexing + search + * - `sqlite` — convenience entry so tests and internal callers can + * `import "@melandlabs/workspace/sqlite"` without dragging + * the rest of the barrel + */ +export default defineConfig({ + entry: { + index: "src/index.ts", + sqlite: "src/sqlite.ts", + cli: "src/cli.ts", + }, + format: ["esm"], + dts: true, + sourcemap: false, + clean: true, + splitting: false, + treeshake: true, + external: [ + "react", + "react-dom", + "better-sqlite3", + "sqlite-vec", + "hono", + "zod", + "@modelcontextprotocol/sdk", + "@hono/node-server", + "@melandlabs/ai-rag", + "@melandlabs/contracts", + "@melandlabs/okf", + "@melandlabs/rag", + "@melandlabs/shared", + "@melandlabs/sqlite", + ], +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 1a2b342a..7d593595 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -1187,6 +1187,9 @@ importers: '@melandlabs/security': specifier: ^0.3.0 version: link:../security + '@melandlabs/workspace': + specifier: ^0.1.0 + version: link:../workspace '@modelcontextprotocol/sdk': specifier: ^1.25.3 version: 1.30.0(@cfworker/json-schema@4.1.1)(zod@4.4.3) @@ -1519,6 +1522,61 @@ importers: specifier: ^2.1.0 version: 2.1.9(@types/node@26.2.0)(jiti@2.7.0)(jsdom@24.1.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(tsx@4.23.12)(yaml@2.9.0) + packages/workspace: + dependencies: + '@melandlabs/contracts': + specifier: workspace:* + version: link:../contracts + '@melandlabs/env-config': + specifier: workspace:* + version: link:../env-config + '@melandlabs/okf': + specifier: workspace:* + version: link:../okf + '@melandlabs/rag': + specifier: workspace:* + version: link:../rag + '@melandlabs/shared': + specifier: workspace:* + version: link:../shared + '@melandlabs/sqlite': + specifier: workspace:* + version: link:../sqlite + '@modelcontextprotocol/sdk': + specifier: ^1.25.3 + version: 1.30.0(@cfworker/json-schema@4.1.1)(zod@4.4.3) + better-sqlite3: + specifier: ^11.10.0 + version: 11.10.0 + hono: + specifier: ^4.6.14 + version: 4.13.1 + sqlite-vec: + specifier: ^0.1.9 + version: 0.1.9 + zod: + specifier: ^4.3.6 + version: 4.4.3 + devDependencies: + '@melandlabs/ai-rag': + specifier: workspace:* + version: link:../ai/rag + '@melandlabs/config': + specifier: workspace:* + version: link:../config + '@types/better-sqlite3': + specifier: ^7.6.13 + version: 7.6.13 + '@types/node': + specifier: ^22.13.10 + version: 22.20.1 + typescript: + specifier: ^5.6.3 + version: 5.9.3 + vitest: + specifier: ^4.1.10 + version: 4.1.10(@opentelemetry/api@1.9.0)(@types/node@22.20.1)(jsdom@24.1.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(vite@6.4.3(@types/node@22.20.1)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0)) + packages: '@agentclientprotocol/sdk@0.22.1': @@ -9425,6 +9483,14 @@ snapshots: optionalDependencies: vite: 6.4.3(@types/node@26.2.0)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0) + '@vitest/mocker@4.1.10(vite@6.4.3(@types/node@22.20.1)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0))': + dependencies: + '@vitest/spy': 4.1.10 + estree-walker: 3.0.3 + magic-string: 0.30.21 + optionalDependencies: + vite: 6.4.3(@types/node@22.20.1)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0) + '@vitest/mocker@4.1.10(vite@6.4.3(@types/node@26.2.0)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0))': dependencies: '@vitest/spy': 4.1.10 @@ -12964,6 +13030,35 @@ snapshots: - tsx - yaml + vitest@4.1.10(@opentelemetry/api@1.9.0)(@types/node@22.20.1)(jsdom@24.1.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(vite@6.4.3(@types/node@22.20.1)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0)): + dependencies: + '@vitest/expect': 4.1.10 + '@vitest/mocker': 4.1.10(vite@6.4.3(@types/node@22.20.1)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0)) + '@vitest/pretty-format': 4.1.10 + '@vitest/runner': 4.1.10 + '@vitest/snapshot': 4.1.10 + '@vitest/spy': 4.1.10 + '@vitest/utils': 4.1.10 + es-module-lexer: 2.3.1 + expect-type: 1.4.0 + magic-string: 0.30.21 + obug: 2.1.4 + pathe: 2.0.3 + picomatch: 4.0.5 + std-env: 4.2.0 + tinybench: 2.9.0 + tinyexec: 1.3.0 + tinyglobby: 0.2.17 + tinyrainbow: 3.1.1 + vite: 6.4.3(@types/node@22.20.1)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0) + why-is-node-running: 2.3.0 + optionalDependencies: + '@opentelemetry/api': 1.9.0 + '@types/node': 22.20.1 + jsdom: 24.1.3(bufferutil@4.1.0)(utf-8-validate@5.0.10) + transitivePeerDependencies: + - msw + vitest@4.1.10(@opentelemetry/api@1.9.0)(@types/node@26.2.0)(jsdom@24.1.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(vite@6.4.3(@types/node@26.2.0)(jiti@2.7.0)(tsx@4.23.12)(yaml@2.9.0)): dependencies: '@vitest/expect': 4.1.10 From ee3bda2d8a391047f96d27c332f0728d9f94e116 Mon Sep 17 00:00:00 2001 From: Peefy Date: Fri, 11 Sep 2026 20:45:40 +0800 Subject: [PATCH 2/8] fix(workspace): drop dead helpers, rename interface, add changeset MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI lint + format + changeset were failing on the workspace package: - biome: removed unused row→resource helper functions (toWorkspaceResourceVersion / toWorkspaceChunk / toWorkspaceReferenceEdge) and their backing interfaces (WorkspaceResourceVersionRow / WorkspaceReferenceEdgeRow) — they were dead code from when the file was first scaffolded. - biome: dropped the unused pickFrontMatterType helper in okf-backend.ts and the unused scheduleBackgroundClose stub in cli.ts. - biome: applied safe + unsafe fixes for useTemplate / noUnusedTemplateLiteral / noUnsafeDeclarationMerging across the workspace package. Renamed the SqliteWorkspaceStore interface to ISqliteWorkspaceStore to clear the unsafe-declaration-merging error (the class now `implements ISqliteWorkspaceStore`). - opencontext/src/cli/opencontext.ts: reordered the workspace import so biome's organizeImports is happy. - Added .changeset/workspace-folder-indexing.md bumping @melandlabs/workspace (minor) and @melandlabs/opencontext (minor). Verified locally: - pnpm -w build clean - pnpm -w format:check clean - pnpm -r lint clean (workspace + opencontext) - workspace 16/16, opencontext 109/109, memory-store 208/208, okf 140/140 Co-Authored-By: Claude Opus 4.6 --- .changeset/workspace-folder-indexing.md | 8 + examples/src/index.ts | 5 +- examples/src/simple/22-workspace.ts | 26 +-- packages/opencontext/src/cli/opencontext.ts | 8 +- packages/workspace/package.json | 13 +- packages/workspace/src/api.ts | 33 +-- packages/workspace/src/cli.ts | 66 ++---- packages/workspace/src/embedding-provider.ts | 16 +- packages/workspace/src/embedding-queue.ts | 21 +- packages/workspace/src/okf-backend.ts | 20 +- packages/workspace/src/parsers-adapter.ts | 2 +- packages/workspace/src/search/cross-file.ts | 2 +- packages/workspace/src/search/hybrid.ts | 2 +- packages/workspace/src/search/lexical.ts | 2 +- packages/workspace/src/search/semantic.ts | 7 +- packages/workspace/src/sqlite.ts | 189 ++++++------------ packages/workspace/test/okf-backend.test.ts | 6 +- .../workspace/test/parsers-adapter.test.ts | 4 +- packages/workspace/test/search.test.ts | 23 ++- .../test/sqlite-project-store.test.ts | 18 +- 20 files changed, 173 insertions(+), 298 deletions(-) create mode 100644 .changeset/workspace-folder-indexing.md diff --git a/.changeset/workspace-folder-indexing.md b/.changeset/workspace-folder-indexing.md new file mode 100644 index 00000000..630c0c63 --- /dev/null +++ b/.changeset/workspace-folder-indexing.md @@ -0,0 +1,8 @@ +--- +"@melandlabs/workspace": minor +"@melandlabs/opencontext": minor +--- + +Add `@melandlabs/workspace` — a CLI for indexing an OKF / Markdown folder into SQLite and querying it with lexical, semantic, hybrid, and cross-file strategies. Reuses the existing `~/.opencontext/memory/store.db` schema and exposes `opencontext workspace update|search|list` subcommands. + +The workspace CLI defaults `EMBEDDING_PROVIDER=local` (`Xenova/all-MiniLM-L6-v2`, 384 dims) so demos and OKF review workflows run offline without `OPENROUTER_API_KEY`. Multi-format parsing covers `.md`, `.markdown`, `.txt`, `.pdf`, `.docx`, and `.pages`. Cross-file strategy walks `cites` edges extracted from Markdown links for BFS-style expansion across related files. diff --git a/examples/src/index.ts b/examples/src/index.ts index f39f6d27..f567f400 100644 --- a/examples/src/index.ts +++ b/examples/src/index.ts @@ -162,7 +162,10 @@ const demos: Array<[string, () => Promise]> = [ ["demo: memory-store — Vector Symbolic Architecture (VSA) verb", demoVsa], ["demo: okf — OKF v0.2 (Open Knowledge Format) importer / exporter", demoOkf], ["demo: okf — serve (live + frozen viewer)", demoOkfServe], - ["demo: workspace — folder indexing + cross-file hybrid search (local embeddings by default)", demoWorkspace], + [ + "demo: workspace — folder indexing + cross-file hybrid search (local embeddings by default)", + demoWorkspace, + ], ["demo: use-case — personal memory assistant", demoPersonalMemoryAssistant], ["demo: use-case — customer support agent", demoCustomerSupportAgent], ["demo: use-case — research knowledge tracker", demoResearchKnowledgeTracker], diff --git a/examples/src/simple/22-workspace.ts b/examples/src/simple/22-workspace.ts index 73b75812..a18ce4fe 100644 --- a/examples/src/simple/22-workspace.ts +++ b/examples/src/simple/22-workspace.ts @@ -34,10 +34,10 @@ import { updateWorkspaceContext, } from "@melandlabs/opencontext"; import type { - RuntimeContext, - SearchWorkspaceContextResult, - UpdateWorkspaceContextResult, - WorkspaceSearchHit, + RuntimeContext, + SearchWorkspaceContextResult, + UpdateWorkspaceContextResult, + WorkspaceSearchHit, } from "@melandlabs/opencontext"; import { info, makeCheckWithSkip, runSection, withTmp } from "../_helpers.ts"; @@ -58,7 +58,7 @@ function makeRuntimeContext(): RuntimeContext { employee_id: "demo-employee", session_id: "demo-session", request_id: randomUUID(), -}; + }; } async function buildFixture(dir: string): Promise { @@ -157,11 +157,7 @@ export default async function demoWorkspace() { update.filesAdded >= 3, `filesAdded=${update.filesAdded}`, ); - check( - "updateWorkspaceContext returns a positive jobId", - update.jobId > 0, - `jobId=${update.jobId}`, - ); + check("updateWorkspaceContext returns a positive jobId", update.jobId > 0, `jobId=${update.jobId}`); info( "demo/workspace", `updateWorkspaceContext → jobId=${update.jobId}, status=${update.status}, ` + @@ -175,9 +171,7 @@ export default async function demoWorkspace() { check( "listWorkspaceResources returns ≥ 3 resources for the fixture folder", listed.resources.length >= 3, - `total=${listed.total}, resources=${listed.resources - .map((r) => r.canonical_key) - .join(", ")}`, + `total=${listed.total}, resources=${listed.resources.map((r) => r.canonical_key).join(", ")}`, ); // 3. Lexical search — works synchronously, FTS5 was filled in @@ -282,9 +276,7 @@ export default async function demoWorkspace() { titles.has("a") && titles.has("b"), `titles=${[...titles].join(", ")}`, ); - const hitsWithEdges = crossFileResult.hits.filter( - (h) => h.reference_edges.length > 0, - ); + const hitsWithEdges = crossFileResult.hits.filter((h) => h.reference_edges.length > 0); check( "at least one cross-file hit carries reference_edges (cites graph)", hitsWithEdges.length >= 1, @@ -321,4 +313,4 @@ export default async function demoWorkspace() { restoreEnv(); }); }); -} \ No newline at end of file +} diff --git a/packages/opencontext/src/cli/opencontext.ts b/packages/opencontext/src/cli/opencontext.ts index 5068c677..775b6216 100644 --- a/packages/opencontext/src/cli/opencontext.ts +++ b/packages/opencontext/src/cli/opencontext.ts @@ -33,6 +33,10 @@ import { } from "@melandlabs/memory-store/cli-shared"; import { parseOkfArgs, printOkfHelp, startOkf } from "@melandlabs/okf"; import { closeSQLiteVsaStore } from "@melandlabs/sqlite"; +// Workspace CLI is shipped as an optional subpath import so a host +// that doesn't install `@melandlabs/workspace` still gets a usable +// `opencontext` CLI without crashing the bootstrap. +import { runWorkspaceCli } from "@melandlabs/workspace/cli"; import { startHttpServer, startMcpServer } from "../index.js"; import { parseAddArgs, runAdd } from "./add.js"; import { parseDeprecateArgs, runDeprecate } from "./deprecate.js"; @@ -40,10 +44,6 @@ import { parseDoctorArgs, runDoctor } from "./doctor.js"; import { parseListArgs, runList } from "./list.js"; import { parseSearchArgs, runSearch } from "./search.js"; import { parseStatsArgs, runStats } from "./stats.js"; -// Workspace CLI is shipped as an optional subpath import so a host -// that doesn't install `@melandlabs/workspace` still gets a usable -// `opencontext` CLI without crashing the bootstrap. -import { runWorkspaceCli } from "@melandlabs/workspace/cli"; interface HttpArgs extends UnifiedArgs { port: number; diff --git a/packages/workspace/package.json b/packages/workspace/package.json index f3b950dc..fa1d3e79 100644 --- a/packages/workspace/package.json +++ b/packages/workspace/package.json @@ -2,11 +2,7 @@ "name": "@melandlabs/workspace", "version": "0.1.0", "type": "module", - "files": [ - "dist", - "README.md", - "LICENSE" - ], + "files": ["dist", "README.md", "LICENSE"], "publishConfig": { "access": "public" }, @@ -37,12 +33,7 @@ }, "license": "Apache-2.0", "description": "OpenContext · workspace module. Provides project-level indexing, hybrid retrieval, and cross-file reference edges over a project workspace.", - "keywords": [ - "opencontext", - "project-context", - "hybrid-search", - "rag" - ], + "keywords": ["opencontext", "project-context", "hybrid-search", "rag"], "dependencies": { "@melandlabs/contracts": "workspace:*", "@melandlabs/env-config": "workspace:*", diff --git a/packages/workspace/src/api.ts b/packages/workspace/src/api.ts index 0d8a1e77..0c68df07 100644 --- a/packages/workspace/src/api.ts +++ b/packages/workspace/src/api.ts @@ -12,6 +12,13 @@ * `SqliteWorkspaceStore` and a mock embedding function. */ +import { workspaceEmbedQuery } from "./embedding-provider"; +import { indexOkfFolder } from "./okf-backend"; +import { searchCrossFile } from "./search/cross-file"; +import { fuseHybridHits } from "./search/hybrid"; +import { searchLexical } from "./search/lexical"; +import { searchSemantic } from "./search/semantic"; +import type { SqliteWorkspaceStore } from "./sqlite"; import type { ListWorkspaceResourcesInput, ListWorkspaceResourcesResult, @@ -21,13 +28,6 @@ import type { UpdateWorkspaceContextInput, UpdateWorkspaceContextResult, } from "./types"; -import { indexOkfFolder } from "./okf-backend"; -import { searchLexical } from "./search/lexical"; -import { searchSemantic } from "./search/semantic"; -import { searchCrossFile } from "./search/cross-file"; -import { fuseHybridHits } from "./search/hybrid"; -import type { SqliteWorkspaceStore } from "./sqlite"; -import { workspaceEmbedQuery } from "./embedding-provider"; /** * Only `okf_folder` is supported as a source today. Any other value @@ -103,15 +103,16 @@ export async function searchWorkspaceContext( }); // Fallback to lexical when no embeddings are written yet so the // user still gets a hit during the indexing warm-up window. - const finalHits = hits.length > 0 - ? hits - : searchLexical(store, { - workspace_id: input.workspace_id, - user_id: ctx_rt.user_id, - query: input.query, - resource_types: resourceTypes, - limit, - }); + const finalHits = + hits.length > 0 + ? hits + : searchLexical(store, { + workspace_id: input.workspace_id, + user_id: ctx_rt.user_id, + query: input.query, + resource_types: resourceTypes, + limit, + }); return { query: input.query, strategy, total: finalHits.length, hits: finalHits }; } if (strategy === "cross-file") { diff --git a/packages/workspace/src/cli.ts b/packages/workspace/src/cli.ts index 18d45718..5758b66c 100644 --- a/packages/workspace/src/cli.ts +++ b/packages/workspace/src/cli.ts @@ -18,18 +18,10 @@ import { randomUUID } from "node:crypto"; -import type { RuntimeContext, WorkspaceSearchStrategy } from "./types"; -import { - getSQLiteWorkspaceStore, - closeSQLiteWorkspaceStore, - resolveWorkspaceDbPath, -} from "./sqlite"; -import { - updateWorkspaceContext, - searchWorkspaceContext, - listWorkspaceResources, -} from "./api"; +import { listWorkspaceResources, searchWorkspaceContext, updateWorkspaceContext } from "./api"; import { createEmbeddingQueue } from "./embedding-queue"; +import { closeSQLiteWorkspaceStore, getSQLiteWorkspaceStore, resolveWorkspaceDbPath } from "./sqlite"; +import type { RuntimeContext, WorkspaceSearchStrategy } from "./types"; const logPrefix = "[opencontext/workspace]"; @@ -39,11 +31,7 @@ const INDEX_STATUSES = ["pending", "partial", "ready", "failed"] as const; // Flags that don't take a value (booleans). Listed once here so the // generic parseFlags() helper can skip its `next.startsWith("--")` // guard for them. -const BOOLEAN_FLAGS = new Set([ - "--await-embeddings", - "--no-await-embeddings", - "--json", -]); +const BOOLEAN_FLAGS = new Set(["--await-embeddings", "--no-await-embeddings", "--json"]); // ──────────────────────────────────────────────────────────────────────────── // Argv helpers @@ -67,10 +55,7 @@ function takeValue(argv: string[], i: number, flag: string): { value: string; ne return { value: next, next: i + 2 }; } -function parseFlags( - argv: string[], - options: ParseOptions = {}, -): T { +function parseFlags(argv: string[], options: ParseOptions = {}): T { const out = {} as T; for (let i = 0; i < argv.length; i += 1) { const arg = argv[i]; @@ -171,10 +156,7 @@ async function runUpdate(args: UpdateArgs): Promise { // out with --no-await-embeddings (handled by parseFlags: any // `--no-X` is captured as `awaitEmbeddings: false`). const timeoutMs = args.drainTimeoutMs ?? 120_000; - await Promise.race([ - queue.drain(), - new Promise((resolve) => setTimeout(resolve, timeoutMs)), - ]); + await Promise.race([queue.drain(), new Promise((resolve) => setTimeout(resolve, timeoutMs))]); } const out = { @@ -245,7 +227,10 @@ async function runSearch(args: SearchArgs): Promise { throw new ArgvError(`--threshold must be in [0, 1] (got "${args.threshold}")`); } const resourceTypes = args.resourceType - ? args.resourceType.split(",").map((s) => s.trim()).filter(Boolean) + ? args.resourceType + .split(",") + .map((s) => s.trim()) + .filter(Boolean) : undefined; const hops = args.hops !== undefined ? Number.parseInt(args.hops, 10) : undefined; if (hops !== undefined && hops !== 1 && hops !== 2) { @@ -327,12 +312,7 @@ async function runList(args: ListArgs): Promise { const result = await listWorkspaceResources(ctx_rt, store, { workspace_id: workspaceId, resource_type: args.resourceType, - index_status: args.indexStatus as - | "pending" - | "partial" - | "ready" - | "failed" - | undefined, + index_status: args.indexStatus as "pending" | "partial" | "ready" | "failed" | undefined, limit, offset, }); @@ -457,26 +437,6 @@ Example: // Entry // ──────────────────────────────────────────────────────────────────────────── -/** - * Tear down the SQLite handle in the background so a `sqlite-vec` mutex - * destructor (which can `SIGABRT` on certain platforms) doesn't poison - * the main process exit. We: - * - * 1. Detach the close promise so no `await` ever blocks on it. - * 2. Schedule `process.exit` to run AFTER stdout has drained (so the - * human / JSON output is actually visible) and BEFORE the OS - * reaps the still-pending sqlite-vec native mutex teardown. - * 3. Set a hard 250ms timeout so a stuck close doesn't hang the CLI. - */ -function scheduleBackgroundClose(): void { - // Detach the SQLite teardown — we never want `await` on it inside the - // hot path because `sqlite-vec`'s native destructor occasionally - // raises SIGABRT during process teardown. - closeSQLiteWorkspaceStore().catch(() => { - // Best-effort cleanup; ignore secondary errors. - }); -} - async function main(): Promise { const argv = process.argv.slice(2); const sub = argv[0]; @@ -546,7 +506,7 @@ process.on("SIGPIPE", () => { if (isDirectInvocation) { main().catch((error: unknown) => { - const message = error instanceof Error ? error.stack ?? error.message : String(error); + const message = error instanceof Error ? (error.stack ?? error.message) : String(error); process.stderr.write(`${logPrefix} fatal: ${message}\n`); process.exit(1); }); @@ -590,7 +550,7 @@ export async function runWorkspaceCli(argv: string[]): Promise { process.stderr.write(`${error.message}\n`); return 2; } - const message = error instanceof Error ? error.stack ?? error.message : String(error); + const message = error instanceof Error ? (error.stack ?? error.message) : String(error); process.stderr.write(`${logPrefix} fatal: ${message}\n`); return 1; } finally { diff --git a/packages/workspace/src/embedding-provider.ts b/packages/workspace/src/embedding-provider.ts index 52ea3f6a..7669fe74 100644 --- a/packages/workspace/src/embedding-provider.ts +++ b/packages/workspace/src/embedding-provider.ts @@ -60,17 +60,12 @@ export async function getWorkspaceEmbeddingProvider(): Promise { return out; } -function pickFrontMatterType(fm: OkfFrontMatter | undefined): string { - if (!fm) return "document"; - const type = typeof fm.type === "string" ? fm.type : "document"; - return type; -} - function resourceTypeForExtension(ext: string): string { switch (ext) { case ".md": @@ -110,7 +103,12 @@ export async function listOkfFolderResources(dir: string): Promise Promise }, + input: { + workspace_id: string; + user_id: string; + path: string; + enqueueEmbedding: (input: { resource_id: number; version_id: number; jobId?: number }) => Promise; + }, ): Promise { const resources = await listOkfFolderResources(input.path); const job = store.createJob({ workspace_id: input.workspace_id, kind: "index", total: resources.length }); diff --git a/packages/workspace/src/parsers-adapter.ts b/packages/workspace/src/parsers-adapter.ts index 33217a0c..23ad43db 100644 --- a/packages/workspace/src/parsers-adapter.ts +++ b/packages/workspace/src/parsers-adapter.ts @@ -15,8 +15,8 @@ import { readFile } from "node:fs/promises"; import { extname } from "node:path"; -import { estimateTokens } from "@melandlabs/shared"; import { parseFile, parseFileToDocument } from "@melandlabs/rag"; +import { estimateTokens } from "@melandlabs/shared"; let _parsersConfigured = false; diff --git a/packages/workspace/src/search/cross-file.ts b/packages/workspace/src/search/cross-file.ts index 76f8bfc1..e4a6058d 100644 --- a/packages/workspace/src/search/cross-file.ts +++ b/packages/workspace/src/search/cross-file.ts @@ -9,8 +9,8 @@ * heavily linked resources surface higher. */ -import type { WorkspaceSearchHit } from "../types"; import type { SqliteWorkspaceStore } from "../sqlite"; +import type { WorkspaceSearchHit } from "../types"; import { fuseHybridHits } from "./hybrid"; export interface CrossFileSearchInput { diff --git a/packages/workspace/src/search/hybrid.ts b/packages/workspace/src/search/hybrid.ts index 4aae3ad2..3ed6606c 100644 --- a/packages/workspace/src/search/hybrid.ts +++ b/packages/workspace/src/search/hybrid.ts @@ -6,7 +6,7 @@ * `VectorSearchResult` shape that the RAG fusion helper expects. */ -import { fuseHybridResults, type VectorSearchResult } from "@melandlabs/rag"; +import { type VectorSearchResult, fuseHybridResults } from "@melandlabs/rag"; import type { WorkspaceSearchHit } from "../types"; function toVectorResult(hit: WorkspaceSearchHit): VectorSearchResult { diff --git a/packages/workspace/src/search/lexical.ts b/packages/workspace/src/search/lexical.ts index c44d1bc8..67da1f47 100644 --- a/packages/workspace/src/search/lexical.ts +++ b/packages/workspace/src/search/lexical.ts @@ -8,8 +8,8 @@ * queue drains. */ -import type { WorkspaceSearchHit } from "../types"; import type { SqliteWorkspaceStore } from "../sqlite"; +import type { WorkspaceSearchHit } from "../types"; export interface LexicalSearchInput { workspace_id: string; diff --git a/packages/workspace/src/search/semantic.ts b/packages/workspace/src/search/semantic.ts index 002aa63a..32483cd2 100644 --- a/packages/workspace/src/search/semantic.ts +++ b/packages/workspace/src/search/semantic.ts @@ -6,8 +6,8 @@ * "no embeddings yet → empty result" fallback. */ -import type { WorkspaceSearchHit } from "../types"; import type { SqliteWorkspaceStore } from "../sqlite"; +import type { WorkspaceSearchHit } from "../types"; export interface SemanticSearchInput { workspace_id: string; @@ -18,6 +18,9 @@ export interface SemanticSearchInput { threshold: number; } -export function searchSemantic(store: SqliteWorkspaceStore, input: SemanticSearchInput): WorkspaceSearchHit[] { +export function searchSemantic( + store: SqliteWorkspaceStore, + input: SemanticSearchInput, +): WorkspaceSearchHit[] { return store.searchSemantic(input); } diff --git a/packages/workspace/src/sqlite.ts b/packages/workspace/src/sqlite.ts index 30aed139..ca2e6842 100644 --- a/packages/workspace/src/sqlite.ts +++ b/packages/workspace/src/sqlite.ts @@ -24,33 +24,35 @@ * vec0 row + `index_status` flip in subsequent transactions. */ +import { createHash } from "node:crypto"; import { mkdirSync } from "node:fs"; import { dirname } from "node:path"; -import { createHash } from "node:crypto"; -import { chunkTextByEstimatedTokens, RAW_MESSAGE_CHUNK_MAX_TOKENS, RAW_MESSAGE_CHUNK_OVERLAP_TOKENS } from "@melandlabs/shared"; import { getOpenContextPath } from "@melandlabs/env-config"; +import { + RAW_MESSAGE_CHUNK_MAX_TOKENS, + RAW_MESSAGE_CHUNK_OVERLAP_TOKENS, + chunkTextByEstimatedTokens, +} from "@melandlabs/shared"; +import { floatArrayToBuffer } from "@melandlabs/sqlite"; import Database from "better-sqlite3"; import * as sqliteVec from "sqlite-vec"; -import { bufferToFloatArray, floatArrayToBuffer } from "@melandlabs/sqlite"; +import { initializeWorkspaceSchema } from "./schema"; import type { ListWorkspaceResourcesInput, ListWorkspaceResourcesResult, OkfFolderResource, - WorkspaceChunk, + SearchWorkspaceContextInput, + SearchWorkspaceContextResult, + UpdateWorkspaceContextInput, + UpdateWorkspaceContextResult, WorkspaceEdgeType, WorkspaceIndexStatus, WorkspaceJob, - WorkspaceReferenceEdge, WorkspaceResource, WorkspaceResourceVersion, WorkspaceSearchHit, WorkspaceSearchStrategy, - SearchWorkspaceContextInput, - SearchWorkspaceContextResult, - UpdateWorkspaceContextInput, - UpdateWorkspaceContextResult, } from "./types"; -import { initializeWorkspaceSchema } from "./schema"; type DatabaseLike = Database.Database; @@ -69,19 +71,6 @@ interface WorkspaceResourceRow { metadata: string | null; } -interface WorkspaceResourceVersionRow { - id: number; - resource_id: number; - version_number: number; - sha256: string; - change_kind: string; - size_bytes: number; - parent_version_id: number | null; - source_path: string | null; - created_at: number; - metadata: string | null; -} - interface WorkspaceChunkRow { id: number; chunk_id: string; @@ -100,18 +89,6 @@ interface WorkspaceChunkRow { embedding_updated_at: number | null; } -interface WorkspaceReferenceEdgeRow { - id: number; - workspace_id: string; - source_resource_id: number; - source_version_id: number | null; - target_resource_id: number; - target_version_id: number | null; - edge_type: string; - quote: string | null; - created_at: number; -} - interface WorkspaceJobRow { id: number; workspace_id: string; @@ -141,55 +118,6 @@ function toWorkspaceResource(row: WorkspaceResourceRow): WorkspaceResource { }; } -function toWorkspaceResourceVersion(row: WorkspaceResourceVersionRow): WorkspaceResourceVersion { - return { - id: row.id, - resource_id: row.resource_id, - version_number: row.version_number, - sha256: row.sha256, - change_kind: row.change_kind as WorkspaceResourceVersion["change_kind"], - size_bytes: row.size_bytes, - parent_version_id: row.parent_version_id, - source_path: row.source_path, - created_at: row.created_at, - metadata: parseJson>(row.metadata, {} as Record), - }; -} - -function toWorkspaceChunk(row: WorkspaceChunkRow): WorkspaceChunk { - return { - id: row.id, - chunk_id: row.chunk_id, - resource_id: row.resource_id, - version_id: row.version_id, - workspace_id: row.workspace_id, - chunk_index: row.chunk_index, - chunk_count: row.chunk_count, - start_position: row.start_position, - end_position: row.end_position, - content: row.content, - content_hash: row.content_hash, - embedding: bufferToFloatArray(row.embedding), - embedding_model: row.embedding_model ?? undefined, - embedding_dimensions: row.embedding_dimensions ?? undefined, - embedding_updated_at: row.embedding_updated_at ?? undefined, - }; -} - -function toWorkspaceReferenceEdge(row: WorkspaceReferenceEdgeRow): WorkspaceReferenceEdge { - return { - id: row.id, - workspace_id: row.workspace_id, - source_resource_id: row.source_resource_id, - source_version_id: row.source_version_id, - target_resource_id: row.target_resource_id, - target_version_id: row.target_version_id, - edge_type: row.edge_type as WorkspaceEdgeType, - quote: row.quote, - created_at: row.created_at, - }; -} - function toWorkspaceJob(row: WorkspaceJobRow): WorkspaceJob { return { id: row.id, @@ -266,7 +194,7 @@ export interface SqliteWorkspaceStoreOptions { * here (rather than re-importing the queue file) avoids a circular type * reference between `embedding-queue.ts` and `sqlite.ts`. */ -export interface SqliteWorkspaceStore { +export interface ISqliteWorkspaceStore { readonly __testDb: DatabaseLike; init(): Promise; close(): Promise; @@ -288,7 +216,11 @@ export interface SqliteWorkspaceStore { workspace_id: string; user_id: string; resource: OkfFolderResource; - }): Promise<{ resource_id: number; version_id: number; change_kind: WorkspaceResourceVersion["change_kind"] }>; + }): Promise<{ + resource_id: number; + version_id: number; + change_kind: WorkspaceResourceVersion["change_kind"]; + }>; upsertReferenceEdges(input: { workspace_id: string; @@ -333,7 +265,10 @@ export interface SqliteWorkspaceStore { limit: number; }): WorkspaceSearchHit[]; - findResourceByCanonicalKey(input: { workspace_id: string; canonical_key: string }): WorkspaceResource | null; + findResourceByCanonicalKey(input: { + workspace_id: string; + canonical_key: string; + }): WorkspaceResource | null; createJob(input: { workspace_id: string; kind: WorkspaceJob["kind"]; total: number }): WorkspaceJob; updateJobTotal(jobId: number, total: number): void; @@ -345,7 +280,7 @@ export interface SqliteWorkspaceStore { * `search pipeline`, `cross-file expansion`) to keep the file readable * when each section grows in later phases. */ -export class SqliteWorkspaceStore implements SqliteWorkspaceStore { +export class SqliteWorkspaceStore implements ISqliteWorkspaceStore { readonly __testDb!: DatabaseLike; private readonly db: DatabaseLike; private readonly ownsConnection: boolean; @@ -394,7 +329,7 @@ export class SqliteWorkspaceStore implements SqliteWorkspaceStore { this.initialized = true; } - async close(): Promise { + async close(): Promise { // Best-effort: drop the singleton reference and let the OS reap the // native handle on process exit. `db.close()` is intentionally NOT // called here because `sqlite-vec`'s native destructor occasionally @@ -485,9 +420,7 @@ export class SqliteWorkspaceStore implements SqliteWorkspaceStore { WHERE chunk_id = ?`, ); const insertVecStmt = writeVec - ? this.db.prepare( - `INSERT OR REPLACE INTO ${dimensionsTable}(embedding, chunk_id) VALUES (?, ?)`, - ) + ? this.db.prepare(`INSERT OR REPLACE INTO ${dimensionsTable}(embedding, chunk_id) VALUES (?, ?)`) : null; const now = currentUnixSeconds(); for (const entry of entries) { @@ -504,17 +437,13 @@ export class SqliteWorkspaceStore implements SqliteWorkspaceStore { markVersionEmbeddingReady(resourceId: number, versionId: number): void { const stillMissing = this.db - .prepare(`SELECT 1 FROM workspace_chunks WHERE version_id = ? AND embedding IS NULL LIMIT 1`) + .prepare("SELECT 1 FROM workspace_chunks WHERE version_id = ? AND embedding IS NULL LIMIT 1") .get(versionId); if (stillMissing) { - this.db - .prepare(`UPDATE workspace_resources SET index_status = 'partial' WHERE id = ?`) - .run(resourceId); + this.db.prepare(`UPDATE workspace_resources SET index_status = 'partial' WHERE id = ?`).run(resourceId); return; } - this.db - .prepare(`UPDATE workspace_resources SET index_status = 'ready' WHERE id = ?`) - .run(resourceId); + this.db.prepare(`UPDATE workspace_resources SET index_status = 'ready' WHERE id = ?`).run(resourceId); } markVersionEmbeddingPartial(resourceId: number, versionId: number, errorMessage: string): void { @@ -524,9 +453,7 @@ export class SqliteWorkspaceStore implements SqliteWorkspaceStore { // resource is half-indexed. void versionId; void errorMessage; - this.db - .prepare(`UPDATE workspace_resources SET index_status = 'partial' WHERE id = ?`) - .run(resourceId); + this.db.prepare(`UPDATE workspace_resources SET index_status = 'partial' WHERE id = ?`).run(resourceId); } markVersionEmbeddingFailed(resourceId: number, versionId: number, errorMessage: string): void { @@ -534,9 +461,7 @@ export class SqliteWorkspaceStore implements SqliteWorkspaceStore { // will surface the error string on the resource / version row. void versionId; void errorMessage; - this.db - .prepare(`UPDATE workspace_resources SET index_status = 'failed' WHERE id = ?`) - .run(resourceId); + this.db.prepare(`UPDATE workspace_resources SET index_status = 'failed' WHERE id = ?`).run(resourceId); } markJobFailed(jobId: number | null, errorMessage: string): void { @@ -560,7 +485,11 @@ export class SqliteWorkspaceStore implements SqliteWorkspaceStore { workspace_id: string; user_id: string; resource: OkfFolderResource; - }): Promise<{ resource_id: number; version_id: number; change_kind: WorkspaceResourceVersion["change_kind"] }> { + }): Promise<{ + resource_id: number; + version_id: number; + change_kind: WorkspaceResourceVersion["change_kind"]; + }> { await this.init(); const { workspace_id, user_id, resource } = input; const now = currentUnixSeconds(); @@ -573,7 +502,9 @@ export class SqliteWorkspaceStore implements SqliteWorkspaceStore { FROM workspace_resources WHERE workspace_id = ? AND canonical_key = ?`, ) - .get(workspace_id, resource.canonical_key) as { id: number; current_version_id: number | null } | undefined; + .get(workspace_id, resource.canonical_key) as + | { id: number; current_version_id: number | null } + | undefined; let resourceId: number; let currentVersionId: number | null = existing?.current_version_id ?? null; @@ -604,9 +535,7 @@ export class SqliteWorkspaceStore implements SqliteWorkspaceStore { resourceId = existing.id; if (currentVersionId !== null) { const prevVersion = this.db - .prepare( - `SELECT sha256 FROM workspace_resource_versions WHERE id = ?`, - ) + .prepare("SELECT sha256 FROM workspace_resource_versions WHERE id = ?") .get(currentVersionId) as { sha256: string } | undefined; if (prevVersion?.sha256 === contentHash) { changeKind = "unchanged"; @@ -614,7 +543,7 @@ export class SqliteWorkspaceStore implements SqliteWorkspaceStore { // re-indexing — callers may want to detect staleness // via `updated_at` vs. `current_version_id.created_at`. this.db - .prepare(`UPDATE workspace_resources SET updated_at = ? WHERE id = ?`) + .prepare("UPDATE workspace_resources SET updated_at = ? WHERE id = ?") .run(now, resourceId); return { resource_id: resourceId, version_id: currentVersionId, change_kind: changeKind }; } @@ -651,7 +580,7 @@ export class SqliteWorkspaceStore implements SqliteWorkspaceStore { // Delete any old chunks for this resource; FTS5 mirror is // kept in sync via the AI/AD/AU triggers. - this.db.prepare(`DELETE FROM workspace_chunks WHERE resource_id = ?`).run(resourceId); + this.db.prepare("DELETE FROM workspace_chunks WHERE resource_id = ?").run(resourceId); const pieces = chunkTextByEstimatedTokens(resource.body, { maxTokens: RAW_MESSAGE_CHUNK_MAX_TOKENS, @@ -774,9 +703,12 @@ export class SqliteWorkspaceStore implements SqliteWorkspaceStore { }; } - findResourceByCanonicalKey(input: { workspace_id: string; canonical_key: string }): WorkspaceResource | null { + findResourceByCanonicalKey(input: { + workspace_id: string; + canonical_key: string; + }): WorkspaceResource | null { const row = this.db - .prepare(`SELECT * FROM workspace_resources WHERE workspace_id = ? AND canonical_key = ?`) + .prepare("SELECT * FROM workspace_resources WHERE workspace_id = ? AND canonical_key = ?") .get(input.workspace_id, input.canonical_key) as WorkspaceResourceRow | undefined; return row ? toWorkspaceResource(row) : null; } @@ -952,9 +884,7 @@ export class SqliteWorkspaceStore implements SqliteWorkspaceStore { private childVectorTableExists(name: string): boolean { if (!this.vectorSearchAvailable) return false; return Boolean( - this.db - .prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?") - .get(name), + this.db.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?").get(name), ); } @@ -1081,7 +1011,7 @@ export class SqliteWorkspaceStore implements SqliteWorkspaceStore { presentKeys: Set; }): Array<{ canonical_key: string }> { const allRows = this.db - .prepare(`SELECT id, canonical_key, metadata FROM workspace_resources WHERE workspace_id = ?`) + .prepare("SELECT id, canonical_key, metadata FROM workspace_resources WHERE workspace_id = ?") .all(input.workspace_id) as Array<{ id: number; canonical_key: string; metadata: string | null }>; const missing: Array<{ canonical_key: string }> = []; const now = currentUnixSeconds(); @@ -1091,7 +1021,7 @@ export class SqliteWorkspaceStore implements SqliteWorkspaceStore { if (meta.deleted_at) continue; meta.deleted_at = now; this.db - .prepare(`UPDATE workspace_resources SET metadata = ?, updated_at = ? WHERE id = ?`) + .prepare("UPDATE workspace_resources SET metadata = ?, updated_at = ? WHERE id = ?") .run(stringifyJson(meta), now, row.id); missing.push({ canonical_key: row.canonical_key }); } @@ -1111,14 +1041,14 @@ export class SqliteWorkspaceStore implements SqliteWorkspaceStore { ) .run(input.workspace_id, input.kind, input.total, now, now); const row = this.db - .prepare(`SELECT * FROM workspace_jobs WHERE id = ?`) + .prepare("SELECT * FROM workspace_jobs WHERE id = ?") .get(Number(stmt.lastInsertRowid)) as WorkspaceJobRow; return toWorkspaceJob(row); } updateJobTotal(jobId: number, total: number): void { this.db - .prepare(`UPDATE workspace_jobs SET total = ?, updated_at = ? WHERE id = ?`) + .prepare("UPDATE workspace_jobs SET total = ?, updated_at = ? WHERE id = ?") .run(total, currentUnixSeconds(), jobId); } } @@ -1210,7 +1140,7 @@ export async function createSqliteWorkspaceStore( // ------------------------------------------------------------------------- export async function runUpdateWorkspaceContext( - store: SqliteWorkspaceStore, + _store: SqliteWorkspaceStore, input: UpdateWorkspaceContextInput & { user_id: string }, hooks: { indexOkfFolder: ( @@ -1224,7 +1154,7 @@ export async function runUpdateWorkspaceContext( } export async function runSearchWorkspaceContext( - store: SqliteWorkspaceStore, + _store: SqliteWorkspaceStore, input: SearchWorkspaceContextInput & { user_id: string }, hooks: { searchLexical: ( @@ -1249,7 +1179,11 @@ export async function runSearchWorkspaceContext( limit: number, ) => WorkspaceSearchHit[]; generateEmbedding?: (text: string) => Promise; - fuse?: (lexical: WorkspaceSearchHit[], semantic: WorkspaceSearchHit[], limit: number) => WorkspaceSearchHit[]; + fuse?: ( + lexical: WorkspaceSearchHit[], + semantic: WorkspaceSearchHit[], + limit: number, + ) => WorkspaceSearchHit[]; }, ): Promise { const strategy: WorkspaceSearchStrategy = input.strategy ?? "hybrid"; @@ -1324,12 +1258,7 @@ export async function runSearchWorkspaceContext( ); } const fused = hooks.fuse ? hooks.fuse(lexicalHits, semanticHits, candidateLimit) : lexicalHits; - hits = hooks.expandNeighbors( - input.workspace_id, - fused, - options.hops ?? 1, - limit, - ); + hits = hooks.expandNeighbors(input.workspace_id, fused, options.hops ?? 1, limit); } return { query: input.query, diff --git a/packages/workspace/test/okf-backend.test.ts b/packages/workspace/test/okf-backend.test.ts index cdb1bdc5..4a3cbef2 100644 --- a/packages/workspace/test/okf-backend.test.ts +++ b/packages/workspace/test/okf-backend.test.ts @@ -84,9 +84,7 @@ describe("indexOkfFolder", () => { WHERE edge_type = 'cites'`, ) .all() as Array<{ source: string; target: string; edge_type: string }>; - const contractLaw = edgeRows.find( - (row) => row.source === "contract.md" && row.target === "law.md", - ); + const contractLaw = edgeRows.find((row) => row.source === "contract.md" && row.target === "law.md"); expect(contractLaw).toBeDefined(); await store.close(); }); @@ -114,7 +112,7 @@ describe("indexOkfFolder", () => { }); expect(second.filesDeleted).toBe(1); const rows = store.__testDb - .prepare(`SELECT canonical_key, metadata FROM workspace_resources ORDER BY canonical_key`) + .prepare("SELECT canonical_key, metadata FROM workspace_resources ORDER BY canonical_key") .all() as Array<{ canonical_key: string; metadata: string | null }>; const dropped = rows.find((row) => row.canonical_key === "drop.md"); expect(dropped).toBeDefined(); diff --git a/packages/workspace/test/parsers-adapter.test.ts b/packages/workspace/test/parsers-adapter.test.ts index 53255506..01c30176 100644 --- a/packages/workspace/test/parsers-adapter.test.ts +++ b/packages/workspace/test/parsers-adapter.test.ts @@ -27,7 +27,9 @@ describe("parsers-adapter", () => { expect(detectMimeType("a.markdown")).toBe("text/markdown"); expect(detectMimeType("a.txt")).toBe("text/plain"); expect(detectMimeType("a.pdf")).toBe("application/pdf"); - expect(detectMimeType("a.docx")).toBe("application/vnd.openxmlformats-officedocument.wordprocessingml.document"); + expect(detectMimeType("a.docx")).toBe( + "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + ); expect(detectMimeType("a.pages")).toBe("application/x-iwork-pages-sffpages"); expect(detectMimeType("a.unknown")).toBe("application/octet-stream"); }); diff --git a/packages/workspace/test/search.test.ts b/packages/workspace/test/search.test.ts index 1cf5f827..c4ee5019 100644 --- a/packages/workspace/test/search.test.ts +++ b/packages/workspace/test/search.test.ts @@ -10,10 +10,10 @@ import { join } from "node:path"; import { afterEach, beforeEach, describe, expect, it } from "vitest"; import { searchWorkspaceContext } from "../src/api"; +import { searchCrossFile } from "../src/search/cross-file"; +import { fuseHybridHits } from "../src/search/hybrid"; import { searchLexical } from "../src/search/lexical"; import { searchSemantic } from "../src/search/semantic"; -import { fuseHybridHits } from "../src/search/hybrid"; -import { searchCrossFile } from "../src/search/cross-file"; import { SqliteWorkspaceStore } from "../src/sqlite"; import type { OkfFolderResource, WorkspaceSearchHit } from "../src/types"; @@ -27,7 +27,9 @@ afterEach(() => { rmSync(scratchDir, { recursive: true, force: true }); }); -function makeResource(overrides: Partial & Pick): OkfFolderResource { +function makeResource( + overrides: Partial & Pick, +): OkfFolderResource { return { absolute_path: `/fake/${overrides.canonical_key}`, title: overrides.canonical_key, @@ -59,17 +61,26 @@ async function indexThreeDocuments(store: SqliteWorkspaceStore): Promise { await store.indexResource({ workspace_id: "p1", user_id: "u1", - resource: makeResource({ canonical_key: "Reference/contract.md", body: "limitation of liability clauses must conform to applicable law".trim() }), + resource: makeResource({ + canonical_key: "Reference/contract.md", + body: "limitation of liability clauses must conform to applicable law".trim(), + }), }); await store.indexResource({ workspace_id: "p1", user_id: "u1", - resource: makeResource({ canonical_key: "Reference/law.md", body: "civil code article 123: parties may limit liability unless the law forbids it".trim() }), + resource: makeResource({ + canonical_key: "Reference/law.md", + body: "civil code article 123: parties may limit liability unless the law forbids it".trim(), + }), }); await store.indexResource({ workspace_id: "p1", user_id: "u1", - resource: makeResource({ canonical_key: "Reference/cookie-policy.md", body: "this document describes cookie usage on the marketing site".trim() }), + resource: makeResource({ + canonical_key: "Reference/cookie-policy.md", + body: "this document describes cookie usage on the marketing site".trim(), + }), }); } diff --git a/packages/workspace/test/sqlite-project-store.test.ts b/packages/workspace/test/sqlite-project-store.test.ts index 86648183..93882f73 100644 --- a/packages/workspace/test/sqlite-project-store.test.ts +++ b/packages/workspace/test/sqlite-project-store.test.ts @@ -22,7 +22,9 @@ afterEach(() => { rmSync(scratchDir, { recursive: true, force: true }); }); -function makeResource(overrides: Partial & Pick): OkfFolderResource { +function makeResource( + overrides: Partial & Pick, +): OkfFolderResource { return { absolute_path: `/fake/${overrides.canonical_key}`, title: overrides.canonical_key, @@ -36,8 +38,10 @@ describe("SqliteWorkspaceStore", () => { it("initialises every schema table on init()", async () => { const store = new SqliteWorkspaceStore({ dbPath: join(scratchDir, "store.db") }); await store.init(); - const tables = (store as unknown as { __testDb: { prepare: (s: string) => { all: () => unknown[] } } }) - .__testDb.prepare(`SELECT name FROM sqlite_master WHERE type IN ('table', 'view', 'trigger')`) + const tables = ( + store as unknown as { __testDb: { prepare: (s: string) => { all: () => unknown[] } } } + ).__testDb + .prepare(`SELECT name FROM sqlite_master WHERE type IN ('table', 'view', 'trigger')`) .all() as Array<{ name: string }>; const names = tables.map((row) => row.name); expect(names).toContain("workspace_resources"); @@ -63,7 +67,7 @@ describe("SqliteWorkspaceStore", () => { expect(result.change_kind).toBe("created"); expect(result.resource_id).toBeGreaterThan(0); const chunks = store.__testDb - .prepare(`SELECT chunk_index, chunk_count, content FROM workspace_chunks ORDER BY chunk_index`) + .prepare("SELECT chunk_index, chunk_count, content FROM workspace_chunks ORDER BY chunk_index") .all() as Array<{ chunk_index: number; chunk_count: number; content: string }>; expect(chunks.length).toBeGreaterThan(0); expect(chunks[0]?.chunk_count).toBe(chunks.length); @@ -106,7 +110,9 @@ describe("SqliteWorkspaceStore", () => { expect(second.change_kind).toBe("modified"); expect(second.version_id).not.toBe(first.version_id); const versions = store.__testDb - .prepare(`SELECT id, version_number, parent_version_id FROM workspace_resource_versions ORDER BY version_number`) + .prepare( + "SELECT id, version_number, parent_version_id FROM workspace_resource_versions ORDER BY version_number", + ) .all() as Array<{ id: number; version_number: number; parent_version_id: number | null }>; expect(versions.length).toBe(2); expect(versions[1]?.parent_version_id).toBe(versions[0]?.id ?? null); @@ -132,7 +138,7 @@ describe("SqliteWorkspaceStore", () => { }); expect(missing.map((entry) => entry.canonical_key)).toEqual(["docs/b.md"]); const afterRows = store.__testDb - .prepare(`SELECT canonical_key, metadata FROM workspace_resources ORDER BY canonical_key`) + .prepare("SELECT canonical_key, metadata FROM workspace_resources ORDER BY canonical_key") .all() as Array<{ canonical_key: string; metadata: string | null }>; const deletedRow = afterRows.find((row) => row.canonical_key === "docs/b.md"); expect(deletedRow).toBeDefined(); From 22ed18f247663aeb28fc176196da2ee6a2c6c0ce Mon Sep 17 00:00:00 2001 From: Peefy Date: Fri, 11 Sep 2026 20:54:49 +0800 Subject: [PATCH 3/8] fix(examples): make 22-workspace demo skip when @melandlabs/workspace is absent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The smoke test pulls @melandlabs/* from npmjs.org via `pnpm install --ignore-workspace`. @melandlabs/workspace isn't published yet, so a static `import { ... } from "@melandlabs/opencontext"` of the workspace re-exports throws ERR_MODULE_NOT_FOUND at module load time and breaks the entire smoke suite. Switch the demo to a top-level `import type { ... }` (erased at compile time, no runtime resolution) plus an `await import( "@melandlabs/workspace")` inside the demo body, wrapped in try/catch. If the dynamic import fails we record a single [SKIP] and return, so the rest of the suite keeps running. This also makes the demo robust in the local monorepo before workspace is symlinked into examples/node_modules — the demo will start running for real once workspace is added as a dependency in examples/package.json after the first publish. Verified locally: - examples pnpm test: exit 0, demo/workspace → [SKIP] with reason - 307 OK lines, 0 FAIL - pnpm -r lint clean Co-Authored-By: Claude Opus 4.6 --- examples/src/simple/22-workspace.ts | 81 ++++++++++++++++++++++------- 1 file changed, 63 insertions(+), 18 deletions(-) diff --git a/examples/src/simple/22-workspace.ts b/examples/src/simple/22-workspace.ts index a18ce4fe..e11a30b1 100644 --- a/examples/src/simple/22-workspace.ts +++ b/examples/src/simple/22-workspace.ts @@ -25,14 +25,6 @@ import { writeFile, mkdir } from "node:fs/promises"; import { join } from "node:path"; import { randomUUID } from "node:crypto"; -import { - closeSQLiteWorkspaceStore, - getSQLiteWorkspaceStore, - listWorkspaceResources, - resolveWorkspaceDbPath, - searchWorkspaceContext, - updateWorkspaceContext, -} from "@melandlabs/opencontext"; import type { RuntimeContext, SearchWorkspaceContextResult, @@ -52,6 +44,50 @@ const SETTLE_MS = 15_000; // give the embedding queue time to drain on first run const PREVIOUS_PROVIDER_ENV = process.env.EMBEDDING_PROVIDER; process.env.EMBEDDING_PROVIDER = "local"; +// Workspace value imports are pulled in dynamically so the smoke test +// (which pulls @melandlabs/opencontext from npm without the optional +// @melandlabs/workspace peer) can skip this demo cleanly instead of +// crashing the bootstrap with `ERR_MODULE_NOT_FOUND`. +type WorkspaceModule = { + closeSQLiteWorkspaceStore: () => Promise; + getSQLiteWorkspaceStore: (options: { dbPath: string }) => Promise; + listWorkspaceResources: ( + ctx: RuntimeContext, + store: unknown, + input: { workspace_id: string }, + ) => Promise<{ resources: unknown[]; total: number }>; + resolveWorkspaceDbPath: (override?: string) => string; + searchWorkspaceContext: ( + ctx: RuntimeContext, + store: unknown, + input: { + workspace_id: string; + query: string; + strategy: string; + options?: Record; + }, + ) => Promise; + updateWorkspaceContext: ( + ctx: RuntimeContext, + store: unknown, + input: { workspace_id: string; source: string; path: string }, + ) => Promise; +}; + +let _workspaceCache: WorkspaceModule | undefined; +async function loadWorkspace(): Promise { + if (_workspaceCache) return _workspaceCache; + try { + // @ts-expect-error -- optional workspace subpath; may be absent + // (e.g. in the npm-installed smoke test environment). + const mod = (await import("@melandlabs/workspace")) as WorkspaceModule; + _workspaceCache = mod; + return mod; + } catch { + return null; + } +} + function makeRuntimeContext(): RuntimeContext { return { user_id: USER_ID, @@ -116,6 +152,15 @@ export default async function demoWorkspace() { await runSection("demo: @melandlabs/workspace (folder indexing + cross-file search)", async () => { const { check, skip } = makeCheckWithSkip("demo/workspace"); + const ws = await loadWorkspace(); + if (!ws) { + skip( + "@melandlabs/workspace is installed", + "optional peer dep — npm-installed smoke test environment skips this demo", + ); + return; + } + // Restore whatever was in EMBEDDING_PROVIDER before this demo so // subsequent demos don't see `local` stuck on. const restoreEnv = () => { @@ -131,13 +176,13 @@ export default async function demoWorkspace() { await buildFixture(dir); const dbPath = join(dir, "workspace.db"); - const store = await getSQLiteWorkspaceStore({ dbPath }); + const store = await ws.getSQLiteWorkspaceStore({ dbPath }); const ctx = makeRuntimeContext(); // 1. updateWorkspaceContext — synchronous chunking + async fan-out. let update: UpdateWorkspaceContextResult; try { - update = await updateWorkspaceContext(ctx, store, { + update = await ws.updateWorkspaceContext(ctx, store, { workspace_id: WORKSPACE_ID, source: "okf_folder", path: join(dir, "wiki"), @@ -165,7 +210,7 @@ export default async function demoWorkspace() { ); // 2. listWorkspaceResources — every fixture file appears. - const listed = await listWorkspaceResources(ctx, store, { + const listed = await ws.listWorkspaceResources(ctx, store, { workspace_id: WORKSPACE_ID, }); check( @@ -177,7 +222,7 @@ export default async function demoWorkspace() { // 3. Lexical search — works synchronously, FTS5 was filled in // during the sync chunk phase. This is the only strategy // that is guaranteed to work even before embeddings finish. - const lexicalResult = await searchWorkspaceContext(ctx, store, { + const lexicalResult = await ws.searchWorkspaceContext(ctx, store, { workspace_id: WORKSPACE_ID, query: "limitation", strategy: "lexical", @@ -206,7 +251,7 @@ export default async function demoWorkspace() { let semanticResult: SearchWorkspaceContextResult | undefined; try { - semanticResult = await searchWorkspaceContext(ctx, store, { + semanticResult = await ws.searchWorkspaceContext(ctx, store, { workspace_id: WORKSPACE_ID, query: "what is the cap on liability", strategy: "semantic", @@ -246,7 +291,7 @@ export default async function demoWorkspace() { // hits from both `a.md` and the `b.md` it cites. let crossFileResult: SearchWorkspaceContextResult | undefined; try { - crossFileResult = await searchWorkspaceContext(ctx, store, { + crossFileResult = await ws.searchWorkspaceContext(ctx, store, { workspace_id: WORKSPACE_ID, query: "limitation", strategy: "cross-file", @@ -290,7 +335,7 @@ export default async function demoWorkspace() { } // 6. Re-run update — every file should now be `unchanged`. - const reUpdate = await updateWorkspaceContext(ctx, store, { + const reUpdate = await ws.updateWorkspaceContext(ctx, store, { workspace_id: WORKSPACE_ID, source: "okf_folder", path: join(dir, "wiki"), @@ -305,11 +350,11 @@ export default async function demoWorkspace() { // and the override here is honoured. check( "resolveWorkspaceDbPath returns the path we passed in", - resolveWorkspaceDbPath(dbPath) === dbPath, - resolveWorkspaceDbPath(dbPath), + ws.resolveWorkspaceDbPath(dbPath) === dbPath, + ws.resolveWorkspaceDbPath(dbPath), ); - await closeSQLiteWorkspaceStore().catch(() => undefined); + await ws.closeSQLiteWorkspaceStore().catch(() => undefined); restoreEnv(); }); }); From e25d89217313328d88b5c2344a2eb5b2d91186b1 Mon Sep 17 00:00:00 2001 From: Peefy Date: Fri, 11 Sep 2026 21:00:27 +0800 Subject: [PATCH 4/8] refactor(examples): move workspace demo fixtures onto disk Previously the 22-workspace demo wrote its 3-file contract / statute fixture inline via writeFile at the top of buildFixture(). That kept the markdown content trapped inside the demo file with no way to edit / reuse / diff it independently. Move the fixture to examples/fixtures/workspace-wiki/: - a.md (contract, Limitation of Liability, cites b.md) - b.md (contract, indemnification, cites a.md) - law-clause.md (statute, matches a.md's 12-month cap) - README.md describing the role of each file 22-workspace.ts now resolves the fixture dir relative to its own URL and copyFile's each markdown into a per-run tmp dir, so the fixture is the source of truth and the demo always starts from a clean slate. Verified: - workspace demo passes all 11 [OK] checks when @melandlabs/workspace is symlinked into examples/node_modules (lexical / semantic / cross-file / sha256-dedup re-run) - pnpm format:check clean - pnpm -r lint clean Co-Authored-By: Claude Opus 4.6 --- examples/fixtures/workspace-wiki/README.md | 22 ++++++++ examples/fixtures/workspace-wiki/a.md | 7 +++ examples/fixtures/workspace-wiki/b.md | 7 +++ .../fixtures/workspace-wiki/law-clause.md | 6 ++ examples/src/simple/22-workspace.ts | 55 ++++++------------- 5 files changed, 60 insertions(+), 37 deletions(-) create mode 100644 examples/fixtures/workspace-wiki/README.md create mode 100644 examples/fixtures/workspace-wiki/a.md create mode 100644 examples/fixtures/workspace-wiki/b.md create mode 100644 examples/fixtures/workspace-wiki/law-clause.md diff --git a/examples/fixtures/workspace-wiki/README.md b/examples/fixtures/workspace-wiki/README.md new file mode 100644 index 00000000..0c9a90fe --- /dev/null +++ b/examples/fixtures/workspace-wiki/README.md @@ -0,0 +1,22 @@ +# workspace-wiki fixture + +A 3-file OKF folder used by `examples/src/simple/22-workspace.ts` to demo +`@melandlabs/workspace` end-to-end. The demo copies these files into a tmp +directory before each run so every run starts from a clean slate. + +| File | `resource_type` | Role | +| --- | --- | --- | +| `a.md` | `note` (front-matter `type: contract`) | Limitation of Liability — 12-month cap. Cites `[./b.md]`. | +| `b.md` | `note` (front-matter `type: contract`) | Indemnification carve-out. Cites `[./a.md]`. | +| `law-clause.md` | `note` (front-matter `type: statute`) | Public law clause — matches `a.md`'s 12-month cap. | + +The cross-file cites graph `a → b` (and back) is what the cross-file search +strategy walks. The demo runs lexical, semantic, hybrid, and cross-file +queries against this folder and asserts: + +- 3 resources appear in `listWorkspaceResources` +- lexical search for "limitation" returns ≥ 1 hit +- re-running `updateWorkspaceContext` reports every file as `unchanged` (sha256 dedup) + +Used in: `examples/src/simple/22-workspace.ts` and the workspace tutorial +(`docs/tutorials/use-cases/09-workspace-folder-indexing.md`). diff --git a/examples/fixtures/workspace-wiki/a.md b/examples/fixtures/workspace-wiki/a.md new file mode 100644 index 00000000..23e0adff --- /dev/null +++ b/examples/fixtures/workspace-wiki/a.md @@ -0,0 +1,7 @@ +--- +title: Limitation of Liability +type: contract +created: 2026-09-11 +--- +The aggregate liability of either party shall not exceed the fees paid in the +twelve (12) months preceding the claim. See [Indemnification](./b.md) for carve-outs. diff --git a/examples/fixtures/workspace-wiki/b.md b/examples/fixtures/workspace-wiki/b.md new file mode 100644 index 00000000..b8d21369 --- /dev/null +++ b/examples/fixtures/workspace-wiki/b.md @@ -0,0 +1,7 @@ +--- +title: Indemnification +type: contract +created: 2026-09-11 +--- +Neither party shall indemnify the other for indirect, consequential, or +punitive damages. References: [Limitation of Liability](./a.md). diff --git a/examples/fixtures/workspace-wiki/law-clause.md b/examples/fixtures/workspace-wiki/law-clause.md new file mode 100644 index 00000000..774a237a --- /dev/null +++ b/examples/fixtures/workspace-wiki/law-clause.md @@ -0,0 +1,6 @@ +--- +title: Public Law Clause — Cap of Liability +type: statute +created: 2026-09-11 +--- +Statutory cap of liability is twelve months of fees for ordinary breach. diff --git a/examples/src/simple/22-workspace.ts b/examples/src/simple/22-workspace.ts index e11a30b1..df353041 100644 --- a/examples/src/simple/22-workspace.ts +++ b/examples/src/simple/22-workspace.ts @@ -22,8 +22,9 @@ * the printed JSON envelope mirrors what the CLI prints. */ -import { writeFile, mkdir } from "node:fs/promises"; -import { join } from "node:path"; +import { copyFile, mkdir } from "node:fs/promises"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; import { randomUUID } from "node:crypto"; import type { RuntimeContext, @@ -97,44 +98,24 @@ function makeRuntimeContext(): RuntimeContext { }; } +// Fixture lives on disk at examples/fixtures/workspace-wiki/. The demo +// copies each file into a tmp directory before running so every run starts +// from a clean slate — the same fixtures also back the workspace tutorial. +const FIXTURE_FILES = ["a.md", "b.md", "law-clause.md"] as const; + +function resolveFixtureDir(): string { + // examples/src/simple/22-workspace.ts → examples/fixtures/workspace-wiki + const here = dirname(fileURLToPath(import.meta.url)); + return resolve(here, "..", "..", "fixtures", "workspace-wiki"); +} + async function buildFixture(dir: string): Promise { const wikiDir = join(dir, "wiki"); await mkdir(wikiDir, { recursive: true }); - - await writeFile( - join(wikiDir, "a.md"), - `--- -title: Limitation of Liability -type: contract -created: 2026-09-11 ---- -The Aggregate liability of either party shall not exceed the fees paid in the -twelve (12) months preceding the claim. See [Indemnification](./b.md) for carve-outs. -`, - ); - - await writeFile( - join(wikiDir, "b.md"), - `--- -title: Indemnification -type: contract -created: 2026-09-11 ---- -Neither party shall indemnify the other for indirect, consequential, or -punitive damages. References: [Limitation of Liability](./a.md). -`, - ); - - await writeFile( - join(wikiDir, "law-clause.md"), - `--- -title: Public Law Clause — Cap of Liability -type: statute -created: 2026-09-11 ---- -Statutory cap of liability is twelve months of fees for ordinary breach. -`, - ); + const sourceDir = resolveFixtureDir(); + for (const name of FIXTURE_FILES) { + await copyFile(join(sourceDir, name), join(wikiDir, name)); + } } async function settleEmbeddings(): Promise { From 766c4b1c9e4b7b1d1e656882003f391309b0e65a Mon Sep 17 00:00:00 2001 From: Peefy Date: Fri, 11 Sep 2026 21:08:25 +0800 Subject: [PATCH 5/8] feat(workspace): add multi-format fixtures (pdf + docx) + parser deps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fixture folder only contained .md files, but the workspace parsers-adapter advertises support for .pdf / .docx / .pages and the tutorial documents that. This commit makes the demo actually exercise those formats end-to-end. Changes: - packages/workspace/package.json: declare pdf-parse ^2.0.0 and mammoth ^1.11.0 as runtime deps. These are dynamically imported by @langchain/community's PDFLoader / DocxLoader (via @melandlabs/rag); without them the multi-format parsing path ERR_MODULE_NOT_FOUNDs at runtime. - examples/fixtures/workspace-wiki/law-brief.pdf (53 KB, pandoc output of law-clause.md). - examples/fixtures/workspace-wiki/signed-addendum.docx (3.6 KB, textutil-generated). - examples/fixtures/workspace-wiki.README.md: moved out of the indexed folder so the walker doesn't pick it up as a .md resource. Documents the role of every fixture file and the supported format matrix. - examples/src/simple/22-workspace.ts: extend FIXTURE_FILES to include law-brief.pdf + signed-addendum.docx; bump the >=3 assertions to >=5 (files_scanned, files_added, listWorkspaceResources, sha256-dedup re-run). Verified locally with the workspace package symlinked into examples/node_modules: - listOkfFolderResources → 5 resources (3 .md, 1 .pdf, 1 .docx), body extraction succeeds for all five. - workspace demo 12/12 [OK]: scanned=5, added=5, lexical search for 'limitation' returns 3 hits (now includes the DOCX), cross-file expansion surfaces a + b + signed-addendum, re-run reports all 5 unchanged. - workspace tests 16/16 - pnpm -r lint clean, pnpm format:check clean Not included (not supported by the parsers-adapter yet): .xlsx, .numbers, raster. Co-Authored-By: Claude Opus 4.6 --- examples/fixtures/workspace-wiki.README.md | 68 ++++++ examples/fixtures/workspace-wiki/README.md | 22 -- .../fixtures/workspace-wiki/law-brief.pdf | Bin 0 -> 53456 bytes .../workspace-wiki/signed-addendum.docx | Bin 0 -> 3590 bytes examples/src/simple/22-workspace.ts | 18 +- packages/workspace/package.json | 2 + pnpm-lock.yaml | 221 +++++++++++++++++- 7 files changed, 294 insertions(+), 37 deletions(-) create mode 100644 examples/fixtures/workspace-wiki.README.md delete mode 100644 examples/fixtures/workspace-wiki/README.md create mode 100644 examples/fixtures/workspace-wiki/law-brief.pdf create mode 100644 examples/fixtures/workspace-wiki/signed-addendum.docx diff --git a/examples/fixtures/workspace-wiki.README.md b/examples/fixtures/workspace-wiki.README.md new file mode 100644 index 00000000..840fcf3c --- /dev/null +++ b/examples/fixtures/workspace-wiki.README.md @@ -0,0 +1,68 @@ +# workspace-wiki fixture + +A 5-file OKF folder used by `examples/src/simple/22-workspace.ts` to demo +`@melandlabs/workspace` end-to-end. The demo copies these files into a tmp +directory before each run so every run starts from a clean slate. + +## Files + +| File | Extension | `resource_type` | Role | +| --- | --- | --- | --- | +| `a.md` | `.md` | `note` (front-matter `type: contract`) | Limitation of Liability — 12-month cap. Cites `[./b.md]`. | +| `b.md` | `.md` | `note` (front-matter `type: contract`) | indemnification carve-out. Cites `[./a.md]`. | +| `law-clause.md` | `.md` | `note` (front-matter `type: statute`) | Public law clause — matches `a.md`'s 12-month cap. | +| `law-brief.pdf` | `.pdf` | `document` | Same public law clause, distributed as PDF (real binary). | +| `signed-addendum.docx` | `.docx` | `document` | Same clause as Word docx (real binary). | + +The 3 markdown files form a cites graph (`a → b → a`) that the +cross-file search strategy walks. The PDF and DOCX are byte-identical +re-statements of the same law — they exercise the workspace parsers-adapter +multi-format path (`@melandlabs/rag`'s `parseFileToDocument` → +`PDFLoader` / `DocxLoader`). + +## Format coverage + +The fixture exists to demonstrate that `parsers-adapter.ts` actually works +across formats. Supported extensions (from +`packages/workspace/src/parsers-adapter.ts`): + +- `.md` / `.markdown` — pass-through, front-matter parsed +- `.txt` — pass-through +- `.pdf` — `parseFileToDocument` (requires `pdf-parse`, declared in + workspace deps) +- `.docx` — `parseFileToDocument` (requires `mammoth`, declared in + workspace deps) +- `.pages` — `parseFileToDocument` (macOS only, via + `AppleDocumentLoader`) + +Not supported (not in fixture): `.xlsx`, `.numbers` (spreadsheets), +`.png`/`.jpg`/… (raster). + +## Generating the binary files + +```bash +# law-brief.pdf +pandoc law-clause.md -o law-brief.pdf + +# signed-addendum.docx +echo "Public Law — Cap of Liability …" > /tmp/law-raw.txt +textutil -convert docx -output signed-addendum.docx /tmp/law-raw.txt +rm /tmp/law-raw.txt +``` + +This README is intentionally placed **outside** `workspace-wiki/` so the +walker's `SUPPORTED_EXTENSIONS` filter does not pick it up — only the 5 +indexable files above are scanned. + +## Demo assertions + +The demo runs lexical, semantic, hybrid, and cross-file queries against +this folder and asserts: + +- 5 resources appear in `listWorkspaceResources` +- lexical search for "limitation" returns ≥ 1 hit +- re-running `updateWorkspaceContext` reports every file as `unchanged` + (sha256 dedup) + +Used in: `examples/src/simple/22-workspace.ts` and the workspace tutorial +(`docs/tutorials/use-cases/09-workspace-folder-indexing.md`). diff --git a/examples/fixtures/workspace-wiki/README.md b/examples/fixtures/workspace-wiki/README.md deleted file mode 100644 index 0c9a90fe..00000000 --- a/examples/fixtures/workspace-wiki/README.md +++ /dev/null @@ -1,22 +0,0 @@ -# workspace-wiki fixture - -A 3-file OKF folder used by `examples/src/simple/22-workspace.ts` to demo -`@melandlabs/workspace` end-to-end. The demo copies these files into a tmp -directory before each run so every run starts from a clean slate. - -| File | `resource_type` | Role | -| --- | --- | --- | -| `a.md` | `note` (front-matter `type: contract`) | Limitation of Liability — 12-month cap. Cites `[./b.md]`. | -| `b.md` | `note` (front-matter `type: contract`) | Indemnification carve-out. Cites `[./a.md]`. | -| `law-clause.md` | `note` (front-matter `type: statute`) | Public law clause — matches `a.md`'s 12-month cap. | - -The cross-file cites graph `a → b` (and back) is what the cross-file search -strategy walks. The demo runs lexical, semantic, hybrid, and cross-file -queries against this folder and asserts: - -- 3 resources appear in `listWorkspaceResources` -- lexical search for "limitation" returns ≥ 1 hit -- re-running `updateWorkspaceContext` reports every file as `unchanged` (sha256 dedup) - -Used in: `examples/src/simple/22-workspace.ts` and the workspace tutorial -(`docs/tutorials/use-cases/09-workspace-folder-indexing.md`). diff --git a/examples/fixtures/workspace-wiki/law-brief.pdf b/examples/fixtures/workspace-wiki/law-brief.pdf new file mode 100644 index 0000000000000000000000000000000000000000..d479f036655e64efaa2c582f3c245347c1e8b9d8 GIT binary patch literal 53456 zcmagDL%1jmtYo`woNe2-ZQHhO+qP}nwr$(C)&G0lchrNhrJGN@EYGAChEupx%q3EPdY|Wg_@fn$!@c(x~(TQ4EJDWJ-(}`LeIGYHY7}*({ zK=JZIIXOF;7}!9$Z-&Ho$qv*16Z!HUNtQ)8YCB3r!6$$sUIYwDa#5Eu4MGHO_KU+2 zqf$_q$yw_#LV5=Pf=Brv+jU);>51oOF;#{-A0_5U%?v0+bQ;R%8l>{n{n`)fJP0lz z(_#wobrf76Sa)$Kc@RyfywvZDOds_b+u8-kdpa%gy~9nS0uG8Xq|9R7-cN-vZywET z9(g9&Y#+7gB?db)U(c7YmZz0(7u0H=doLMQUb+}wwmi_HhKXMGr{})0AdLtL%ET;&x5$R1%kOj)DF|NwnVOsk=NBg017>z^ zum=*$?D&5J;IdiyM|Ny^-FbmhQQ-lh@Pk1Bb#{}{0GRFqkmP4ogtj5Y!}42wqRG2B zyBIMtx%kKd3#hW<Fl`KbL7`W8m9a+Mb++L1epBs)` z8~>s4!CGOd`Ii+%;q3qHfSSX9OQE~Pk91|*uJ-q#y*E(-h+sM4kE6VoNo+pc6%TbK z-^tuLwEYph(gy#h`sR-oyd5auk9Bl&`K3-*L`6l!Kd1tB92SBRG@}o#Go!Pk3rPAm zvZv1M#1k?(#79PR>;S?#k}+Bah1z%F*rRBaHR$cl|UL zUkyu*ZLLoY1c=(^I>2we>vw|*YhI6mgu#Tvg2Dxsbj0l_AUzp-Zv~K{%cJ8*<;RSN zK|;hAfR2jIKPC>T|I=2HB0eOzC}@bwN-dc0QDU?E>iG2haOSZu#s=K+8RX*^4^j#% zKI8=NhGsiQomqf&_LiSx;*Zh~p5U9x6x12G9>{+dAU~~xWhyVg@8d)+{YLIx2f|N} zOpMIGq@ssves2`<{Wo;~%JLErq`jjZ_{Zljwloj(_v7 z(A@LFWDUUuf9aZ)!Bi*5I?tu9dkB7*IzCFx#&Ua4+N?e)R70?*LkV z;S;|gzh>;d&*n)j2YGyZVGzt1RR8P{EWNWm-IJ)k*DD|=3VMI))z%t1e@)L6u%12$ zduLY*K=#)57eGo3Etn5$atcsi;ScE(`r!dUUHUJ?9gy1ePtY3x_32N*y1E(wHOh~l z6#)7kFCM4g@B@AvQorFB{1~Wy{TsM;B$if`X5Y>RKd3;*`d|4y3c7jUo4>leDQPwD9w z`u9t<^0#C5m$R(;Z*%Lf^XzZ%(v~*XmEQ&{>g}%GFPZ$m&0KfCXzImdY<+wG+Vk^c z{B8qO{_Dj1U7(@8JwKTmo*01AH$3xd13CL@>Yf@p`m^gf_0#L{!?|}u`epld+5`0G zr_1LUd<-n=;4inaRC4$5S3YnCWk$h!vOtNgkM$;8b{usEC&)G4W#y1BCT}MpPwE_3 zP)OaQ$LBy-E)+tEwE6pNVyW=F92Jyt9%JUG@c$U}BvYlUOLxZ>3*iL`xl z14zX?+AL?C;k#Olk8LK6J6h3(<%Gsvs($=|bYn~U$5V=Y6ZABPGDD>PL%t>}Oh|vo zFIfmhg#j>w1`L8d7wt>3Z9G0sfqXwqxJLW757c1+nVB6o_P#f7mWunqUi;%L!lEjq zSR$+T1*ddF{;`B(?9FQ&s?`$3>;*~S2}Qgl1&GM_BzJs;m*P&wh+(&L$EJ4)|AOs2 z_D{F%;I@w@t6zjH!JkpCGlO6hYDjNZ;~1g%T`4|Dw0{IknHAEnNxc9=ppfbSVe65E z|I$HERYx2zH1(hXXt)S@*r4%@=tarO#8jnIr`Y#5^s5!zm0wfEv1)3It8oI>H)3@Y z%92EYU@=3kvgC~z7B+sOj*vO@3sD7NO~2tb+zWFSY*JAjLO)Lg$b=)ybWUXp!f3jm=iXT8n@&AOa+bTogBE=AdV( z1d<0QnVTk1Au*UzNUk$xv9sbKVI8$|_tS_{`NS>~%O$TlwJ6VwZ?Ag1=hT#k%;3|3 zC$uAyY*>aO-~iZwTDg2-FS*8&Pw9|T>CX0&_BAEy1cA1rPjx9YY7+5pnbN`cQjO?- zx9i^kYALHOVv0h&wE@mbP#WWS-9=@Q_Yr_<7YDh<9$LI3L9UUB0paEbb+q{?Zt3ZEfWF>X?G`=?&gX+V;j34DBAA%R|RqT9%a zFjPY3UmP`7C|@2(!iXhQ2F`0}1F~c*)JfD?%yV!iZUt&;h>ziCi$clU)2kvT_GScq z!WsnplHmUHvi`9J1nrQ|sBrb0unobeUBjv*uEpoU6v|0U5|zH^pc+UU-b`Sn2zv(W zy~5&cb3Dg0PO*8nh--#hAy1w(8;2btcqdL=1*i-)WPk=pE9~F7K@d~{+rrw9ioJs7 z>AsSvu;X`HUwx^)ku-xMZIkZ0CmkixfYk2H^mQW&6pd%*s%8j!eGX6 z&Xlq6%o6KHCF%=TV-)3SUmrEMKQJjEjwM+p^=x&AwUtl=>4;?`jQ;L)#n$-*7v#D9 zZfx05j`XPc3G}e$^*jJGTP7>=YH%`gY{%sS6q1LeeoeM8k@wh+^j3n$h}VB8I|IWo zVti&zuoOTA3I(&7aZL`g+!8JRiV4M{t=_l#PHgYZ(_VfxF&6>NQ63 z|El6{nulJvI#~t;YXe~@pU^$~d_N46MARybn{lM~6~saSPD(eegnuj5?tutQFZ}2ytLn;_64Tan$pkT zFez<-TMnA|?_%|z4}4y$B`}?G>1}?ybYc+iJ*5CwS4v^}*%45gBw_t+muKv9Y-y=N zUhe5n<2mf>=&NMn<`0F!oJy!pU{SS`uw`kDo+BYm1oh`&?F=^}rLg+)I_B+MP2NN* z+B(iih~2?9IO=gx?8hg0xap&>p+)9ctgl=Zy!k99)TSl;_L~ZIx4)!+9x*k$c!t`t zq_zMmvks2Ro7pYDyUhkNRXa64!V~h*5tovTqB#(%_Cs(w<0y-#kp|d!MSY4RAck?B zmYrJQ9yF$1U12u#Exb12-ZNojp*t+E*a{ltw}&ezIGJ8hMO<&-?_w*cFuL-db`s~# z8;be<;H1m3GL=u;sY$Oz5pf(PO}{i2>&Vc~ElUGqbt?{Fz*536W)6!jPt)L1qA|&Xc$1rcIKMj{^8Aj2<7>#Z*Z-C_~ z#}>&bJ-FPf{C9Sw!FU58(-62XF-`{5=yl@lWk$giplwaqyXZW=5`hNS-eFOW9)=FE z_BNrgQfEO!B23DKps&{G?bD7V|?>TZ*jZEOT(qR6A)Ei7sW44pG^tM2oMa~1S zJ`)qblk%ScEIlPZKBsPOX6oqN;b-mFL3UOS<2BjV9ty0ultDG4JnuQ}nIk7hOv0vg z00=CnWENVFwT2Dx`-EsAJQb$%pPXe{DaKn+e5vXP#6c#!+BoGj&)#W=B9!VE4y1fKjr;}{;64aN;;qe0y&XbAA77~I15lDFut)FPvj94M zQyY8>QHN4U=fSbpmv?9HHTqV0z&`eLO>$4fwqS?gcrUBksz`WjkZM6rb{cLZW6?91 zzeW=MY>DYQTzw~ZW2oFW#&%fPmIUv*(G#Y%6XEvuZg=AjYe?4#$|dVBrrH!}*O`}@ zPV2B|d z#5ik`;8DFl5Iw75zD)MK7Dg{Zg7^jnQ#!+C32h(Lv-aq|k9MSj@=_ppCh|80Emf~p z5u^rGp$K-5w28E(0@ac{kM+xUeKk0wN7Bc>E1<#g(s-)7f3`S|(W}nY_0e^PY2wp8 zvYZ6Ag1GB`h16b&gm%tBy^-*mc19q4PC5nG@ME4}A1a=jHG8?h&_sWA-KEjtp?Ke6O~O5G;w!-RnsZ)6`(6&yCelxdjmYdi3nCV=IqaW1%zU1z z<8}YT=id?^Maau`IH3Cu!BZ^dkZ&kTxhd#O!PsENLGZ0tuH8dyX2?gemrJl_!rY>@ z{4EZcz=ul^M88}#b8hB&+%ZDGf3OYujcG9vJjywpOp%qW^`NIMr@Ueicb z&xh%en>6OOe3;`5+iW*$Pd2*BZv4;u^--b|S@X$G+@gN-7A|f;C%6AIk3_rB<8oPi z;W|tab!Z{lRk~w50OPV3qH6g)oR772wb;^)mXrysULJOZ>^W}>P_tLX7Vd^emwoDJ z)$6v7MPdF*(K!oI^VW#jxue^rN2#Nz!wpFw5&qS&`m)VytgK78he4udDihFOn>ka2 zSUhcydVw&pMEq+V%0_XKO^AX@wC%0=$2ImHp%V6Y!3a^KfdYk}EmE}jb;Y_x<_-9V zLLwP+`gvh19}nlP`0Ic(yFvk`+a<3}?Ynh04?g$B+X=D5H-31%X`YuGm7H9XE{?D; zu6%4*4<`%Rd^`iS42k^1(I_nW$=DkkrEmAtPx~N5gs~Y;=I=u!`@xD4n|&7f$Q%)5 zB3nH}haE(9EZYP(@E1aX(V^P2!*NyqJ7&j{{@{rG=1CH7Pn8mBW*>9qpH*vbRgMz& z8H1sG^!c={a%jpW(E3KOqnfmcfKa7l9bb<^c`O&|xQ(Mp62ldZTiPLTiJ{YMWst%( z(nRTs-}Y^WB5th5n{fOOjVI%>X0|(Mf^gTH6e8Q}v7wI7pZ7hTiwCqgNL6@-Q?q3b zciCyWx1TbP#iD-Qcz^^1X#(WqEPyQpmkl>qRd)uGLIIe^aFRAE_QMXqtwDo?T|*hD z&|Azj72hm?!-jWmxpS)qR;}1d9>DQJC|-f#B^v^XJLF}#Kec+pnQ7Pn5yueMt}x_r zW7{(@ODbQo48QnyO_n?I+Ob?3^XpFdpzdb<5USlqC2r~U3R|~2p-({UfY!Bgd&oyu zWaV5L^d0W^reKeo3$oK2D4~N9Lz+7|UU_4o z!_7upiFo*{A*#x7VK3>Yh0Cd!vp^$EO2D9vqeX$-=tEizbM3mBe)riK!s%J7*cura zQ3(`siOsGok)1+6m)A9cnxe)MemZp0^66SF18InHHo!IE7w?Wo05?1JA_qwAanocs zpzYS%wU|alZ=rQA2&2QR4Qr^}@p1dE(t(yD1`^nSl^|PbscL<4Go?kRP~Zh}aC;c# zv*#MWTpaCRQW45Xn0FCL%9zaa8va5eZmtabu_{v8YhbH9*>X6G?@R%FV^@_eB|dSC zkN`Ehmuv)U-#73vH(vyCD6$85&ugUwmfTR{Z^z|=h(W8YYv`6TY&EodNPYg(V<9zee{ zpDo?cgyjrJd>Yc;y&s#$l2Y9e6I7d`Gvj~1Dskf*b5ION;hH67Oa$JzK=4dWc(B4G zEqx}#>f;6b8J~u0fEBOQK212*5srfBQQNh%V`(@#JMn1@(RfUlC~3(~-L1LpJ5k;| zX)70_Hg|!JNf*v8-WtWyK6xEwz=iV&^2B9O#Rm0q^O-gbnKY&0WFd)nz>uY%`*9AJ zQjuj{zUs+uPm3vs9SBt09qJfMl10cyr%aNw9f+JzL+3O}o<~sOgiCr)r$R!;Y4VZk2?T z0K09~yFA+At zZsGQ{%EAdZnHK!c;EAroZZ#D48po$j62L`#Ew&)|rWbPwzq9E|;3zgG()ZoLJjNV; zuro~SQrC^U0s0UfBrDi1qfJHC;G5{|wBPs{Vi$yQf#H5u*hA43K?;hL{{*mn3IOEQ z$xzf{t#$Oq?8xtS1epf=R=to!a8p#=>cgRg8|-(HViJ~>T^acQDxJuRQ#Pxq?^(ll4ovs zj+yqgSO;y{!?tW+!UR&4jw+`n(p%=bTOp}!>*0@-s5cN6c_lJCOARFj=sXgVLU5$= zwgDVU;T`k3Orh38C`+X8lfZy&4C>tBFM<2+vXb)1-jO1GKKN3Wv(t>9XJM_DJA;bfKsVzfZK_v#I;gf?Pl=Sw~_;w^?t zJ@N6lr8CqK%FTe0R=#QY*w(G>BqRsG>+Y!%CB^~<+$In)3iFF2)7@#`KXrdW(;De6 z{lU>JMw5~{usRr+Y0H8=wI<84_*$=;TOt%8w2C_Q=<2+zIcy74d6m1Hc+;!`^8>c45b-dpz_ zPagvTGS62<0HYLqRxoIW zYgNTu|D8@x7RMIsdTOXy5Hi*->fs}*uS9a3MxE=|L%sN)>E zC8g4yyqq$e+2C_1NADJ5Rac` zD2(6;RT^X@;<4gJ@oHZ}okE)VSRu{DFZgcSl?L$Y$5Mp7FMU?5(c4Z4Gt*CB)8l#T zaOV_$?*Ak`JHzYPIY6p9{H>2!ZO=KosQ!MXL5pp)Cf8|`*o1eMc3_^up!E5IeVk3v z>an=N6qIGV-VJrOyNqWsxZwu%^t5`DE&tjuyYN3;keR#FFt+2Zbe*piKv{n7#s&uk zZA!Uk8s}Xxg=!t+EGJC0MFA_8CTDM+(qvr>Dor9#bcCNYw#i5tQIosP%+l(ZS2x!E zYdNXw)OGkjwCHaKKO0UV|U-rAMYyWsMXD=GLad|ucrU44?`e&+_X|BBR&ulTfjk@ z=i5r>nVrz2C@@NeBGTJ->ox`HT&y5QrvKg8Q*gMv(Hf{(Qmi2=nA2!uhyR^%cBN`g zaBuZ9cS@bhK8F_iVsB7T$*GnESD*kjEur2CnFU(f=?_kwU7X{|@%WdzQoPR@Eb$Lc z4>&k5ZCU}!80uYTrryFeMd$Pag_m;6?Fym2R}c~DVI`v0fy76ww?{m?fKj6tH><)m zow^y5-nB02WG8>4GCK=3)}&>ERNXgnHV)Z9@MBHJoci6#=hbqicAYq)U)#lWqP$;ofWZJU&(*|Y0b;U007X%# z;z_o{a-VYzL7PB{#03#0L4N8g+)3y%>Zek|o$))w5gq|bgFECISyP!g)V_?7?TNb1M4P}Fe!?k44CoM0C1-jPgp_5J+?_X zv2sO*^7}e#vABv7FwGq2I$^C6*MS*$FM9F}$Ps8Ne5*Eh8JrmlDHE+=+Pcu3)n?Tm z?-#Cg+t3;JfSf1T#Z8ui1wKGg(_<5inTjV~m5?4+cMWnWROK9T2P2{>7o*u8a`VFi zN`jQBWfcj$9%j(K@(8z6d*pQns?6)NJ2Z))u%Gs9hwSof`l;Qcu?~5WmsX*vb+y38 z$>&IYBlM@!OvzSG8~DHSd@?F6Q#%UccNNLei0;eL_Cfrn?8HQoA>#5lM*Mi!DFWCT zQtVyhQBA3PbG=w(3$>K^maqBBR{1YDo`uQXiGr__G30_s+_;+0S(fOWm$S-{l%O~M zpU4k~^3>x82c;GPhxewXA*w>5b+_YbSjp+6yaeZFR3?q-9B&A<(FO}|dtta11qs57 zGefo+d+3oN^>Py*5H-BIN$30YJUqmPTA?en#_cHd?e9h$b%&@P1~OEoc+<64Ju@;4A#__vv0*V*{d<>o)!PW6l8BcsTWlL>8NifmH0q zF$godQw~g5))&oqHNom<`!fMvi<5b?A^GKzk6h|H9|wf9W)t1ZwEq;#3Rm9VpfKY(ACGNzCT_8KWWa7Eufgw8GL$ z;;nwiu#}W_cUa1trDi=h@fJE9ca=-^>JSGa2G`IY$zESE`DbqGn(v?bR{J~tnmF1+ z7$~|A){I@MMyn`}f_zr!b- z2X4z=?>0tJrSbmT;#J@8b7^^eCBKO8fz9;BxlmX4O`;{ViE?m#7TE^Na9R_{ft=jcOhu_l-Q3lW={iHFAQ$Tn#eat1O{dBBW0@R==GTnaXCeYxgB) z7$6#-&KJ@5ZV=>duQNsaUU#R&3lX+ox+vf4ta<8Z9Ol@4NlNKmFaZp9Sle0EBM7v)`!;CFA%K85vqUlL72*r%Ci&%7Wk3oMl-6L8ijF!nMBv%$Fu zf3r)!1coMiS+uzUvlifgm(p#DO@4xuC}3AApBr*eitv@A^q!_jF+GL9grFi<%{4lO zrGZ*?*lfOY*s-l$Sv}5y4RXWqZOr1I!reaUCPX~p+^Hjdp^-aQy7@Vwt~aBApMPGqny>_G@5Md; z;qSASh~LCRbs(fR8D%KlZ{hC?@%(&Mu$x{oyF)@(t#)B2WNTt(7|N@0@yZUpgKQjY z;4SN;LZI1d+_|bVN&1vq`+ZaN0?!>u@#IeVhbLm$yeVqD`rFiLhfeKF|Q9( zNIC|6&sB-|xu-nJX6T+n_8*TEX(ONI z$bZu=@cwSiU<8YoWFsleit%ESoFYVlp(gw|_}#G4W^>`4%oHUpQGs8C~ zqhfTt>L4d~MVV|nZ5HKoGvN$xdnZ2hH)a8FS1ncN_)m{{3m==K-BB)l2su14WelWzw_r?S zX2~}t%;MKK%W$-Wv~1M6Pm&MRfVND14+;dbZ6FySd?Bq_5b(>#+XZSe#zj-8PcaQv zP9PIX7Yrauv{8{&<>`H&vZqVUq(}d;xm>+AHv!ni>c>E^JA=IfqM&xUpHIt=TpW@@ELs zEm1Wb(WM8_z99ivpA(az%W~$K+E%pjr@lNG`b6Xc zVcudhafP9?qw*GLh+SM|YAW>DCH+Tnj@U78?Ho3J^}TORtGL1iiHFCN_|CT5x8WSq5qrPUD(-p{|7(9YejPnmtIP`k+u(^2%c(RBmZ zD!wRo=lL7-G6=hnj0GvZKzXfpO;WedB!-Rt-}7^GUiQ&#CEevA99%5C0Em$qqocEb z?lG!%_={!k+9CCqk;cz(Y{z)? zZT)F~`*YC~n^q~IR-;HIS?{UR2gxcvPNbccJwSeQo~F6uxXhV0LN8aA;#09nOlOCySJIm;7cRCuvx2@-w~b z?ELI^hP6D*-XnAtN#b3Vk9M$^0p!?UT%H2ade79BYF35d9#VeHfXaBkJ9cZs>*MeyHp<1o5%4N8*w zzkL$eb-bP8i}4YNXI3MQgEpv9q6*Xiej#`#8KpfnC_b56jPX$DNAZJ%w6q$^=_|7n z(}5T>qVS-tlud4`;hFfgL2M~Mt3-?vEN3e9-LS!HY^py1cY8cNfFT^K+4(%;E8509 zP4nU3W{W$7BhQrUyrF?qhw!O&C)O`#TMHLeMPL~Y)en+`C6^tdIAN07EbCaCzN8U0 z=Y(V9JcQbNd-eYw=2_*j-C3=LA13*!KS%4k@JH%kgU_Z3YY&)wZmsg6$ES$h53@=1 zEDXW7yR9c9*y9VpBvg}gk&W`ge1@~wkMx9~LRP^dM;IemTu@y#Cp{VQz@C6F1JrX}7OIshBZL!C@|aOZJoMBL zH8)5|3`q#TzK`RUxa6^0LI=fO2UCOE75H935aTMWCFvt`RV_xPM$YPN6-2`9>aSvb zDEg=X29Ty@JF&&jsOrezMi`w!z9*u>S7|`H4%if41 z>%Gr@8_Mv%;yKasIzxRN)_o4Fe_2-f~y&kyVGOt1(u`rt=AQJC~ zx3G*XoW+db9l1Gl`DK~qLnUOk_-Wjic~vFCdnOPBjA4$GJgnUAC5dWV;5(43XXVPG zaIMM1Vt1)WFv4V$8`F(Gb7ddI({9t5T2#< zKpBA;CeGR`2L@dPNG0;RP#N&Y6-;Q5La7Oo~ zsaG6j>gm$K_|yWPcB)!AO>HEKl?dvaoofg$VVS{+>gP)L;%#fLf*y8ywM0}QeDYOBz8|GGsV4boeg@wU=8n8_gSizS6=3CV&ru>r-w{`(@p z$pH~740k+>!HiB&4-Pk(L_Io7JDVDCuQ@#tn#JV-p==cUo$B3p5$h}i+%4QzH+8-ZLcl!shEO6bpykS(HYyxJ3_s_WrUD6V#La1ScQ4l2^>D=|T1 z3$eds#BXmxC8hhPU)CS^QeP8S-=?g~F{!TY;!(dO#Ly*qK(j@Re6S%_Re_D{l+4f5 zv>R|B`@IRR%#3;*`d$lGz*M~5${VR=@DWyYn7Fn)ZfG62VF$p94R_8B-4s+40uzln zfCl9joYsEMXT_18Ha0WRe9bSs3^dG%_kOH5bu&0|tUryjdG2*`x7Q{`JLGV|mT@_j zgy&E(%KH+Rzd0Y54yAbsD+*qS`G$Rg%wi+v0@h80=Yfe5n%sQ90yh^5;J_O(Z&=kj zxI%Za&N@4u$BP!2rwAn1QF?O&L~p1*^zq|nisNT|-&|lWUK=_Lzj};;w;C^lm!P4{ z6?)!kEr-z7j-1OGvf-9oXx(f^SqMAe>hEFwv98z0FE}f-hZIxRD!uRa5WGKVD+5$_ z>?3BfvGcixCN7W+lNJ9hE6ZU&lK;tq=Z%?O^E#-D2pi+%D3ZJMfI|nX0F(0SY52$aGd4RUSkhOOHwKUWaP%Q zW0bAto++aE#g2m-PQz5s{vuILS%UbUO~f-@VBVE0YbLneShjVlktVC#{rB>XIxOYNz-#3bC8={T$BLTVZn31IPjgmw>8jVhO* zsMJJO>SZAnt0yn|>3H&X)X!L-49LymE6|O@htSRcHT9a*(BJwr*{Jk z;p?2i#C@sIjm!Q(_tQ5oOpAu3D3216+~vb|@|-M`^yAJ)qq$s`SQjoi;tiM9i!jFQ zS+=0vOJU44^4TiA2Uwg&MaiGuSur#ubS;&Vpt+xk9Ahu8!`&c=Tq$;eIsFA}I{}K~+1Z204E6x7t2&Wanp$@&GVVsY0f!s~Jj^9`0&u3IDfDFyb{Yl7Y zcIG^_S*>%vAxiq1Izkix)U2=)nmK;WCtZ=HY-;zBsxa7rkvUSUJ4~L3W3A7Rn4f3F z!7Q0N1bX#!@y)aj*X!IWL_kSO`23g5@B>V^!RLScckIz+1;p zFYel89lUns&281=C=;$_x&`LjWqNy&C^hi&3dR1nF2N(rb}D9Q<{Kg|4`l(M_6f^H z9G(K2DBNXNEysHZ`iH`sE>^O%1onWGInPWF?B{#(>;_QcewQOgs}V%Xq|WxC z22T8DJ*Hzrd&~jQ=)yBUFD0!gY@OFGVU8;%O~w{0lH13`#v5cpS+cLvxVro3Wud7N zrP5d3VfP!YW7F->DTh7+AZo^3I88PB7*BfnJXNICub@$Q5d>jd4R1nH#06~zHXxdw zyE6izVU!HK%xh5S-;-YTp&$E|=3?tTZU6aQ8elNZk~Z?9;lT@TANS>(pmZ#6lvXRO zfxbuji!2B8DEI!nOy-dcxe>5d55f! zx!%t%Rpv=ld3*p*OZY~Y_8R|~Y(x2xKX1yrt&@ONV+WjV4(-w_1FR$LJwvrsfyCi7 z(gFSvcCLu-Iutm=m#&P$;D$*WR)_mxJhSp~11P7?kw!SbPBp#6_3`k6h8l-<)y3*L z1bAx02gz&Xyc)n2LCh#tZHKvpim~8`--zvXu61{Uj@pg6ivVr$!kVRmVG?Sd^Y()e zfuhtF$@zS-Pg`Lu#(LTLI@{;Xf_T-vCuOf^2wMRHerrZMx%7(tusg7lf5ZH*^d;z- zvVKo79-y@1(p;0bJ8S?lpfPZcV|ZNh-_MHDU)p8&$xwzR(61CXNse99DkDzXJ%*Sk zFatZLelEV}K7X#uADcU>0nBV@3G^mK-PuGKZuc(HA=3~w4ZajY5;a+1`NKo(~>GlGOluJ5H3W86uM`P%BZ(2 zLhz%eF+Gv(Lc9cI)j0dMJRLc{EGa$5zt+JC+NOEtaI69r4E{%_%qX6db{WpgAfxX0BD$r%F} zd}1Sm)tykHw0<27Ja6W8O4Req*CjvoV1e;R2V;1fUv@cPv;C-cZUoD`c2dqz)0tNj zn9!&UnneMNO$qD#_)!U7sKbw5>B_~7R(}`gno@A&9n_<^qRq-HNpVRftulEug!(~t z$(YTXx%LMSUR7E#*HGF5nzHU|?O+?;E&I~ZQ2nX}GaQ2Z&Q4acYvaSxcOq1r_H_Ye zUTaM{UDe5jTzZk0fuO}euOFEC$69oAEbWl#7@715e%jFfe1wNJ;Shzq6_Y7S(Z*%G zALym|(B=Ey&axUaNp>{3BRg@oMvLHE53FJ_}-|n}?t4(AH zowBBvmHX@%zXoPe0ttI%-?f{ZLJzzj(rx? zl;~j0MSO=b*qqZLRmIh_U+JTNlPhZN5dkc@_2hZ7rq_WP$FfI>p@uAz zzdG!oU6v)G4+6UfD5y>>Or=d`qMc}agEfZ1(y5Zb2;BmUCF{C%2y9KvEj9IRK-Tu+ zT3E?#T;ErUm{EcLqG4i6KII9BsGV%UH&}F5Egk@?J~BLkPglbW{U!Qltl$**&1skJ zHkOqU*={0>{nwP^PDob`y8 z9TQCZP~M3k>m>R>-WCpU>LTUKV&NKz4t%qxH1aDpE`(%(XfHB@FTPjRk868AYQ*P{ z8nFuQQgO2f_c;#qfKP3aT@slO3F7sNuu|qocTB>r$yrMzp;(0_4!)(%8v(d^cLqRV z4@R}pP6xib)8(D|OD1~K!!)Da&}fAV@Rafh#)f*P?*aJIp$-HUrevEo)cw8 zM$x~@0I9!2&MonS$K2c|S5fAz3&0H2GWs6OE=~wXKGrL}-ShWib3KUpa=r8vti+NC z1QE~{H>`2aS3VxrmX_Nd3$VRniKOo=LPKH{uvK{Nxx1wndW7rBc><7w4iM6jHAuo| zG=g+q&s+uH>xUI5m>}qTqt*K87 zW8^+Q&8s2&A}8VA-mJj?M#%+LRI@tRynW68VIjAgoEWN!oJVvT$QhdMIncKM_>c8D zE`7?6ual19PM>7U5%|cV3pg$L)4dM+DyU1UfN^Sc0RoG^gq%xj7R{49LlgjN>N)l7 zi%lrGHfMs_XY|G{2RDZ&4`P~&ZF2T~q!(He3{o+AOJS}!!fD8amP%c4{6Q)__#?#1 zPf&zJB2IP0+s61##$F8INyIcrE6D?}tQGipGG zGz-b7quKbu>AM0JNHR#TXESX}ma!+yCB_q&+7Le$WJBbmLBl*Z)=NDRVtB zU07=)tVJ7Efhz(zNcqe&7j?b~#f9a=3#wrM^%5R16)?W%?6|6u7F|A@s2SQ1o z!>v~ihQE&E=1@s}YQ8be8YuQnSVLUM3i!ogQeeAIuPUTQR@6QY zL=?iyg^PQ;7>crg!QHauR)C?A(=8{!X*rJeF{EHhKW?uKGexD0a(kWPY2hG;4TlNz zq&$5u9%66GfFpn%ru81TW+y*Gh(sUaaO!T)Ab%ore+uY&CwU60dZv#4vL8_c=#}9s zt2z;qhjijFCDjpy!l>*Bpv<1Igp5<`+$9C}O+|SFXAJ3}SQXGTWT54Ap1SHe7H`rm zwInE;r*|mQQF%g;de^oj*TW@&$0l`9o+ytB^$A&Cb1ewynLvrZ zb?{#d&lKUn=0d2)fx~nRFQ5+>CuByDPuG9MMQlumx+YRS_A7Mpw!3iDP%{9{G6u!@!TbGAx(KraaVYMUys~)&osUYCN zH_w%kZAh=(taxb_opn0W!UL`J$h^S<3sOl2x&ku&!*M5dZQ z;2@MvI)=i_f4DV!KcW@;#Sisa@U%8JG@FPHuDKHPl+t1JKPXE<`&D$tewXtfwl5{* zSCi!tYfI;<*Gw|_ltyUN=RJKF#7@pCx5vDehhItp0tu!Gm2I6`pf4?)VQNgdyrIwfnj7Hw89rp5_UF4Pkl#D;uLpU^?T-KMf+#Kkjbfri9& z&`wP#$21x%J-j_n1;J`c+q2dxw$F7@LYQS&b7n}c6Hh3U$GkC^Rx?ncN|~v@%l-GX zKg;XMULB5d;Q(P-n2z=#>;m0MeIax-`8PYQXBYgfh@+RbV#iAcW{OxC{K0i1g!9`$ z+OJm5HY7iL6Ii7cZL}4PQ$YZZw^wntM&Phk$E-ny2BQuCzZg5GU1692x*pp$zGK_A zZQHhO+qP}nwr$($-zII+q!+!Ji+O^X?7jBN_K}s$91Fb8##tnX9*_3T%S*>nHAacZ z=O?jXOgPI1>I^ADJ^)uZ}M(#o+p$;h={mcB?i)3K!VlklwAh!Y1g z2JBN$+2#}AuSm5~>*GmL{Au0uB%RAJQ!cuy{Q!;9H}k$9Vbw0)!T1ApWGeYawIc8x zN|5jL#>kFc#X~!#c5fY-upjPkP9K&?x_Dipdy5ma+A0bruL72wHaR|lfrU3-Wz?*} z=t%W23|%2Wn_`YPTPcfKXxXU(7GvIM8?nlcM-6uY5k|2In;)oSd95l|j2pDK`pDz9 zb;PY7o$LihwSnRZ7Rl^>;p&}%rJI!KuD;egd_I4!=?fskC#iJn9wc2i4L6rJkM#1R z?DjeZH5ltAPvz!3L}sgv=8#pOJR*Y6DzA8^qx9V)YIt+Iaed|UN-XmpD0%;2Hf$CN zdJUo7U-e`T?t1dhro<+?Jbx~GMYcrT&ELAZj&BjIEg3pZC$|fj+*p zU6j%kG_LB{fW!l~Lp~5gz@f6R9p(+*qQ6u$94&y#Qoj$&fo3oYQ1YC_pJ`mqg9kST z7Pqr|W=cj>j#4^dvPQk=@ZFRbf*uKeEO~i5zTt?j`pP+*In$*xSd)xluHYV74c@GO z$k&CddE9zZE2VF>-B1iPkY&~#+vU7j&H)W`%1le_bP>~xUZPbkM~}IF(`2+%%M2ld z4&Xo-@j&4Qh%^4y)8X2jJ_KG;Avk7F#OFCR=l1ZL@AazovMtysk#z-()RC+5xHgVL z+c{I#!ZFED7HurRB#N>K8 z-&cdN$nCyf0-UklMc_AaVold`i-L6chBI!_>m2_UT@y)1x|P!jdscaSs`$3!L$c9P z9VW=#MDRNb8l?Nqc)v{jibiAS+etA2Rb064hEW8}Qz6x5Z*Rcnf&J89e)yTS;$Zjg zRURRDmiQnW{tMoT30?Cey$+q$|oDV_uSd;LJ{6Ue?C| zT$*VyGxWm^L@)i`#;Q7uim>eG1A9X(#iY=IiL2F4TW(@P`g)3%ZLTMUFa!l$TF&6j z`gQ~L8`UYRk&?l=dy{N*t9CR-5D(b}Q6mPzZC_1_ zU>MotW_#ZuM>nMOt!-Mht-fmF1CZ)Wfo9xVjDz`)NOnCDHOC~lg}vxsO1D;>28^Bk zK(^INd)#-QSv(zzN2e9F+@LnC8G|V8hwL%!KGEk7)W$uxva& zmRcOR)6x1%Re5iooCfI9f5Fpet|8R%%E<$Y)b>Pb*atjcsTK<@11yrHdxMg*^IjC1WOZlf!5SjmkR)t5`m@qM#*%xfM?Lz z)`Dnc6l+2e-jK|VsfuSt^boFH;7n$vSP5+c&7Zh1zOf=;b^Z;$5LDw7=nG~gf;SYJ zLiUxh+J4j$qb<5KFavxEch$YG%pVzCrMLyG{S!N*l9?gSOns4oj`>( zkUYx*%!IHRLEJ^JZ#&iORKW@hAB5nOa5#Ms*9`-ZxX8;HQjibTdUHSBpvOT;StP->07!K}-RjDhK z48%%m`sELW9XvfY!}e#3V=wADUzc+|3E8tWCo?cPla*^^gQOE00uXLkt$en3R0^m5 z6DL_P+bToX1MO`@Fg90|wA$mYKlW1pfl}6Z+w^b`f1o7bkC9{s-@X9|UwKQ*o5pkl zU8yAzHoB-ZC5|eMj=i`HJ}sH{rg~)x$gV-lRo$(Iay<2*1b?DxZwq9FvaMN?+eP!P zfy=vZI7{fv0uMq6!(f7V-4EInTadH^J#J}kGXuE-oY4jvTzZkN=D{qiEnOOud;jA5Loj{QnAiRH#qOEpt zsC2Yk8nhe7W}O19l8x8jLt|x&`h?eNn7JO)2%w40)PP$qQA8h8t*Tv&Q(h|9`m_Y! znNfdDyt>kYc0OpO;ww2^%w~+sjIk6xBra^rVMQ4zH9V)>EoqatZb%gb3k5HNd(YW-*b)-(t6Q))mo-eLc%_#WS_)QCGL1)W$zm?S zbYTuI`>oN3z|R~`3l=eE|Ea0OSbSOJ>b}KJS@Qxta^T2))(D@St6Up2+etvVAx6@e zoR>Dt8G$N+kx|VGfOK!LCM44)X$QqpFBPTrQvIeD;c&nniaF`~9@d$52EL1?+hNyp z++Eb)5x}8!A|DsoC8GGU&yb;GxcRst88DAi4ffh0+kkNVX_17KaC_J99G7%mhuZTF ze|)3RCWA#zr#W$OpAwcf)m~Y8ZItTU2`dXLQ|03Cu-anTS>(BZ(QtUr&1Jsv1qqJ$_~n{v?_N{Zuw zOY$(onKZ(ydqWKY|Lb`wES?Mjw+$&}Y@BdPrx`>%7g+auk39dxN=W0_QhipW0is9&f7xT0gnC}iyOxQPz^8T71WCD; z_I8G(Wb%A*R^gCw?IS=zRSw%WNy-D+BhlxTB~)RZO>(BLDYHlV(jPfw<04*Oc6AKE zyn+{`(ie@R{Q!h9)&!WI&@vzAo7`jXnN~H-*!@#>rAUA}t0H2L@Gf{w?9i5m_CvT4 z_7+${?mb0wVtkGvz6MIp%)SHq;{x@ulSd3m|Qj(dbj< zN2m}l5rahSpLbkugub_+U&@iJeS0Ghh|hY}u1{MH$2LvsvM;O7q5$P+a6EjZ8Kq9w z%Ca@5rRix!lSxNtyD!KB_#iOl&!o}L;Hj2o32(Qd^ylcO@yAs7dH zGuSX=dyc)z$eVCwcVtloXEqsv3?PIw6qL#8kKd&FSG|fafO)F!*Tt=q0;LvCKfLa! zEw+5zc_Guc7wMjw_tYO^xXX{!dB?Qv3>4`IL>XG@q<*q-X7?E(<4QpWs*a!@_Qf0kpWhKLre=d=GHjT8B~~Eb z#LkO_mN)`b|Azor(>6d)^2y80XXMLAq@Ga@pTl_RhWoj_Ddkeja}^#-jU$B;1)x;A z{3(^Sx~zmhVYa;CgkndYZVb2rJ-5}^+>Glgsg!eK-5XGIIx9H1gf zN%$}=SE7c39E$+1Wou16v@C`b558DO>rGqIGuwH!XpO(N^)!SSE-)MXp&nur5Y0&D z-Ho-6`-mfX*4)G$6g#dU6dY4QU?lK2%IkYJ#782?q4TD#Jxp`w!Sx6^$TAfLvklS} zwd@B593mJH0rW_khA~(N#ivHofKTYEa6;_)rXev+qujz1ZE`lzXhbDj*O#7iEem7D zLd<}L3PG0>zSNuiMzV2kcby`b-_B2}&OpIVvAfjOYu<{GkWBtgX%a%}7x^UyN{iAv zI6uN2yb6c*O$y(Sr4pW#kV*f_n_B-auKcNTsj5&c>t0x_hTWt=E-TGn%~F+ct4THf zU%PTqS7XLpw`e)c!a^V{W2X59sEuBEE@w2_5a@JM@p(NvBcnN(FB*cUpQ-FL!XWVb ztt`Z!UaX#=mlrA7D<`3Iiwv**lrh^tq}?f#z#En~)yCb8%))hOmEmQ6keJByeC#tM=I8{rs7*Alz3p~Ct z=>}Fc%ki6E?g##NUW^=-x9H#Z$K!-}T}6<9H5B+FS74>=jW?e2$%?G5sLA^t_5pXl z)e)?a>@l@IzpLsk6J|1~e?0h1=wc;^?)pn3q3}9&{LB13eM8wXFxsD|b4yx!d6jPr z_|jEnMi_d5F-5GJ*=jhBaj%0fLZzDT(uxL1@)~E(zXfTw%G~Nd?{U13P*|Okl)0xd z!Hh>KAMwsNX$4sIPue;_`xc41^ZG{KmGR8`qd;jX51_^Q|KR$S%T&B6Vy-^qx_J%J zST1M6v>iexxz;&2*I%e4Hdpsie5a(NnuVb%e}beD)2M@$f9^F%4LUb~Iw zu<9+Or(#tetfLve$UGBSr(@he910_OypX7VVUuZw_fWn^`YKvB!Ovtr(fS75dF8}? zh&w_e}<0NQAmCY)EiQ= z#m{TS`ToYHtW|IIM&X4cA3n371W@r5WCt`N;AE?)H}&UPyMoik=4KH!d7kRqXZd!k zrJDf+%v6w>l$P((Kex|$%k_Muo zDzFXfk?u91cFIKO>DLAm9gnVIbNF$akt>Y@=Z9DUaQ*)K9+ zsqA5hl?t-&h}eq~57Gi%O_tq;iKgIY?$WD*Lld3@y5bitO4+WAlAS6J)!5p2V9zZG z!Y?nKYi~)q7q-i)LR*(@dG!-#mH6<}kV`f)bC7qenTsM~sz%Y#gb)uL`+Y}y7EL|B zo05eO^1vB3sh0cINqPtaLBx94PTc5A?aC`$rJ-~Dv9|OWbofg=(+(;!jJ3Q?f7K8_TNl@;G~#fl9O0fNF}FQ5iPwvE zmjGku%klET2ON2ys5L8v!K>lx9T026PGYF!OG1gXQ(KHU?D32Bq7!SQ_Kc$~1)@STR!8fkZAD5{?}6 z{RRZJMYT>aGsV7plcFps($ZvZ>kSEwq^Id;D1@)O7em%S{otEg{NAzJ(XmYx$5N7R zTb~U*Ors04sV5f{mOM)(N6uM`zOZ=zD*MxRC(ePZ7eeswB$=V)ky}8z~yu0VLSs- zDJB8np23;(zapQH%)oy_$Zn-5_EM`j`*=Pf;OJpdV!!t9{}!a&A`#2O%0@iTFjR0e z&=#AC(TibB_fC(|Yu*=ouz)lYv03TAFR*SQ*_i?SK!%L?I`W)oQub~%!+M=MBZjx< zas@0=xvq1|%9dqOCinuyKgF1ez#T>0t1)!60JKl|99~f0^VaKabEI5;4=w$=9KA_Q zf+HK7ea#2Cz>(PGNMWHZFP*kC5s09%9%N0-mZj7Y-Z8455XGQ=Q8QtDXvwZfPz0hT zM>uBAC7Ivz_-Bt&J@K4nwbK{AX4c5vBBYjQiS(PFlP=~v8Y_=2C%X;wfXPH(tcq@T zKxm-3*NU1ckM?;WSqs^9i@g2zwZ0s4rKN@Er| zFCL;NcEj|(qi-K{iUoJxDc$W)mVz*Tg$$5j!oBeofs#tT@f(67*-CE@@-`1Md|Tso zy=Aa>8knAOw^|<4$36XI^+&E(854bwzO9&n&2j_V*f8`OB{{(*1eRR{Tft<<^xOg( zPb*z6MxO^p5O}H6EQCPV>-GuD`qc0Z$>VTTbS6gKOx)UND7v3*z9V^?c1YMNyc2%K z&KzMOVQt#J6w?x%|2Q{uoC2+eDl>rEGO7zRonRC?(&#|61smC=QF{XZ&lCY_O|K$nN*6c zWbLN!y-}MP(^qBcKgL1qH-aw)#*3(;QOR9$v)3*2J^%BSe$fsu3Ra5|^t1F! zMS&3lfsJ0btS8zLqXhg-x)uH&MO^_Pv+IKIY9?$I{Wqk$ zI-GMVzJ<=};kAc4SPF21p3F0k(9%oLYx->XjQ)*!>#-3m**MJs zz7I5nN0?RQpqV}S{eTQ|5?>8tGPai3oTpT~?V{{r3XApg7>-F{AL+LpOfGAZX)GxH zPmYl9lyJ`1skgw!skeMaf$XGD4e(=m{TLl&nF-i55`;vd-Xqghbh%UO{DzCCpEk|6 zmu!sVg+5YS7%p}Z`_f6_-vH|kK#o6opVWRCW!;%T#%2tJzv{ySBj+|bD#Cd?rLOk5 z;zAV!d@NsDFc&(lR2t*bgOp7n#hHd06DPnG5G*sNRd`fBDepH5H*`MJ~F)D=A%q zmtkVcCT)yUg;JSd9QmZ(xJFmIFtDq#b6`Q2uTAENP6=w`dbNC*6BE>xu4zt4htU@* zUp*vw8MbSZLnEuz%;ZZJ4psX~Z~%Qjn=T!j1>L9{fk2lNWxq8$zE_M@wXL*aj0l`B zHa`hO?jGno!La(+XIZY664l2Cj&HByWnsl7qX-cc1R{_H{NqR>Oj?7wlrMk+g72Ao z1Ki*UIf$@IAWz%E0xdC%k@RbN{DLI@oAn}WQ2#dKNx}tfcpsbxSqClpS;r<(3DAEX zYaBU{j3jyImX;zhn35+UgIcYm4e*%!)&Ao+TrhQE1?~PQl?naLp`gj$+Dsw)scB0t z{qwZK0;uVf__iID)=9Aa93g+jLoXM1&vU$rDCYIelhNF=B~uBD3T3mD1+l=R48;KW zKi*<$vE@5JzG56kIjywpRF9cq&S>(y^ z-7g^f66H1p`W)aw7ST4%DIGSHZSQ&_!N40F9jFAOiVv8!b=VUwhj#vIe?uA#(tdJn zDc3R(YLz;s1QYUvdjfw|(myCI8Xi54Z7cWsMF8upQR?4P5*e(Fc|%EVRF+`$Ls?A(q@3a zq8mO*9gLm-g-zURRi%CamGbCcZ2yD6Ghu|DqtC$boQ5}tGA*KHaxt($j8g(z0$McAW$KFpj= z(HV9JI7V_Buh7w*5d$8+g|D33z0zQB@rNk_Juz{-A;qr}v$ zz1>8ch<;Y}7L9U&@`mOJ*_D%xJY@jQQ>z?H<4Hxkg1cI%Hy|rJ+gzx9+UqvAGM>en zg2Yk>hq$(;70#OCXsZ?I1C7YcJmf8?pH0?*^tB9sEp7eU=e*v zb>ylL9{6JrKn>*<9iOPHIso!D>RNIEQ;Wz@Fkq;1Wxu&H2(F~I`{JeQLqfIG0@WMcv1 z(S=u~*Z4@u142O_-!|X@R*h9B_V}*)o)vtDrz3-z7LndBBr_+y0nb;#Ze+Be`Jv4o zaC1xd_yJ*ap`bb}^9wZkvn9ZRJgOlwuQDALlu3nb)*JVx)~=+Z)Y_+x(9u6(UD|Bx zvmp`ZN~2X-@f+7(9I2ZF+WQg~v7qEXKh)Km%ocg}oGPKKR${xDp!6m`C|!vY0UZT1 z=7n3P$?BjINHv!@ht-_;-*EU?&5ZNve zs&LNGG=q^a@Nwipb2oO3Cg9X^SE4OUX4d)V2@`!X`Y5xrl|~q>#^dlOr7ShxMqA6- zoX`!0HO&0z`RY?qtP{iEIUj;TKXrV-uBc2)C<)JK_zc;?%ggU%C5Jtm`tnI3W1II1 zG^ErS)H`y3J2?EA?3d#Q;SjQxhWG+YSLF%Xio>RDlQ$klEf&AD01mTdB`$P2j})R9 z_jJEOw&*%9XxH%wk=9}!fB=sdt{!5?9CjeK-g&o7p|tMdZn%g3LK-k2a~c8S`>qR+ z5-%ZDnMunON;=gSM_l)wB5)owDZT`A+G;Zpk|3;623;ZCq&yq3;CGD4WgmV;zzWAs zcqiy|$ri@A7@I?d2=lMJrxvpx@(?R~RUQV9Kwa2alz6E}aYDk?2&qk(|5JXd{Y zckz_bH_U1l|7=fB`C}j3kOs!@MnFjQ4Hl5CPn>C;uLI>;_lP#j#DPiN=L7SB&Rqe1@&t3M=yfTkU9-z+e;9aFa@&HSjVQp~sYcF4VDbatf7g zPXNu5R$w}ae~GlogGJ{-`QLHO!`HNB<*mRaInR`5WS+gVG2FT0w!2L{s>g8JkkoC! zm%_A9r;_D=;2{hjf+k=-xL>VrUSZRmIJm}k>Cz*p-vZZ>{$3#~^aLf6>Q6XUdz9P0 zqbQHD1su^dXX-%rdqKn?L#CO_aufByKZ_=QbUK{6w|8AF5v;mBZZw3yvk{^Fch^fB9Kg#@Vmbstght9mhx&p7cBUyQ8w0qrbOWF%g*i3rJ?WjbjFHY6@2)yVc!ABw|zw>o;LtE8ICz=3siT`xjhl2`NC|``W|KmSqlz~sGnkT{zIArdA%<1dG@-osu?hC zfQfAPBao6f7#zn^%o;QD{=L4l^=#DjK2iGQl0J_+xAP)J=e2(Gb;5NYmo2RiezxPu z`x(w?y&yZ6kX@k8S4r}l^`?;)U{D({ukd=ng6-nIQ4SRcsu$^mQ2OPuN9g{MserR-TBQXGM zWN2t`ZjzF8qWN`9kXB?iSFlIG_8b~O$hAkw0E&Fv`3D&bF!~1oa)6we z74GYRY5=UGK7YpbiHwEe^$Qu;<|cp*K#M)?;r>^=SORFNGd_lOOw2#w&m)>ika!l> z7O*Z3jUa$gS5p8DQczO=f9p3g5J2<(HvpvkQbyQL0P8sUV9P+|Rg&f9K*h?*%PA@r z8M{DLo$FgcxPM_$>6MwCD8T&M!U{4#0L%IS;uTeuKOQQ;n>`P0`T*pWJr{q+9!%dy zwFCvEg{3vr1C!rpFaWClRPzgv&tx}#4n>|uVDD@#Y-&zo^*4Hu0L-bgv#;LM)3&y@ z(rC1@v(oxv(4<~jeeV?-f!P4I`eIi7TU|u+VDD-M+Sk*&lNZ*I-(voJZ9wth89+DL z{(jlS1Ap?Hy(y0~7c#fsbcS{z_P@wWr}(D;WCZtoi|SZhKPOam)zt9yEMOZPz&Fu1 z(t9vDF*!Qf0HS^qySe;KmwpH!K*QQNxPQm;?`4w@dKSMDIw;?+j6B$Kc1CS~y$nWN z=w)+$RWp8gTPW6NAdZf%pJKuU{W!+J-$T0>b2$Omzljv#HqFV zTU&v|LD!J-vQ)9`Yb^Q?T{H7+rj= zUfLW2{-`>mb2u>izu+IRll&@w%OK5vu~atOI9PzUHrxTG{ZH?wBlXYPQu9Clm@NKM zq!vd;MHkl5#Gm>sKlegMTM4k^{+NI0kY)DqTbU#E#k7pF{uTqWi3M)?nM~rFDo*$$ zzu-Hfn;x|OqSiIm0rqN7=n_8j5*j_OcG`Y`Kbv|1qOG&7|4O{9q?D|<@{^GLukdaA z+XTPSkJ|p1O#@90bxkp`=x@*FDJ>^BhhiPZ*bGqr=okotixbPMzp48L2%C(oKk22W zVaET~j|JL4V*>8_`xL0&&EXX+JqY*UPn#T_&0pg>?woFT22?-w5B<;pke=!f`k_|? z>ks<@AYJA+#vO3F^iQywx*EWAzmME$AM=^*9c|}yFaCuF z;N9+iX6^TX9v61^H!!n)L3YhLzhHamoj=%(JtD8~(7$FcDSy`aeWFn(dutDTRUdTK zm)8)jV3vW?yS=HuriJ>~*AR|tCLUSPt85=zM|pYAe?)*MePW+>yM=}NZgQe!XBFN?wqyZ1Qniq=Q< z7HAVEo+r0bpY*pQO*OI@aBb#y{0a`;5WX|7{hfV##-bybZxJ=}vlu4f2k=rtTJCOz zSJwK)Rp>OZ%Q97vcF^7l!QX0nt7OfuCHJ-!1SpIuKZ|@#05`Tds8fJ!a|{)b!sdrZ zz&N?KW~_nG47$cQi$71{F|`;=sem9?4wZh*25!8_j$xI)N)!HG5ih6iQzo zY%|D&zp}WfQ2d5P-p>?Hv{a@fYyNXDO%i1Rbk_Uqk=$UmqsUdVi-HF$eV4oXlkIwRzBi* z-h79O3$vFf>c6CHT+q%go&~$|n9c~0RRV2-wH~_m&J2T6+8s{Ofm7Y-b}6WiynI#a z)|d7Xayt!gNd!~5wzh8Ku)IKq2E3rUOpI9AN*H2MACj*vuDj^9u9#ZkD(|G zPB6d zcLWLT#(AKrwGmJxU&?5MS(6H4@dB1t`45U|O653K&P>!z7pd#r1zG7s)lh%_U>iA3YKs800q?-s&{>>!wOiq7l$Y?rfE>uJyi;@DaaMA5vE(reu>CdclN zmCZTr=&ryOPsdgTxt&mqtRu;c){q|8L(WeJ1s2I&N-Y2eQ%aYQVVou=3PP0ys_7{5 z!YKBQ48eUu+pRLgv*Fo(YO@aNq~~}PzRns$121WsEpWHWST9zomKVzcNo#X+3aaq|-IAk| z%~;Z0)1re64oA&cN0k79hb|bhEtubWO^T(}FQiB_%Yt!;pt!P`q2fr9eIO&q8aB-( zaLQ&jy;~X=Lwb2XV7m}7tWAAp6 zc*x1G^K{%;L1LMs@nPVNN>%T;IE;kpZ?GWKOMm^G)i}D*)e>xLfMtS^WUok{KsvOMl`Pv(iPyUUX;j7g`z!>zj&hYWzPi&zpdT( zMY;i#A9^!+8A)08y&+mUtB@CAH?a0GN-ZC>I=;GMEz`}n%J9YT2@`k7XlS}z?_%SZTzUMHd<-cuNH*Ms&i&y3x)jp?p52BeLyz7@~6CXXY z?u77Y20GgJ=Pxo=R`9uqAKvwh06o40=O)rnARoHM zagV$AS|pU3K;t_Z`@sZsN?j@4WC*j5m&0yXa5{=qd-NJRH!;~OXCwFRNP^Vz3AH(j zDks?Ir03pn?BTV2B91xBP9eo+CZ*clVZ%HyFQ1_1L9E7=Py14Hm(D@N0s`aQIOzlr zBlaJpw8UR*+aSlNBe>F*J>N@_`O5I%*aHTpNbp-*-|BJ1&zn}r(RH>0Bci@7HSjGX z#x*^*9-|BbxXLTu8C=o)WU_*gT_E01;8HK{s7*d0R6}STtgQ_Tq!(YpgFM_rkn4VY zz8l{LC1;`RIQAilZ|34=j#4ezC1ak~<67E?gLV1>Fo%`z8*3TII{pbIM%G%uwA`Mg z^7jhRso;Svs2`{M>1gP)g|4%LtB@*RO23_p)wuAF>14qzEhR2j!)g8;aYqd_>my}y z#AuLgtDeGi$zt!?M(hX6Yn8&;VhPO_y8E77!l>Wvl8@Ror@WHtDr+s{bisd|l%?bn zN2!$Gev-}@EY7+xw+Mn2sBP<2jcNp&kQ5TyWt#~Q)fYI4T;d-fEEq>B8+MIUZch|@ z(glqVVo7&F)v_$NN8@Qg>c?QxzB8Im^c96gkc)ZY(PF4CL}qm?LgFkkawvHCy;0hH zdqlvFDP6%gg(CG!UXM|si}g<$CQ&Dif<}8_`6?+#X(SV3ktmL2QR4G5i==lWF=6jz zCLSWUJ_O8ax#O2OQntg3P7*&HaiJV0-2lFDsPylt(ULu^BI5M*bkW84=D3H9gMn>Kf;24>b!K=+!HuN@9h8gq^R| zVB=hL0BS@U1J4}}4E|QrhIQ%qHZByRIhe@TxV|i(VJbMHVZ5Gkh3i&LwQGS?R0$ zG-}n0PA;}O`$G~QxR&SfaoQ2AF;h$rbKspTta1g}40UZ2kMiV^6($CJY9n8rNu=3| zx&TM`S8n^osE=xD@?c6~f)~jN*28cm-#Y0&*qZ-DF>ZyvK3pUy51;XP6E*-Bb%}Cz z%=?!opLDs1{IjV>ET)ma<<<v z2h)zVQqxd%8@Q$KIC|o7MQOPRP1IqwT`AgA|5D}?w1WAquHVJux}$Kyr`4JMD@|Fs zNIiYJoL7m8Z;ryyBhea=vr1JX6(J>8?rKKUQOxm?u+(^vnI`Evf4eq9Z%i3u)p2t4 z14_qGdBru1ldL+`bjxUi8v5Gw8Y?-jHkI3jAG;5}P+{h_H_5KDyU4Hgh}~66wAOaD z4Xxv}-L}1|iUVL@u9#k9)l+AUYMN`-Id0+w+Ot@?HcJ@QXUeoi`=W&28tgwu_Sx@S zXn|G54yW?NaT<7IoiV)vXVJ_p;@W_{z_~9&ID6W=ns^Lp*qP; zASLl_-cU%6=t+g@<|Z>UW94Isi5*$vDc#d)~n%EYFfmzt8tAaU&IQLD>~_v78p5CYfBu8?QHXNXgE%Y|kWa zP}@O;AW!>&$9>HW{nHV(ATNnv9+1oGl?mL9cBy4_kc5Z2!K<%ZNdGddTw8zMjg+b| z>_ETOP$gZ1PAqkN(|8VbM{~?)t>lAdnWixA$4ivP7>X;qoU0vTQ^gbH+{3G(&3D6v zs$lklL~_{ZXM;EB=jX@im?6l#N(ct-2Q^8hWxu%yI z3dQNu6R^##2^%rB?z>OBD%}BkI

)k~yi=sR*eq7wU-5)p~w(<}g=j*erE#tH{z< zMcg{>eDY27V?$DFtkTD+H606*x9OD^*0VJ80fD8Ce5{w6sD=?h^4Y-ApKH|C(cFQj zLov-PKe5=X&n8Yqx=_8L6^y{Q_TjN1Eq1*#O6p44alPGR+bynREm)(vM+@xA&cXwa zMp>XP`^ZQne8Y|Y@J)T)$o#3*@82L!`>a>OHar|jz-LcbC#*#D?xhYtt-o{RcX=3! zS|9JuzZvU7QRzRz+aO(nBe`n`Co=H~N9{IxMWp90Mrxa3aa?J9I5@-VamU`yQeU#E zC=M*{0}Y=77+r{=8|S!(;prq&on{LQYP&W8Z#*8}HkD<6@y|EM@A}`OGXKjNw-ogQ&sg1Mj5A>BpCj$*{4~GdfNit*)r2t4wWS-4SNZ`GejR z$+Fd;wkNkhGL8tm>U2z2cziD+xS)TPWlNC64F7^O8H8?6)W)-5J6>^@9ojw6WsN1vE^PIL)3bc4&Ew% zr1fQs(eOL@%<^ICLVZ$~xT_?RDJ4Bi@9Qh&Q-yE5GPXc(U*Q^|X?7jQ+Y=Ah%xnk~ z8egY>%XQY3vfFco`@hT;+9r5Ta}$d&BN!nJ=5}2Mlq@GkGch(D6d%FVSUolWG)IO+ zaGXa_cb|8<k9Rs7&)OTdpqm$Ml&DYW!kb-MP%W=6)P4L$!_V5RB|rx*$?GB zV@Tzu$&Vike$ZjWLX9t$qyrS@gau$>b^uk)mBYHf4=d;^q-{zRqjs)W&B_R#uXc%*U13Y@k#5cL7cUK< zD&1v%VxjTeb0aZszCraU!MJ4oa*I$LsA;^E2Y~q=U%>XTp;>(a4e_86mJe-}%4}OA z?$jCqGT$=$np+u2>=Do@cNMdzG56PvZX^kn>sE$Qj6HGf-`z8EkDHty(Gmn~(A; zdGgz{R39!&wTnzs;PslrlEgv1T5l4EezMKm$kyPwSiEkU8_Bp5#z=OEKLF4}0Edz! z*1haf3Q(MYlW#4AkOgpC+>h>T=XhQA)-obqv838Q+smNNQ(8GaaZmjTlOCK4B6%a0 z!zgkO@f9V#UkEAmX3VkdTUGPrm?8;s5ciX?_PIp?*?YtMGrux)y-mlyG8z+ z5CAK(M%{;fm(8aJY8*hN{}4>Wr`sP(7sr(y;z;$H5N)*nFyhrnF1>7$-3Sv)$1dO^ zrfLrb5T9uT^|d0<3|aIbJhMP?KF}TQuo31yh_OTw@cDNe^5Cx|QkvcDLi3bj-^F*$ z^-IsXf!?j?7gsH0bmujX_`%Mw6|M9GOy^?j)lZYBf~DMAyNfA>p342E^Yb0!bcN!U zb5fELvDY}Huca`vg3Fx4h!~dOO6fr6GnNeHI`{kQ8gpM0pJw**{Cr|XTNKYtu5rQC z8foT}&b&PHIlHzA1}37(G+@gS(Fw}EEise8dK3oJe1x=dC}aX(b?tk2Wb+6ney{~~ z-L3ZxRa64ie#-2Mbh&d?gf!ON0#iaxaP&t~np4e4x zD?#xAhNb}9#}i|NZurB=(;jWH;KjQhXr9j8m$%t9f?~*l(Fd$(lw=*r*SMv7IqORT z-Q^R(Oh;?-?8;#A;{*IQ+tm;Vaigp{B}SyZ?NNZrOHN4jWD`yPQ=VL!G3$ zpRBdA32*T9AB#&Z0A()9ZVKaYoX9aZN24PvGiEg>mN8A&<`^GGA2O@@6Ue z7)@>4SOCW6sGD%h*0zCUq;7O?P%9+Nhln%$b(f5my2Mo&YN5$0p#|BWDITdo1xQG| zkb270A9ahk>6sNVbAug@acGhU1n;OYuH}j0u0M<9DB!-jqA=5?N&Nex>*{&5WFX4+ zc=Q_|i>ikljI$0&IC_;>Ty`b1|Gwj-rnSImX;&ZL0&;*2iOB@tG*=&2$9&XjiAgcu zXC)@t;FC&W$Dp$11D}YHzF5t~5vx?cPi2g9E?B;m?mxbO9yntUugofd{-e@1VPdGV zgLf4^<@d~`+R;nZ3pa*9^aLIb*mJ7gfk3*NlCWwuEdGFw`NgbKKcLzD9B@+>IX*4C zNtc(BN?w3V7HqJ8>_}}4(zrzmGRG?{&Mh#}=AsFHU;#EAK^MMh|6UURsGFA-LNOCBN2#%9}F@ zp;|FE9frjb`>nc8yO4%dFON?-`bFg)pN9zMh^&~RmM7et_z2@F!=u1@+c^gTI(Y~XgjWW&&WQeRRFM$EU3~U&{FqDM*7k;j{*F*ja^n$r=PdCVgEee+ zCaWw^v5M-u;o6*zFS{`J`M~9|LSfSZ5^;!G)b9%7;wQxC6Twu9E2OEpx+U2QaJzaC? z7c%+C*lzd=P!!g9lJ{CwWnpW)I8&`xZgSoUB8ju%n6NYA^x#kZgE-MHVO&xpk!Gog zaae;fY1@c-K;mk~r`s`AO=ECTR529eTh>$H64h;=<)YPmlBM5oue>5JWF!?(MoKatFT0%fisOcRSEpy57SGj*Mgs**@F8%Xp$mbtGsuw84Q4NG(+SsH<|_$A zvHt;GK%&1aj|Wj&%U|kl>{oW7=diI=q7;R*L*KEmx?jZO7kF4qyTj#XqZ^0vUAvs~ zuJL24uT1eOV$ng|*LuMu>ZjYtP2sa8l5BFuG+QPfBXy3mZUW53qn4Hxp+LuXN|%SL*;`;4`&Kg?J7rd3Ue<^3POXjrSd1QMxVsO&!_C~|?lJQ*sc z2ee5$#b}w!hi#{yU*XKpD(`Z*bf_rZEM(A%x5DiVx;MZ8OQ(mg?{Ddnca;1WFL!KL zif8j~3^}Orr6J+ZLR#*40{nC=B8b)x+5>8Mr~O5Ew<6_64mN(UI!~T5v>LQ2THfgApZT*&=i!03E{2@I&MsFqd`Xfo8gZdhHs6 z$?{i*8(zWLcAlFtx4CSc?__%nL@!2}r2ZV=L;GSa6*iqr%9qKsVxuNkbN05?qy0%M zbzL(?OPkuWm5+sZ-x%42CN64c&CH~zTkcau4o}agFtWlq4v z1aF}Eu$rI`x-=5-ht22XCP|TwbWFm3D*GisQJmphC{|hZ8zW9nuTbfVaE!jSqQQ3& zQ`_azh&(y3xpykoj)uB|`4@s_*1I02E1UzJX#AiyM7i1GPUQKThP+30MjAA7(Ov$1 z#d4!pQF_u9I!J?x5Ok^*qDve!PU?sg^+m$7-Q+r2&LizcozxjwyV&wyQ}on7Kt0!P zN;yq9EYC=|Q&`OqE8`J2W^@M*OMdBClqwgCrFmH#tZAAc#M>Vl0Tc%})(NNls;~Uh z@ga3xu&J&{?aHsxQZ+LomuVP>WZ%|J^^(d`OMIU+ z)=mS2*nZl5t(GKS@)|7E*8HYtjTi3}Di#3#^-I4kHgN-H1kvWEo2=)!-2o4O`m`k@ zAmgNQYsKhVOmJh@G4ef0mt}Mfn}M&v9{Xj_PZi_g+01wT38r^aWQJXd-5QEiR3K(T ziYT=qCeH;*)W(CPSVQL|TdxfEHB&oINRpeWl;KWgr>tK)3ty5R;Xe!|wq0~aQw?~@s z!h!z&oE|zk%)CHJEAEiXO`Up`54Kn?X)}1gDq+sj1w!%$nizlZ_iLM%q%a2!i+Z8? zlC!L6?Q=ZJ0!&>Lc=+-MHS6b{q0^JGlaMy>y|d6XW_C_*5UbMFbb*kmo>6Eb(ay-E zNU>(3?g6fOh?Y=kGrMNie);|z@iQo@+nzLWY9z_v4+t-plmpd*ex-(2kbrIQjYt}F zB!so|N)#;O&I~IG64lS9s2Q>3d~{O)lMP8`7RB9QOlTwwGUTN z67w_x-xhkq^VLuuO~m*KxV|bU3C$lVD*BaNShi7Wchzf#Xh8Zncz!BdazslQ@xhXY z1O|UvOhk}WAC1($f=;Zi+d|{PA9G13KvvVUa_8r02fwG-pK{MnKEHnd(v2P~)I33n z^kb}_c!L5PA%YMTdT-i}2W!04o{++sU7Mmqz9&gk*gD1tLbX~bPL)eRnrtZu}W2~rZB)r8hSk045yBpSYlgee#J z@FgL}7$$d9C|`e)f|L)%?@Mm<%SQ|_yvT>l=Q%vbUrW;!gOg|j(?1QN8*X!!8Ve3pSf8@R#Q0nHu;=5_3ZY`~@7zo@x(+yQa#M>6tkk0 zyL4WSf4@U|LI)gsQa8}Xr-(i%?eViNyw_FWZVocM+P}=a%glW?>M6^h<(cK`ZLl>b zSeUVP#W#@RK;-sU+IaN$^0F+d84nJ`t^+3n2dXd_?0;f?aoVW}ls4gu(ddppySHkN zJtay^nc#AV!|wJ8nQrxLf6>JMh@|zDb4{rPyLm8%!)x*I}2*I(k-^(Ft2 ztnH;wrYrwhXhFEKd2t#u{hlU*H9CqddE#d^+YbI|!*2FSN{Qz>aU$evGNCMB=q%Q_ zP7D#Cpy>Njy*lfwHzU_srzxx-J`L4SS3aj5 z&T@R~aVzvQ0K-OJz-+{&B?~^Zs?&W_!mOIgH0BG*+}V^=E4DYMA1iy0-Sz4kp6LbZ zUT*jRCZt1W`IJ&6U6xZ#lga^G*4vmx0%vkuK3jIeFJ2ov$a>VWJq%sR0+aL3)Sa;H zrQch44>i^`21r|LWcX>)oCZSXM(S9bZ7+qo^2&_Xz-3~96WZp&_r5h)ETA+!c7DuixLDfZtz+Rj#rRlXnZPj!J9$SU7vnl=O0qAc{Kur={C^tpOC^F=kDe?{XZ` za!p`@cR(4i63-X4vCzxP23OwbN(t@si$~zzg!1Ki_+d6oWnEHz?D0hdldp>pWy&ue zN?hK%Y(T76W28+;;6>60Jk>}`JbF`C`I)i{YV{EB)0ZnMOmQ?~!mrr&jOEVI5Mxwo zB@<1CG&t(D&O+@*7fFn<|vdEWHrd z)Z*45EwOmGYEOE*6O#yzC+K23NZ<7fqU_&^74tm$a#E4dp$PiMyfcX5vXA89YeGhj ztbsa*G|lZ>ujy@Y>OVCoOD`jPC=$qm?w5jXJXX&c?=l2MAv+wbVqpT;=VLXa``MoK zFSy+$nGgtHC1XL1OS|FmBM&8XRt~}mKsY3=c0~lZtRkA7b~CCH^_8T>_=n~E>^#=l zfHcG=c>uQA^pt>`8>iu8zQsqD#xqTjR`{pp!PVCKMW^8)61O?2)K?T1=Tb%v=U^ll zg^uh6JlZ}(xab3#K>=hiJs_hbE(H%a_Tw@1v8&}AsJ<;{7S?Kka*m(|W{f5hne4kS zAtRMaf`AgMb+Gp@B!rt|X!aKxc3xL_rMx?oycE_&h#LmSAyO^%>OOILlM$-JF#F}g zOFfLtwhpDI^GzmVj4PUG4YBA0fp@?d|`Tt>!XD@dVOtoHiP(h-tKjb7s58?lzkK~?2<%)YkO1H z#Oz%w1P=DO!ZWr)ViGVj784_64W)SE!>y^kN$wl=;g)2ZOHS3kKew~&`L?Rly2RBN z`wNFV#R%@^0~(LPMdFL<)iMVujJex9NoO1nts{bULFcR#@QQm&K9*;f%v?potrHsE zX#BBi-5E9ZRzbN4Y0midcFR1Smto+_!~bfMp!~u?-L`++gABcfs>i?RSeD+nKZl`g z`u#5daBet4b=bS2qqUsJtpe$I_SrH+1MNe zCXEWdNOPR1GtScr5b;nmF*1h4#J5OfjAox@&o|P?*CLGB*!M*m90-t`e{w9Pw^=Hn zQYen-y~q0$cOmXi0}my-PrgZ~$Y!1~M0-l$GvElTzGu_rHOePzpGXz~N$uhcwa;p9 z>RQWF zg$CuCm}A;+w5}Hn57dv18^AK~ZfN0*!2RO72~!wV=xmQZV!i|v@8p8p(Y7k=6qrox z?|s>O6sHT0xGA`J{UH<|a<+0rz+ay%aYJJmzI7LT?uB@2Cye1a-G{uze9wYMQhOnI z$mtGSFgXv)9ugWGF7~_9NC^TSv5P=VX06}0;NTod5d_6O8b!GzI{p!ov%k@+_52oj z@MyVLp#bxy;Ck5E1k1-Zag3vIe`}Z`Ud9WkZ6B=VrDny+eQbMDV>IO`aE>+jw5++* zn80M~nKRAhBT@e4>)zC-8fl2g+PphJK@d^ruW!Co#+7!wqo0u17=DSNJh&k6$hvHkl4W=p*9%RF&QlQ~zNXQ}6?Ljvx(8)rHYc8w~jyhs| zLJpYK)-KzF+-%@;g)jg1#f2v{(zJ~%tsNks&ej0WeC*}k#n#GBIOp?`Ui6}|4E)$C z_rbS z1EFs8JoPx!tkp7ou8>t{{l@WoPpr;z*OQs5u+48`ymY!X`>2(XL!6%)(^Ezf_Uvc8 zs=PabR7kGva=G-5&rPg+p!6FvjAMr!zfZ2yh12!D3^EFjWx^J1pNR5=N)dT9Do)KB z>sV4z8ke14wg+lEm9*HcWi(?i%D_aZL%%K22vR{U?o&((WgA(Jydm?YfJvdpHsf~| zKN=_PjF$;B-$2Of%tO7_Y%kIP}LcY@+UZ} zdiz->U$8rgS*)s%cDIJPa;=PVq@fbt|Dwn_tZ=;97^z3w+hN`3%tdVOB)AX^ZTDUL zmf;TX*X?~SLvWydE)t_=A28(tMUJM8w<0MW;H)j;-yS~y+|)1fb&hI=w4h#7ZD&I4 z3r~*J>Plgr9}V45W86iHn+_XA2eb+TPf~(k)7htr4!DlYIS3`OC&Y%x<8Xim&!(gI zyFhnZc-;3q%J_4EBe{aaj$JceCCh<~F3%6Y@bghi+=Bj(t&wYZe6SEyt)_B{qm75{;)UHn*TTi7- zKlZ{C8JTS+c|u^ZY=c&QU7cJG&5~Yov?F zBq7-ZwU>xcwf-v(-^AU_9AdO`^!r$Z2?0`+PIS*{JY$^K68s9fp+G6`LDHrZW^n9d zlOgU%Lta6#6%2%&bVV!Y3x_4H-H+JKbHk;tNqEFIl7=$ZpDDH-R>O_>3E2*908#ed z+(u`DvR1G$sl6IXqSLn*MwqMIeaoZPEP?mtzu`2J)w$sjUV+mc-w9(HG(^Ali5}0) zSu763nWxKEIvL%y*kgS4z9MCLP<;9Mg+=mQF&K&E+cUkDOnw~I@PmX0flvE=_z>Dl z(n)=G!a&fr$w#lyWEPNy|wpyTQA$29SHdN(6&Vf8HY{n zp=Gr#8QUg#%$(aAPLJrs`16K<(yRUY_yAXuJaM+p7Mw)}yt5(80%=!>QT-oW8%?@p zTM!tz0~da>9T7$yXIw+Pgz*yvEkK?$$7DV&FcDKaTEuZ4E$@HNb^^2$>YI`dH-ELOc*$oqKCn`foY zL9Z1rIVb~X@ZfR@FsN_C_fS`iyxfVwlU0Uu6o$-}swamFDVlLK{YhUimC9r_I?Llg z0kmWdVLl@q4e^EYNjmHHJ@!EtbMFnr6HoReng4Kq=>DkQrN;uHuPSF5hE+YZep}ZvgERut5(+fE+x&{UYCgW{)Ex+!~APTp)!v>4QHq zo~Tskyfwg~R#T5znkYw>HgBlDF7;WAH*}OGeE!uWrO1^x0)8Y{KCM$wo9TEWUOW3? zpKLbItwZnX(uS^OHqm(4K=zVicL?_e{jLG1tW}fHQ>~xAwSW^}Js@cqc7L=2n;h;} zI^o%FcoN|{wKZGA8<8BD^%%h(*qMrRsovjO=qTjLN1ekK7q6Sn>2czSu5TJT=~g9J z$MQ?I)i_+f#B-4B!3f`&yfCVfaap}YB!MhYjJn{JJYQ%w#0V$C6Un)xU+|s0KA8bh z!ERhQlc!u_*vsQY?2|QgNGQs3Iz3CB19}Iz^mvrP6y^W}!%f z&cPCXflVKTa|pV*zDFFOUuRyZ(Z?h8zROp^6Cz45n4=1}>Q0=-9z3e<^19m3hp_g# zbUMOliim54`Rr|Hg5D2ZrpxCf$mOH_P)4VneD{MEN-j8%R>-cUvM;z*t}}?zl?3nD zMmvo)B!IK>r?Vi4fmmhd<&)Y+15rZp3`F5iY3um%o?o`FXODvprbz~(kT7%#2qZuk zS0-^H6E5IN>cuhtncCX+)xopvh?lbS$}&Ou!u`btLZ0U_3-?4+=B5rTXe3Wz6Kd7r zlC!#QDpo0o+m7?3GFo#`T_yP2Px1=cn^nD1l-TTi)mji{NFcWXRCSOj!c)J{zCTCh zQlMjj2`8Q#>61Du5pN%13GVePO>2QrChY+s;6iEQvc=3`jBm34kipI@xvR|Vk>w)w zbOJg~uWmp@1*zqM*A8X^CeEJX<08EX4N!{;LNibB z*UY4`Hy-sYGA8@l=&avO6P%Fa0!j{otPwM#Iwc28F)bBJ^C=xchtdN}{7iY1g-|(U z%Rkso1hEN+>IxPHS?9DF$=*?`cx5r2G3U-mPY#w*h#M*{Y!$Tc2nyRCb|Bg4cEuBc zpl}qj;S9}RJ>yE154L`DhlKT65i>?|jn(dd6i+;(sGDEQz9COo_UOttr4fu4N130f z@<^!pWks&>BnmfXuOtvFHa+jrBxE!ox~MTHILDm{Fc94A%s91E|K-}5ABZIc&t^|j z%dw9RIGGo3nU#=G+o2!&g_8z3Tq^5&lkZP=w%Sx)k}B>||Kn$QWjbYYV4F!12E(-s z57Pr&PkgMoHsZ&AD281HQ(J`9``mEZhu{iMsDkK0Ws6j}Kp{M*5ELhKz|^3XrW{@l z1~)VJ+3_UU63>cxauX0@yo7?Bh|$vNN=Im^twb_pv;eo`vQahu=P%zayT_vT)N|aS zsWZHzFy4Zy+1k&(9Y_p9+@XfCc5ZWA9Npa2=^Z?>n^Q@Vrz$}1TU}B$`cye~r*`j~ zczm~D&!J*V_NN_b?ijkU$%8v}`Y-^ed}i?F>lG|LGY+6zgcBDDqBm+`EF$Q<&-T_M zIKur{?1Wyr6}pi~i}L&TJn=i}GGcQnl^*2e-ve-Dv!{NxHdd*49ZI67w56l+Qb@Tm zlxaWE<_8#UFJuM^!Y5-LX=*|l;%dLD^Q0><9Ig$gG{WxeOiWF;VtS+iU6-DM{hx*HX$=hNo={M&u}c6T&M zfWuuIzU;jL63}7h;iDUU%NG)typW7x_yj1x??@aGVpgK$*o&js)}PNQz7-lp&8o}G6mjX_JmunQ(GcbN+ENhzVIsPd|6`>sa>c75JD>ZH%? zo4lxTmE-d6D~{UydpLCOE#?DK*H$xXMec@A-{iJ?{E4`Sp_1d*q7!dkc%>9K_zp|? z!G4$oWnyLdW?_Nv3gaYDrX=X6OMl?iH14)a6j%4cY+yfIYPA^rOX^}Ueo)2mQ~vG50lwJ$+Dc*tiq zyjf2usq;OTd2S}zs-G^vx|gm30ox_yvx*Mx!@vNC3%re6o2lG#Ne(}#7! z@;;|yNuB@l$3)LRdKbNqj`9_HWs9UXIo-f6fhIT&*=Kk7cI@oXU)lxP2NH2T4TUb% zl*e`W_kj7f-ia7of*()L7p2+(n1Ttl*py^M^pwGPLy9&whF>CQ?x?nE1wtvWGss-K z8A4+0>IMa<8}QK8=shf{^_+88(lw9H^Pgg9FHm7^i=l%UnVtQOSVn#L$U>^qZo6Un zR*0*A{8G(aPU2@lOYf4hCax{l_DcAS!u>^%r@fr_$>!M=w~F-8@@#sj{i>Qv zm`1w_$T)Gy%Is@m@to=z6doZ>VW9NgcZA+#JCiY6vc(%NTfb%#0%a;%a#%)k!UgTi zC>Xh~qN50A4qxJ2o-mXsrH0tr1k!Mycj`5)NN&{127Bd>uOb&5Z7vG4BwocgBantx z-_?qlvEyw=^at?_;0hwSW!eU$Rj>cXQcZ*fHL~ z4!-mzZ0O?#-z&zRX@hQUZYUF$9uBI zFOy6m$a7V-NM`lIfU!~|>81Q}!mIFMiZXqqU7-?x%|3KbLhOf10A^eez9?Ou>Blvp z=eC!RbP^FUP#aJ|$d;8ialC?7d~2y6@eYzGZZg0|u@t&{1*5%P}B%*aL7Fp6eb7~7B|G0X{GQc)D5ac^`z zba8nd@w|T9BQeRGcutv@fKRQ|SRKnYJ4(r?VWC4=M?S8{q1|Sb{&F>D>^KI_5L@)a zK6P1_l}XILMX6lN*vDqt`{%~TGvi(#ZCtm#Kk2~YtG$hGxAEd*1wA$Hyku61! z^)r$5DU&O|`zX-tu3PTtI)nr(nVxRP0!&ooIZ)5Ycy7s}?K2o2l#-kDOr=kzQP9(pT-5OQORT1 zY_ljOBZQpmHuc2CvR+WxG$D=Wp+N{ftWeviy}GNmn4yS$|QMOq@L=av&B(DoG5d0##b zmA0=Gz@UsLPx?C!#QXgM+%tNYTB8*wOutjk#aH%JjXh)L zNS7?9QNqCit2QnL?XS$^q8J1!Y>!~d z zLma9(PK57rjLCz0rHEy`V=b$=5GA#Ann=2XgF6FDt3Sur0R52`gQzvT;~e zg`~sr6AqDm zw=?e-mv1JE@LxG`+@Ln7>;rs10TgeBU(?tES9OQ&BdjM;>}0Ix(>?XTufeRJo#=OXoJ&hA?A*e^weN6S3*gtt1J|5xzRIP z;J#Ca$h=FJWk|~rC;oNvNJz$ z!!5sz$C=64Wu1DDdFPm@5;Zcw^Nnrf&B=rHY&N zs=}NgXn5Eq09pG!$rpi95-dAMqfDHFO*liWL%oAX&Hl1$vw{8h?35EWa74ciUdE#H zERTFDW@Z(3dhoo+;1z8?Rel`hx7_O7n$o}187cmMZJgs*B>R2}#Gr*m$3;1!(Q)-gdou4Uc z(Qd6+-MOcM^a-d0H0TF4r#m9w)?%9RCrod{{$o_d*ADPK!k-NB87&Lz(~_$m$$QpW z{v|Px*udppjNYq6_{Y`@I=mbu-INyNLZVNJ3GM%vf~s@|rLV98B5gP%F~8r}kSO7oua*&(R)JVuE#VJ{We@v?!dciQr?!9)l=Bwy@26(G z&s`L(+DKkS?FE=6i00d&MOI}sKFnwpS;#~uecpod z)Z_;T*cSbk)t8Eqm?jNGj^e3Rwz1r3?jVxsQjE4t67ZaR>=d-%q2Z>(!W(>zd{3>s zhSuFGU`U9gUR!~ca%su9(u^hrrcJD3AMXOxl}0N?e&}k+$~>|kSG=xjnNIh&42O6Y zZZ<(D*TgsEYx2AT)=XXZBzVOjE{4w)vPJ|bgZ^KGRFPWB3Lvx7f@&f}Yvybg@OXrlq32V3*-^`-}Ah;?p3 zpMQ$4E7gMU2%j**tqQ#e62AFLDWlF)Bg?itR)4`Wu1*j@)Gl!^_r2OPkgbos(pNyU z(X7{vQObOdIZ{|w(o-{b?!Fr}t`1KKX}WNlSu*dc{nZr~PHC}>Bwc2{nfEuh?dayQ zE8iZv@Ax2`XU6D{=|VTCjnz?ATkH^x>gmY!AQQa0r=SVy9frSI^?P(dLCJ1CW1eMx zL`WvzS2Y<0j3uaSyE@|qOfS;BKzz}K75@Ic=B)`27z1GR@h#ZuE{pI^gOXog$gfk` zi%I!Q+_W+wt$?`#==`t~=>+I!8Rb>)DrWGnKkos*7w7(!NQ<;X_R&F5`6-_!Lj4X> zW>^)4yC7NBpDnwW_Z;MWdcE|#+O%J2F%}HfH^5tlLZRJY+syyxw28K$Q(iOFtoQ~u1^~1sbLv@s_RrIh|z;0Q>XyrEU0}186>ERg+ZRMS4RnA z+W~4fyVonvs#{qBJ2X41b^>lh(FX+=_knvNf%X-6pxGDGNG2~xS-$5@PFCx=n*hyr zM@zALen+u25ix#@iqCU>*{n60?S8U;s*dZB$UO$EL0A<6yG$4pI8Kizu^zGFyEYgX01k1B^dg){(r=M z7ZEiFJ2vto=G@-Cg>(EmBORYwuoDBOfDCp^VHI5mh9O8=*8{Ro|6yLJF>XDDt#qyz z6&o}F=G+rJ`59&N;Hsh(wT}yIw1#icJrBoe3RSENq^2$?7sPpYYST8%>q3Y4|JrTR_bP$YA;O9e{UxEYBSmlzV+fEf}Ai@+3 z^N(yV3+@>*8CrQ{9c=W_Bt3@`fp5dWb9Zv}GN59~A^x=P&!WppEL71%q-wL5Tsgm! zA9M?R2-$FmXgOe|5tI4Z;Wy2nW?xdQdqu|mm7^yrD1)@c^d9R7@9GFk>VYw(ezTKE zu&5;5kM-|pl2hlR_Enzo55m6|WjjYDi?!!nIW5bU?<`&C5UR(LTAs6M+lyw?6}|B| zz%>cq4bNA)w-5u_nJGvkEbA+_f)*_Xpx7|BTE2OWbxZ$NIN>s6Q;eVOrQ;uk2vOT1 z4qn0*3Z<2yzfc8>`eNv$+NEx*i@lqwYEC=qGqE2&8pyM?3yFCZ+Tqs#WXDsKhep5a zkYc+|muWp;?|z7g9swy}f-`<8OJ{S99be76m z0`JPUzj1jpn_K>(m~pjiFb|d~j-VzPg!Vtv@#XK+TSQc{rSN(pg_e<2kpaW{STc-& z2sLF(r3h|*?>#0c6%nwfT7*uD&xY7vjUpaO&Kt*Pa_NiIheNw(YfJHT+-8L9$xh} zt&riH2Pv@6Qu^s@0C{)wf99RSy@$a4r>#yuc_B~^QXns~Ma%bK!w&k}&^~+&>G`c@ zyBreAMDIxuTN%Ol0`WzsTVPi|Z$`lsWZKFZWfX5$m5`##KB-x@pb{}P0NuDl6<-8n z-5@z-<1DU1YBH;zVuN^KRm;dKO9{b0b@ruo$C~2jo4#+A2rg__5*60D7*G>56Z=QZFzD-^9nIh!zvS(G zqe%y+cW0>pJUsp#)C%4jQUdRM^qAMey`EDwYu*MU7BZO~*)|mUc=_^{C$G!SYs?7{ zni$NB-IW?$^=eHy2ucOIvN`!krljk#n9({&rg8}hk6OW&K1L;7&BoHMKTX4nBtURM zgQTn14TpcbA-7Z)V9F(GZD*-n9IfL2yn4mBb6{q5k97(`UN4l5AS7q(7CNhQ0QndF zSmK2W=3j#hS%0B5tJ)6`5D8(-;Xb<_#I9Dj zH6bjAOf%Nqd#PxM;-zs@w)4*fT`u;|?KU6VMK>HGs#~=cQsz^d<%4KaBVSw)Uy({Z zvJOO5I4t#@On~u~8~u$+NWwNMJIr~po_&qGrIwuh+ixkk91UD*zA z!}N)yA9LkpTtP*E+xT69AuK+!rGUe@+-r;wW16-z- zgN2t^u)K*|zVMoY8CGm69%Rs{@a1H#>x_g80A{TE`xh{W{SGukP%!g+(@oAJdTz0F z??+8An=!heF9{&ij|uN`Gb|BS2g&I~7#j75E~w;g+}8;`4@cRfC$}0*7r_Z!nfzwR zN-;`OtOCP!8ZxTA2#RA)&;41(W3u;0oikLQ9?BWpBp>b40AK2ByA0jAt>8^6I$N7N zx=Jd`8l15y$m%&;u?qmxzu7Uf3Bc6mp4pDVU=y5DfckM!qJEuVuxF<;=I%%;l=>Df zU1XtOEGdeyUbnvx8;dyGYMp0>pt6IP=v4|y56Q1JtAMBcEz@^j2H8AS+jrRiLUGGc zqf$p9c70K9XL@AGM}bZb%9Nr(q~-FKtzQWC!%Xn6rJdZunp8dX1N|m7<_gBHr=U6u zbLh<&LtSVG3Q$Zd7|N`e!v&1IWge{bOI<7C!!OvuYX)jh6223ck19TkOl2Z603#p3 z@;Ny}zYmIyb&Xtq#*XDdaeqGElCOoBi%G$dJ5hVs&ipJ=6QZWytZV=H#Fz`f(ZWX& zpv^#I7@>g)i}>IQrj)Ub2-0gD4H*drt9ufY$08S%mMaGdZ}PU$|7Nc4-RV2Ln-^^p zn1r2g>VgG1!(LIhxu|)ZU!5Ii?%(hdj~sR|Np$c+fu=58YH6`(Eb}1z9>oXP<>zI6 zh8T&Oy`xzS6^ZLRnUS)%4Xs>qg`E1hX>=33E_CO2Wj?T3#Pkwk?`FzF&WW5fsUzh- z2RXVvbiVNmrU3cIpWM~pHnRp+6vSV=x!;(Q%{@vqNLTC90>@jENYWM1pwxr%lGPMv zb|4C>4v4CF*IU+W`B!*Q0zB5C!A@+$fsD{8XK;e{pzq{kRG}i)!Q&Z_u^h5&i~I1b zu%E@3OVCwbFKMLT#J&unAZ?d#H27#j@vU))7|V#BzPt)L$0}kEkU6QoICJoD{q#Sb zaNHe))FWtOE5p=0B4f-{lIN%l_DqEY`@^aV;%8ZsQcGB^qUUI7=8rWz4t;l@j_*kw zvVe{_R>w6PIHv{GY~XWet`IVzx86+uZ=Mxyil6(Opt%LA^ZJGy!iBlVRf$87sGkoo zDb|FWu9ub;xiA9BAbP|xvb;W$4(*gb+dUS}4G#32;q1(`VnDP1$qtQFM{GEnDjjV& z18AC#Lu4fIpaf2*tLIca8-{ZOLBj&N5^+rNbjp``)Wsq@v?UUvCwg7m##?;ywfY`O*BNn%5lX_AgE~gYHMVd9i(dp? zpQyiNV%TbG<6HIx3RTY?i3gPz{hIXGC#rGDx{gFBzmW~_1z<_&Q5jOJu+5^#ncPO` zaQhE7DIljYIl{ zgtA-Nf}hgD0|3orMu&J>E#8`$z|8#F;sw@#Eg&_S!@helo57ra5WLTtw%`4(whQDm z;>fzLQ=HT~<2C!yt(e+vXj@kNmfgu;XHBOgR}bqJdCa`KvgD42hOA_*AL7J^Vx5^0 zM3}6!V$*@!`o~DrG6wlqO1rLL+y>$lP3%!H`h)5gtyY0ajhT#9NG~2Xsz$r$pa;@Y z)(ZqpR_3qguKsxDs!tbesA`j%KACOL%F#QtcdF~dzaOID;Sax2K}-J$dE)T#-7%Q& zZM7VyqjSmAFN%Fko}kmSO7V9}3yqusE_rV!Jr+iudDOr@2c(yP9;k>mlGAVfBhrM% z!bB@f*tS@Ft%yam7H}6nE>XfFF{_3Pl#9kc+UXP&0K1^x4w=r)gV(X$=DbOK{ zm2d)r00R%I!Q;2sN0-Zu!J7vxMB&O*h!f|pnXCO5_7JV_D+pytvrjzd!)5Lx+4<-h z(Z{pKkGzxN7|g=Yx>YjIv3qpukHlkU%0X9s}wSNW&6B;bEU4j+rBm*PH zt-qsyMSn7))riw-vRtv0QoM9#Ar%JK`ZeS0P85uHw8h{P|(hN_T907>J+w+kMMaa6}rci@JE?Bzj;Kq9Oq=H1sQD5EKDpfV7slrYYZNnXh6 zlBXwHg@owqP~^isr18&VY7p%(vk$?ZOburgz)p!5oKk@|syicCX`$#% zUV}Xt1m_rKrDR$DmuqsJYh-%WgzJkk!PCNUozu zvRqMVL`s%eALLXbDN*Q*2LG`Gc=^aYt6o&BgoaCGt-+GJv+00 z^L8LTye`z30ug%N&l*RQ6)d5ic$xbdwmvg^aURR~)jEf;m{T4fY>`zGX_V@~~FZfs3 zc|OQuZ!8`P+H$r2v&cYM!IgJmtH~Ev&-MW5Mx*Ws7xFz5LSI~dUkGopetIdX`;#GR{IUR5bpPgoSRqcw=knaW;VT@v`Y`_q*(ipkPvVoxJ) zOq?|8_7F9~C9*@Oy)>=-cr8|BPinMxn6*#n|94yB&)?S@OrAFH@%svL?)SUalh_pQ z=uNf6VQW|NTbozkoE{y1$3*4CDt%N#N)x%o&=|Q%XwbQb@MMUgei? zp(bHx1#^SY>^k!}QG`A*wo>M}`UDZVHe~YoKIHKPiSEiUh}X&s#4n??f8Tkh1{W!@ ziIXrOs$^p%+y40PJ^0AYH&Abtq80xe;&PP*))$+f-ud&!y;S}>cWgL>A6)fnZI&#W z*#U^hA;nAy$h#dihLdday^9NPyr!vF>*ptQr91F6QES(%S3Bv$9qjK^agnf>!gU8# zpjftYXw<{rk=<>Jr5zVfPU5agMZ;2clBE8Ey$}xyQD=+?Sf?2&OBA)ElO(o@1c=Kb z0!ZWxelF2+49c^CfS;M-7-Uz!Ym#>Hpku=cqLdF;2d7G+~?tz#HWub1rH90 zUj5~L8Gwr$ul|8io!1f|d&#T^%x1Uwc-2AU?7pgB%6jQlpa>8v=;x)y)rLdjH_>pg z-&ebSy1j^*2@(XQ)>6_t_iQ@*UoFLF{bYghV8{J}roq3iL&&IId>ChEnF)FDq#bfH z&c|&wnC1G<3Sh;U>$OLLPsz%*xLH%)LxlM`T1uWcqAj1G18)gt{K;v7tNS1Ds+0+2 z9sqhf_CII#xC8ozB^xZREqd3~#lJzQh(EQLc$V$1Z+AU zN{LEBBUJpX>$uQlgF!%$=a5*z!aKV<^n zehY%c4>bm&*#78zI%z+^j^OH9Jiy+`Uw0v4!82FEJC);1L;Dndor*m>G>MU zFyD6{I0UJu`Q2(hDtdeg!2m+@RHHnL$KQ;Yoo+=zQ!i#wZ$CHs0C z96byYW4jxLF5nVY;7A$ylH5l04%O|sTqdx)3EKu|1WTZDc#^7`_X&o8bl?^`)4_pG z^9$C|HXKq-I0?Pi&`)nnhNF2B{?hz)OE$YEE zJa^q{1o3Ca5RXZ?Pg3c|N}x1HSBC{V9IVJv7+pi99w!KO`wp zATS_rVrmLJJPI#NWo~D5XdpN>Hy|J&ARr(h3NJ=!Y;9$RZ+7 zL=g}~E}$Tb3yY#C;6nic5y7zLO}OutR^3;1zrVk#nXc-d?$dqFncq1*HB-}|Fj^?Z z^Aw2r=9Il0i^5|!Q{<&~+lY)foT_^h0J7Q-jZ0QMx4E{IC-r8w zlD~hE)5Ga$8+J61Vfo&Zc7F$J7?^%qkkma~@3azjMyR^FlKlYe8q>_Erq80!G}xw3 z^y@a(HmGY4+?emxE^M00+bE@r&z!Lt8*Kn8ztCSTU0Bl#>j{jJyto|XA^gppF?#03 z{lzZ+fuW+}>nEtXbbL(Nvs;62_y?n0MELK9fF|WldQ?IhtAk9Rm$eLAJE&4 z=Np$@-PRn-OClv;!j~eQ^*L0e2eSfrchMZsh<%69s)m!gltzR?b6wTJEkS2CUjO4Zb4Lu+=G5T0MN_5c0WC7WEZp_!Z9BTZexnk{R^Yr1iz zR=QmtE|OR$=L(B`UN6%98%Ex>^|yzRw04xvBy(pz;@%_ZryGx?I?uDhsKS9*kurMg z%-*t(%?@==@%Kvky7o#BF0? zmWCvma!rE!zs zEQ!uRDO+`H;PJ}t{S6-zogU>@c?k75J(ZN9NRTUu8+npv1dp{oR}4EFQC<1(v&G^9rkd(G=`KYNQcGGOpO@KT zvMSL-jp>5Z81N+%(RYC_?NgrPrPp`*XS^f)51?ZI-vHHe`u(E=A9LrLeq0awlr36$ z+sNct%UkAj(x%l_ms@qBRcrUaA}1$)e`b<$jW{0e_CBrH^dZj812+)f<3#vQ6&V#( zG-zS<&ecpY@HMWLE14|af=D$P!IOcwJ z+#A8Fw%ELr9iLUQ-)&1reS5!V2yg0KOjSmdX! zolCN4DxyCXtc`^`wMt&Ay^PfDI@DpuTGd7_8#`zu_2w^ilC*s{kR0qxnO+uv*wdkd zENk@{V*i=mRon5}&9j^;jl{`(`VCG8b|zIdcMGh-z=z7UCUBq(Rk}lDT3oyDcyb@9 zJEdm3s??6QXWji22Uj86mT@c(lG{~Dy1~yRDU~eBg3h6LX9gR>&VWA+n5uJ82 zC{z^+xAUU~QdC_vw6%a!+ZBO8Sn~*-N4$6>kVpJ@gv=wZ2sCga5EeY*$s>$k#dsdG zWwLc`tC=vKjPg^9*8mFrlSd%7thNk-e-#Ju7@uDyTVA(<+0xz$aEtvJC5@*a0fpPr zz5FR;8qh^NkS)-F&i>lTpW)l_^tYlT{pu;4$Ncp4SGTv~+wx82DR}e9R+wZc+?*1@ z2nzKMQw7F%%coxu)fh-zAgabXC83u8ngoXjGAsXsPG*T0b($ds8 zj8fP7YxV&H3JGH27fyljqJlyKf<0*`!ziAR@c$ZWsiW0VP+q1*p?dRfCe$xCm0d(I z1p+5}`PXXuC{f@MCuh|PT-49O-i8h9pkFfc>>xJOc z%_CL5bRu+DyPsZAWx||_#ZD-=^ou#Y;L^&qFBs`0rM~oq>djRGz2a>nUsDa%euSFS zJu{S{11VU10J~7uPd3|AUqvRLPb8%sx88Nhkl8%vlX*H8YEP#LF`c+z$S^VcN5poA zWbXQfeIZM5hY9=Ydk56Y@5v8+ua%UgRK)%@IMWv|@$UoES ziYd+bvyBb3L1;52z6(6x!B-3(oC$O`BkB|<1xGT??>LU`I@=?B<<9k!Axk%e49J4L zaF4pnaWIns)2OV6FzD&WB>c5{e3k0K(^7#i>Muvova&@? z3o{-su#fLVjv1HZu%p69Z%LAyEM$a8^F<#<&ux6vB&l2#*~`A}fi*sgY;$&=dqE;k zeeTLSgTWsMNv9?w&7a?9=l6(X!-G<#7z1Z>fsFU(c@mku|U@6V&OU!l|$MeJ|o_Z*y+ zI_yEr{(eWQuoL)-`t7{0?jl&2qFd$AGSoA!r59>3I+Ce`T_u<2^aQrJ!2-mjtddse z1Q)K1+227rU%oKn-$M@IS0Rlt^2fjdO~AX5;PZ@OuXAoL6ClT>hN;Q=4jKNpA4Jy= z&NIl(ox<%IKDvqT$>S$OY4qi(I03GRc>GmWu;^6$)f%$$cU$G};>ustES`8hJ1eOX zlB~ik94`M?bDrbI;p30b(u|F(xDiWpl%tS=p*+y!`f21C1I}y>;68&1JGI}v;cY4+J z9l_R1bWX!VI)-)k5?o-cHu7!NTAW`A4y9Z1Vn=)L*8mM!>X~M@-#+Jnnyr6pr~G(u z>Zs{632roX)NJYE%SD(ySJ%GO-Q2!~<-dZmZ+`qe^4#mMmWpk}o+jdt6#mVXCz+w2 z%JXI|ll=9*;GPJEJ9rm#y9Mt_SLt!n&`qEkrT@rjP)Xej&%{=JE@xbT`P4m>Eae;; zXggD8ClWn3Q`=J$=&b%B(Ek2fEqls2!=;nEjeD5!`~TSg(0^O1NH!E75d0X`o5BSC z_X|O3x`E)fey4%ePZI=pgt#CfCI=~jl*eZ zp^P*M8aO;g6Qza57!!=JXbb^otc4+9^&$VP0(h~YPPq`9Z=6hlh zVmtc1SF(fJ?F3}}OLh7m3-Koj@;wst7KJ~R+oqooH&lW)vowCMoSlk#qWBQ~;_}TH yoyndf!Nd;>tA&(HPikWst)e-W0-Jvyv@n`yD2*8k43j4ClM79dl9CCL0QwIR9BsV- literal 0 HcmV?d00001 diff --git a/examples/fixtures/workspace-wiki/signed-addendum.docx b/examples/fixtures/workspace-wiki/signed-addendum.docx new file mode 100644 index 0000000000000000000000000000000000000000..362e319d37450f974b5438d75328857550d4045e GIT binary patch literal 3590 zcmaJ^2{e>#8y@>uipIVUW6zo`5|NZK_Uw{%P>f-$S(1>khan`N$P!r+N*LP|CR^FJ zEMqB&GBGIp<6Hj9`TqYs=XsuU-gB<|Joj~P*JA>sqGkgC06>6Bsg!-zU}Oh`0svs9 z0svUaf9-UT-u`fJe+Tm$K5##K*!3*C24;E06j-E&$}!>*O7fK2$KL2(Q|l0U zkl7q$@>N*SVfggo5*L6wo8PhtCV3VZSw6=}Wt)Tfwkt!wIbhl@XxLU#pjLE7J6 zcufaHxD~7JJxqG}eaq6s-_d?+_UW~aMOas0(3fW+gezB=ct#8vy4ys?$ zSVZ40QllO5S)goOyZ#-g2(6Gzhv{1AN$`lJ`o~)L;PUiYSz~t( z*Cu&;TDS*fvaTbQ7n+-Chrz54?a3XpOf#2=D{5VF99?vCj^mYPm76wK_%Ci^E^!=F z$?4>g)ss|aKNdXU;n(nXX0X6r2zK7Xd5Ng`2lsOH$U#E^{B=riXb;C{tL;fzPOE`% z!0&uS^`T~Z$UHnDrx_pl3__w@<^CeY73mUyAZOcQn*Bx4T|*d|pgZgP@o9~{Jpvb8 z)k9A>At6%1#}G?YC5G3Y&F_aMKwehUF$<*MBX#T4H%Q7j&(ql1G(<;=y;f4fmWC zGn3MrwRvrvtIdzouR4D#Vi8eS(uyH^u#UZS6fEwq=A>;OeP8uRQJ`Q#vSMEQO4TTG z+kkNLJ)K=2n^d8z&%%Ui45kv1fg6`TF$bFFL#da9nX^#@;M;6gFeNdF?R)!<-!^7G z$fY7E6F-V1^~D!5CNUd9OgG-UeZd9SVe_}*=GlCq49ht%F)3 z%KB`*1b;uH#v7L;de^*c9&EzQl|fx^@m+Q!+J9YIBS~*}Qbx>jH*bK#|5cTPTwcZ_ zzY#lv901LMVYL#fo&7RUf zJnj{7!v-i}Kr9R1U;@H(jb~F3SmOLw`l<-3--GQEx!5v8?x^DANVESFY<~|p0xtLG ze(Esc7p-APf(i?kWX007jt@~ovA91Cl>Rorp*vsTZd)LVkkA**R^1`-o66y9K$Z=e7PGjP zseP1ArG>s6s=)CiNQz?tP^z>fc%XhB_gv=tois&qo!#@(mg2af=qEy)E5^d3KwL_4CTn*kYpo8_H(>Ouy7B?n^Kq{c-bTV8 z0HixxrqLeanoXl@1CepYIQLpeL~EpTKc$?sd242=aNE5tY^{tx#4p8MFmLWYl|t*+bS}g#;FESgZq5r;`~pi6+1YZz{)z4?R#1E zerc{{M6sbA&aQV}7*WqKK5lT|0L7CHesCcg18a%}QFb&@Yj(3H0%5b*6mtT;F?MHc zZgBElp5d9ZCZ-IJ*U$!@l6$4i$8I272%#2yvV_*S>;8G{-i`+eme!q*i@s|l4dL|9 zA@bOb;X(Iz-3xn3oHmM+xh_FJYPg({!pQ}z(i{~KOoY6 zK(ZV!f-Ad+##`=AjaK9o%A%lUYtN|Tm`*;>rzwgkd)*(@<<;5bU{_Gw=f8xn*`I#v zj;|m7_`qu41yeu#h(7u?zj*JNfR};LZSY!HQ(WKj=dFCJg>YlJd2ZdA7YrO=#X0pV z+vVEMK37piA0Nrbj7W86f9`q0QW+V=JR#Jr?^1Bt*Htsr8E@TbudPSLKH;f&M* z<($oXbia3mNzVqGA#x}1B9DN>_D}BoCMcwjpPUO41wZTt2=e6X&|*29eAOgZ5F zyFDFgmO+EkMeLJDfZ7E&zPhH(d-Ec?&QX15f!c45rL9ib`z5gnZZJVzDbh1p{Yo?R z11XVa*@{aFCg1K56AKHRvSf0*H8ljwbu9SVpD=xBWuIfjr&!DVkXz}(9u$SP#H`jO z?M@>ceh(zF&fhbceEU-J`Yn(MxWCh(8@g|vGPKLH+?ty%0Y1O)$yf)0o&g%^bLx%? zYfPL^$B3Y@%_FaN4sH%(r=;`v2HBeKm~jY>O%52}9Pnw(s}E(7OkcltJWshvP1{oi z?<@8-Jp7&fQWGCbI{&z7TCKDqjF)f1k;=(}P5nx^advBIq{bz@P!3_Dam`n4hJaCX zwm4nq#kL=}`(TMYmnbON0RLO>lf3|tQ}XZcU)29m14q^EA@VN+m&wO3TK6dMs0{oG zbRi4n|0fEMS~@DFepZ2x(R`X929`yfG+ehI? xE8!>Hli?Tm-*s^meKZ4qqEC^Z8~<%A|Ky_yjD{>Z58r!00415}wZk_F;6ET%6)*q* literal 0 HcmV?d00001 diff --git a/examples/src/simple/22-workspace.ts b/examples/src/simple/22-workspace.ts index df353041..dc9b2efb 100644 --- a/examples/src/simple/22-workspace.ts +++ b/examples/src/simple/22-workspace.ts @@ -101,7 +101,9 @@ function makeRuntimeContext(): RuntimeContext { // Fixture lives on disk at examples/fixtures/workspace-wiki/. The demo // copies each file into a tmp directory before running so every run starts // from a clean slate — the same fixtures also back the workspace tutorial. -const FIXTURE_FILES = ["a.md", "b.md", "law-clause.md"] as const; +// The set covers every format @melandlabs/workspace's parsers-adapter +// claims to support: .md / .pdf / .docx (and .pages on macOS). +const FIXTURE_FILES = ["a.md", "b.md", "law-clause.md", "law-brief.pdf", "signed-addendum.docx"] as const; function resolveFixtureDir(): string { // examples/src/simple/22-workspace.ts → examples/fixtures/workspace-wiki @@ -174,13 +176,13 @@ export default async function demoWorkspace() { } check( - "updateWorkspaceContext returns ok with files_scanned ≥ 3", - update.filesScanned >= 3, + "updateWorkspaceContext returns ok with files_scanned ≥ 5", + update.filesScanned >= 5, `filesScanned=${update.filesScanned}, filesAdded=${update.filesAdded}`, ); check( - "updateWorkspaceContext flags the 3 new files as added", - update.filesAdded >= 3, + "updateWorkspaceContext flags the new files as added", + update.filesAdded >= 5, `filesAdded=${update.filesAdded}`, ); check("updateWorkspaceContext returns a positive jobId", update.jobId > 0, `jobId=${update.jobId}`); @@ -195,8 +197,8 @@ export default async function demoWorkspace() { workspace_id: WORKSPACE_ID, }); check( - "listWorkspaceResources returns ≥ 3 resources for the fixture folder", - listed.resources.length >= 3, + "listWorkspaceResources returns ≥ 5 resources for the fixture folder", + listed.resources.length >= 5, `total=${listed.total}, resources=${listed.resources.map((r) => r.canonical_key).join(", ")}`, ); @@ -323,7 +325,7 @@ export default async function demoWorkspace() { }); check( "re-running update reports every file as `unchanged` (sha256 dedup)", - reUpdate.filesUnchanged >= 3 && reUpdate.filesAdded === 0, + reUpdate.filesUnchanged >= 5 && reUpdate.filesAdded === 0, `unchanged=${reUpdate.filesUnchanged}, added=${reUpdate.filesAdded}`, ); diff --git a/packages/workspace/package.json b/packages/workspace/package.json index fa1d3e79..b2409e06 100644 --- a/packages/workspace/package.json +++ b/packages/workspace/package.json @@ -44,6 +44,8 @@ "@modelcontextprotocol/sdk": "^1.25.3", "better-sqlite3": "^11.10.0", "hono": "^4.6.14", + "mammoth": "^1.11.0", + "pdf-parse": "^2.0.0", "sqlite-vec": "^0.1.9", "zod": "^4.3.6" }, diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 7d593595..09d093ca 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -378,7 +378,7 @@ importers: version: 3.8.1 '@langchain/community': specifier: ^1.1.1 - version: 1.1.29(382ed5f2a3e6bc1e3575eeff382cf5cb) + version: 1.1.29(8e07a13ecb0caa2defe560ab6fa2e6f8) '@langchain/core': specifier: ^1.1.8 version: 1.2.5(@opentelemetry/api@1.9.0)(openai@4.104.0(encoding@0.1.13)(ws@8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(zod@4.4.3))(ws@8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10)) @@ -1316,7 +1316,7 @@ importers: dependencies: '@langchain/community': specifier: ^1.1.1 - version: 1.1.29(905166cb4cd0093f294357964e3103f4) + version: 1.1.29(473a380aa82688979edbb67c4e0acfa5) '@langchain/core': specifier: ^1.1.8 version: 1.2.5(@opentelemetry/api@1.9.0)(openai@4.104.0(encoding@0.1.13)(ws@8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(zod@4.4.3))(ws@8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10)) @@ -1551,6 +1551,12 @@ importers: hono: specifier: ^4.6.14 version: 4.13.1 + mammoth: + specifier: ^1.11.0 + version: 1.12.2 + pdf-parse: + specifier: ^2.0.0 + version: 2.4.5 sqlite-vec: specifier: ^0.1.9 version: 0.1.9 @@ -3454,54 +3460,108 @@ packages: cpu: [arm64] os: [android] + '@napi-rs/canvas-android-arm64@0.1.80': + resolution: {integrity: sha512-sk7xhN/MoXeuExlggf91pNziBxLPVUqF2CAVnB57KLG/pz7+U5TKG8eXdc3pm0d7Od0WreB6ZKLj37sX9muGOQ==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [android] + '@napi-rs/canvas-darwin-arm64@0.1.100': resolution: {integrity: sha512-2PcswRaC7Ly645DGt88///zuFDhJxJYdKAs1uU3mfk1atYkXufgcgLfBpk6Tm12nCQBaNt1wpybuPZ4qOhTo8A==} engines: {node: '>= 10'} cpu: [arm64] os: [darwin] + '@napi-rs/canvas-darwin-arm64@0.1.80': + resolution: {integrity: sha512-O64APRTXRUiAz0P8gErkfEr3lipLJgM6pjATwavZ22ebhjYl/SUbpgM0xcWPQBNMP1n29afAC/Us5PX1vg+JNQ==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [darwin] + '@napi-rs/canvas-darwin-x64@0.1.100': resolution: {integrity: sha512-ePNZtj7pNIva/siZMg+HmbeozkIjqUIYdoymH8HaA3qK7LfzFN4WMBM8G6HQ9ZC+H3+Dnn5pqtiXpgLykaPOhw==} engines: {node: '>= 10'} cpu: [x64] os: [darwin] + '@napi-rs/canvas-darwin-x64@0.1.80': + resolution: {integrity: sha512-FqqSU7qFce0Cp3pwnTjVkKjjOtxMqRe6lmINxpIZYaZNnVI0H5FtsaraZJ36SiTHNjZlUB69/HhxNDT1Aaa9vA==} + engines: {node: '>= 10'} + cpu: [x64] + os: [darwin] + '@napi-rs/canvas-linux-arm-gnueabihf@0.1.100': resolution: {integrity: sha512-d5cDB48oWFGU8/XPhUOFAlySgb/VAu7D+s8fi55K1Pcfg8aPplHWqMgibhVLU8ky7Pyg/fuiVLz4Nf3JrSTuUA==} engines: {node: '>= 10'} cpu: [arm] os: [linux] + '@napi-rs/canvas-linux-arm-gnueabihf@0.1.80': + resolution: {integrity: sha512-eyWz0ddBDQc7/JbAtY4OtZ5SpK8tR4JsCYEZjCE3dI8pqoWUC8oMwYSBGCYfsx2w47cQgQCgMVRVTFiiO38hHQ==} + engines: {node: '>= 10'} + cpu: [arm] + os: [linux] + '@napi-rs/canvas-linux-arm64-gnu@0.1.100': resolution: {integrity: sha512-rDxgxRu69RvDlX/bh9o22DxLsGr8EqsNgotL9+RwQE1S0b0cqeatqsw6aW45mukm0B42DIAaAacKaYQ8cqS1nw==} engines: {node: '>= 10'} cpu: [arm64] os: [linux] + '@napi-rs/canvas-linux-arm64-gnu@0.1.80': + resolution: {integrity: sha512-qwA63t8A86bnxhuA/GwOkK3jvb+XTQaTiVML0vAWoHyoZYTjNs7BzoOONDgTnNtr8/yHrq64XXzUoLqDzU+Uuw==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [linux] + '@napi-rs/canvas-linux-arm64-musl@0.1.100': resolution: {integrity: sha512-K3mDW66N+xT2/V439u1alFANiBUjdEx2gLiNYnCmUsva5jZMxWTjafBYwTzYK+EMFMHrUoabuU+T1BIP5CgbYQ==} engines: {node: '>= 10'} cpu: [arm64] os: [linux] + '@napi-rs/canvas-linux-arm64-musl@0.1.80': + resolution: {integrity: sha512-1XbCOz/ymhj24lFaIXtWnwv/6eFHXDrjP0jYkc6iHQ9q8oXKzUX1Lc6bu+wuGiLhGh2GS/2JlfORC5ZcXimRcg==} + engines: {node: '>= 10'} + cpu: [arm64] + os: [linux] + '@napi-rs/canvas-linux-riscv64-gnu@0.1.100': resolution: {integrity: sha512-mooqUBTIsccZpnoQC4NgrC1v6C1vof39etLNMnBwCY+p0gajWJvAHLGQ6g/gGyS5YrpDW+GefSN4+Cvcr08UWw==} engines: {node: '>= 10'} cpu: [riscv64] os: [linux] + '@napi-rs/canvas-linux-riscv64-gnu@0.1.80': + resolution: {integrity: sha512-XTzR125w5ZMs0lJcxRlS1K3P5RaZ9RmUsPtd1uGt+EfDyYMu4c6SEROYsxyatbbu/2+lPe7MPHOO/0a0x7L/gw==} + engines: {node: '>= 10'} + cpu: [riscv64] + os: [linux] + '@napi-rs/canvas-linux-x64-gnu@0.1.100': resolution: {integrity: sha512-1eCvkDCazm7FFhsT7DfGOdSaHgZVK3bt/dSBl5EWHOWmnz+I7j8tPseJqqD81NF+MH21jKUK4wQSDjN0mdhnTg==} engines: {node: '>= 10'} cpu: [x64] os: [linux] + '@napi-rs/canvas-linux-x64-gnu@0.1.80': + resolution: {integrity: sha512-BeXAmhKg1kX3UCrJsYbdQd3hIMDH/K6HnP/pG2LuITaXhXBiNdh//TVVVVCBbJzVQaV5gK/4ZOCMrQW9mvuTqA==} + engines: {node: '>= 10'} + cpu: [x64] + os: [linux] + '@napi-rs/canvas-linux-x64-musl@0.1.100': resolution: {integrity: sha512-20arT6lnI19S68qNlii73TSEDbECNgzMz2EpldC1V3mZFuRkeujXkcebRk0LRJe9SEUAooYiLokfMViY8IX7yA==} engines: {node: '>= 10'} cpu: [x64] os: [linux] + '@napi-rs/canvas-linux-x64-musl@0.1.80': + resolution: {integrity: sha512-x0XvZWdHbkgdgucJsRxprX/4o4sEed7qo9rCQA9ugiS9qE2QvP0RIiEugtZhfLH3cyI+jIRFJHV4Fuz+1BHHMg==} + engines: {node: '>= 10'} + cpu: [x64] + os: [linux] + '@napi-rs/canvas-win32-arm64-msvc@0.1.100': resolution: {integrity: sha512-DZFFT1wIAg37LJw37yhMRFfjATd3vTQzjZ1Yki8u2vhO6Hi5VE6BVaGQ1aaDu7xb4iMErz+9EOwjpS7xcxFeBw==} engines: {node: '>= 10'} @@ -3514,10 +3574,20 @@ packages: cpu: [x64] os: [win32] + '@napi-rs/canvas-win32-x64-msvc@0.1.80': + resolution: {integrity: sha512-Z8jPsM6df5V8B1HrCHB05+bDiCxjE9QA//3YrkKIdVDEwn5RKaqOxCJDRJkl48cJbylcrJbW4HxZbTte8juuPg==} + engines: {node: '>= 10'} + cpu: [x64] + os: [win32] + '@napi-rs/canvas@0.1.100': resolution: {integrity: sha512-xglYA6q3XO5P3BNJYxVZ1IV7DLVjp1Py6nwag88YntrS+3vKHyYcMqXVS4ZztJmwz2uGvz1FWhI/4LgbR5uQDA==} engines: {node: '>= 10'} + '@napi-rs/canvas@0.1.80': + resolution: {integrity: sha512-DxuT1ClnIPts1kQx8FBmkk4BQDTfI5kIzywAaMjQSXfNnra5UFU9PwurXrl+Je3bJ6BGsp/zmshVVFbCmyI+ww==} + engines: {node: '>= 10'} + '@napi-rs/lzma-linux-x64-gnu@1.5.1': resolution: {integrity: sha512-oTXEIha4SsuXdTA4Iyskj0kpdx2yVXdhd75c2v3xGrHFfVMsbhTPZU/nMPL4sWKo4pBHm3aucLaqGlF696dTyQ==} engines: {node: ^22.20 || ^24.12 || >=25} @@ -4027,6 +4097,10 @@ packages: node-fetch: optional: true + '@xmldom/xmldom@0.8.15': + resolution: {integrity: sha512-/5NV/vDALVFDXgLmfsy9TRCBlKwO2LNBFzpzvb9iIj+jR+eSc6DLYYvVOdivT/jm7MtU6TebYuRmzEOI7w40UA==} + engines: {node: '>=10.0.0'} + '@xterm/xterm@5.5.0': resolution: {integrity: sha512-hqJHYaQb5OptNunnyAnkHyM8aCjZ1MEIDTQu1iIbbTD/xops91NB5yq1ZK/dC2JDbVWtF23zUtl9JE2NqwT87A==} @@ -4247,6 +4321,9 @@ packages: bl@4.1.0: resolution: {integrity: sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==} + bluebird@3.4.7: + resolution: {integrity: sha512-iD3898SR7sWVRHbiQv+sHUtHnMvC1o3nW5rAcqnq3uOn07DSAppZYUkIGslDz6gXC7HfunPe7YVBgoEJASPcHA==} + bn.js@4.12.5: resolution: {integrity: sha512-3aRg6/JxfffFD+OlOjOFR3Vo79l39ooBTFucxx+MT3dhCtzn3EmiUPQo+6/OZuI2jbXi3YKgmiTFBgChQMwIRQ==} @@ -4680,6 +4757,9 @@ packages: dijkstrajs@1.0.3: resolution: {integrity: sha512-qiSlmBq9+BCdCA/L46dw8Uy93mloxsPSbwnm5yrKn2vMPiy8KyAskTF6zuV/j5BMsmOGZDPs7KjU+mjb670kfA==} + dingbat-to-unicode@1.0.1: + resolution: {integrity: sha512-98l0sW87ZT58pU4i61wa2OHwxbiYSbuxsCBozaVnYX2iCnr3bLM3fIes1/ej7h1YdOKuKt/MLs706TVnALA65w==} + dir-glob@3.0.1: resolution: {integrity: sha512-WkrWp9GR4KXfKGYzOLmTuGVi1UWFfws377n9cc55/tb6DuqyF6pcQ5AbiHEshaDpY9v6oaSr2XCDidGmMwdzIA==} engines: {node: '>=8'} @@ -4827,6 +4907,9 @@ packages: sqlite3: optional: true + duck@0.1.12: + resolution: {integrity: sha512-wkctla1O6VfP89gQ+J/yDesM0S7B7XLXjKGzXxMDVFg7uEn706niAtyYovKbyq1oT9YwDcly721/iUWoc8MVRg==} + dunder-proto@1.0.1: resolution: {integrity: sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==} engines: {node: '>= 0.4'} @@ -5728,6 +5811,9 @@ packages: resolution: {integrity: sha512-lyuxPGr/Wfhrlem2CL/UcnUc1zcqKAImBDzukY7Y5F/yQiNdko6+fRLevlw1HgMySw7f611UIY408EtxRSoK3Q==} hasBin: true + lop@0.4.2: + resolution: {integrity: sha512-RefILVDQ4DKoRZsJ4Pj22TxE3omDO47yFpkIBoDKzkqPRISs5U1cnAdg/5583YPkWPaLIYHOKRMQSvjFsO26cw==} + loupe@3.2.1: resolution: {integrity: sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==} @@ -5745,6 +5831,11 @@ packages: magic-string@0.30.21: resolution: {integrity: sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==} + mammoth@1.12.2: + resolution: {integrity: sha512-MH2vkgafD/2MYUaEOtoXLKrHQZ7yLYTHGQgyLNtYZHiUxU1K1QQ+8qMFquDAfhBm06wSGSws3J+Q9QTsKtR44g==} + engines: {node: '>=12.0.0'} + hasBin: true + markdown-it@14.3.0: resolution: {integrity: sha512-RCEsPjR+sr0x+AuYp601tKTkgFG4YEPLCzHST3cQ/fhlJkqAkz1L2/Qbp1j9qw5SBwQHFBoW8+hoN5xssOF0Tw==} hasBin: true @@ -6060,6 +6151,9 @@ packages: engines: {node: '>=22.22.3 <23 || >=24.15.0 <25 || >=25.9.0'} hasBin: true + option@0.2.4: + resolution: {integrity: sha512-pkEqbDyl8ou5cpq+VsnQbe/WlEy5qS7xPzMS1U55OCG9KPvwFD46zDbxQIj3egJSFc3D+XhYOPUzz49zQAVy7A==} + os-paths@4.4.0: resolution: {integrity: sha512-wrAwOeXp1RRMFfQY8Sy7VaGVmPocaLwSFOYCGKSyo8qmJ+/yaafCl5BCA1IQZWqFSRBrKDYFeR9d/VyQzfH/jg==} engines: {node: '>= 6.0'} @@ -6144,6 +6238,10 @@ packages: resolution: {integrity: sha512-ak9Qy5Q7jYb2Wwcey5Fpvg2KoAc/ZIhLSLOSBmRmygPsGwkVVt0fZa0qrtMz+m6tJTAHfZQ8FnmB4MG4LWy7/w==} engines: {node: '>=8'} + path-is-absolute@1.0.1: + resolution: {integrity: sha512-AVbw3UJ2e9bq64vSaS9Am0fje1Pa8pbGqTTsmXfaIiMpnr5DlDhfJOuLj9Sf95ZPVDAUerDfEk88MPmPe7UCQg==} + engines: {node: '>=0.10.0'} + path-key@3.1.1: resolution: {integrity: sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==} engines: {node: '>=8'} @@ -6169,10 +6267,19 @@ packages: resolution: {integrity: sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==} engines: {node: '>= 14.16'} + pdf-parse@2.4.5: + resolution: {integrity: sha512-mHU89HGh7v+4u2ubfnevJ03lmPgQ5WU4CxAVmTSh/sxVTEDYd1er/dKS/A6vg77NX47KTEoihq8jZBLr8Cxuwg==} + engines: {node: '>=20.16.0 <21 || >=22.3.0'} + hasBin: true + pdfjs-dist@4.10.38: resolution: {integrity: sha512-/Y3fcFrXEAsMjJXeL9J8+ZG9U01LbuWaYypvDW2ycW1jL269L3js3DVBjDJ0Up9Np1uqDXsDrRihHANhZOlwdQ==} engines: {node: '>=20'} + pdfjs-dist@5.4.296: + resolution: {integrity: sha512-DlOzet0HO7OEnmUmB6wWGJrrdvbyJKftI1bhMitK7O2N8W2gc757yyYBbINy9IDafXAV9wmKr9t7xsTaNKRG5Q==} + engines: {node: '>=20.16.0 || >=22.3.0'} + pg-cloudflare@1.4.0: resolution: {integrity: sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==} @@ -7015,6 +7122,9 @@ packages: resolution: {integrity: sha512-rvKSBiC5zqCCiDZ9kAOszZcDvdAHwwIKJG33Ykj43OKcWsnmcBRL09YTU4nOeHZ8Y2a7l1MgTd08SBe9A8Qj6A==} engines: {node: '>=18'} + underscore@1.13.8: + resolution: {integrity: sha512-DXtD3ZtEQzc7M8m4cXotyHR+FAS18C64asBYY5vqZexfYryNNnDc02W4hKg3rdQuqOYas1jkseX0+nZXjTXnvQ==} + undici-types@5.26.5: resolution: {integrity: sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA==} @@ -7355,6 +7465,10 @@ packages: resolution: {integrity: sha512-EvGK8EJ3DhaHfbRlETOWAS5pO9MZITeauHKJyb8wyajUfQUenkIg2MvLDTZ4T/TgIcm3HU0TFBgWWboAZ30UHg==} engines: {node: '>=18'} + xmlbuilder@10.1.1: + resolution: {integrity: sha512-OyzrcFLL/nb6fMGHbiRDuPup9ljBycsdCypwuyg5AAHvyWzGfChJpCXMG88AGTIMFhGZ9RccFN1e6lhg3hkwKg==} + engines: {node: '>=4.0'} + xmlchars@2.2.0: resolution: {integrity: sha512-JZnDKK8B0RCDw84FNdDAIpZK+JuJw+s7Lz8nksI7SIuU3UXJJslUthsi+uWBUYOwPFwW7W7PRLRfUKpxjtjFCw==} @@ -8728,7 +8842,7 @@ snapshots: - openai - ws - '@langchain/community@1.1.29(382ed5f2a3e6bc1e3575eeff382cf5cb)': + '@langchain/community@1.1.29(473a380aa82688979edbb67c4e0acfa5)': dependencies: '@browserbasehq/stagehand': 1.14.0(@playwright/test@1.62.1)(bufferutil@4.1.0)(deepmerge@4.3.1)(dotenv@17.4.2)(encoding@0.1.13)(openai@4.104.0(encoding@0.1.13)(ws@8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(zod@4.4.3))(utf-8-validate@5.0.10)(zod@4.4.3) '@ibm-cloud/watsonx-ai': 1.7.16 @@ -8749,9 +8863,10 @@ snapshots: '@aws-sdk/credential-provider-node': 3.972.80 '@browserbasehq/sdk': 2.16.0(encoding@0.1.13) '@huggingface/transformers': 3.8.1 + '@lancedb/lancedb': 0.37.1(@types/node@26.2.0)(apache-arrow@18.1.0)(encoding@0.1.13) '@mozilla/readability': 0.6.0 '@smithy/signature-v4': 5.7.2 - '@zilliz/milvus2-sdk-node': 2.6.17(bufferutil@4.1.0)(utf-8-validate@5.0.10) + '@zilliz/milvus2-sdk-node': 2.5.13 better-sqlite3: 11.10.0 chromadb: 3.5.0 fast-xml-parser: 4.5.7 @@ -8760,6 +8875,8 @@ snapshots: ignore: 7.0.5 jsdom: 24.1.3(bufferutil@4.1.0)(utf-8-validate@5.0.10) jsonwebtoken: 9.0.3 + mammoth: 1.12.2 + pdf-parse: 2.4.5 pg: 8.23.0 playwright: 1.62.1 ws: 8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10) @@ -8770,7 +8887,7 @@ snapshots: - '@smithy/hash-node' - peggy - '@langchain/community@1.1.29(905166cb4cd0093f294357964e3103f4)': + '@langchain/community@1.1.29(8e07a13ecb0caa2defe560ab6fa2e6f8)': dependencies: '@browserbasehq/stagehand': 1.14.0(@playwright/test@1.62.1)(bufferutil@4.1.0)(deepmerge@4.3.1)(dotenv@17.4.2)(encoding@0.1.13)(openai@4.104.0(encoding@0.1.13)(ws@8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10))(zod@4.4.3))(utf-8-validate@5.0.10)(zod@4.4.3) '@ibm-cloud/watsonx-ai': 1.7.16 @@ -8791,10 +8908,9 @@ snapshots: '@aws-sdk/credential-provider-node': 3.972.80 '@browserbasehq/sdk': 2.16.0(encoding@0.1.13) '@huggingface/transformers': 3.8.1 - '@lancedb/lancedb': 0.37.1(@types/node@26.2.0)(apache-arrow@18.1.0)(encoding@0.1.13) '@mozilla/readability': 0.6.0 '@smithy/signature-v4': 5.7.2 - '@zilliz/milvus2-sdk-node': 2.5.13 + '@zilliz/milvus2-sdk-node': 2.6.17(bufferutil@4.1.0)(utf-8-validate@5.0.10) better-sqlite3: 11.10.0 chromadb: 3.5.0 fast-xml-parser: 4.5.7 @@ -8803,6 +8919,8 @@ snapshots: ignore: 7.0.5 jsdom: 24.1.3(bufferutil@4.1.0)(utf-8-validate@5.0.10) jsonwebtoken: 9.0.3 + mammoth: 1.12.2 + pdf-parse: 2.4.5 pg: 8.23.0 playwright: 1.62.1 ws: 8.21.3(bufferutil@4.1.0)(utf-8-validate@5.0.10) @@ -9004,36 +9122,66 @@ snapshots: '@napi-rs/canvas-android-arm64@0.1.100': optional: true + '@napi-rs/canvas-android-arm64@0.1.80': + optional: true + '@napi-rs/canvas-darwin-arm64@0.1.100': optional: true + '@napi-rs/canvas-darwin-arm64@0.1.80': + optional: true + '@napi-rs/canvas-darwin-x64@0.1.100': optional: true + '@napi-rs/canvas-darwin-x64@0.1.80': + optional: true + '@napi-rs/canvas-linux-arm-gnueabihf@0.1.100': optional: true + '@napi-rs/canvas-linux-arm-gnueabihf@0.1.80': + optional: true + '@napi-rs/canvas-linux-arm64-gnu@0.1.100': optional: true + '@napi-rs/canvas-linux-arm64-gnu@0.1.80': + optional: true + '@napi-rs/canvas-linux-arm64-musl@0.1.100': optional: true + '@napi-rs/canvas-linux-arm64-musl@0.1.80': + optional: true + '@napi-rs/canvas-linux-riscv64-gnu@0.1.100': optional: true + '@napi-rs/canvas-linux-riscv64-gnu@0.1.80': + optional: true + '@napi-rs/canvas-linux-x64-gnu@0.1.100': optional: true + '@napi-rs/canvas-linux-x64-gnu@0.1.80': + optional: true + '@napi-rs/canvas-linux-x64-musl@0.1.100': optional: true + '@napi-rs/canvas-linux-x64-musl@0.1.80': + optional: true + '@napi-rs/canvas-win32-arm64-msvc@0.1.100': optional: true '@napi-rs/canvas-win32-x64-msvc@0.1.100': optional: true + '@napi-rs/canvas-win32-x64-msvc@0.1.80': + optional: true + '@napi-rs/canvas@0.1.100': optionalDependencies: '@napi-rs/canvas-android-arm64': 0.1.100 @@ -9049,6 +9197,19 @@ snapshots: '@napi-rs/canvas-win32-x64-msvc': 0.1.100 optional: true + '@napi-rs/canvas@0.1.80': + optionalDependencies: + '@napi-rs/canvas-android-arm64': 0.1.80 + '@napi-rs/canvas-darwin-arm64': 0.1.80 + '@napi-rs/canvas-darwin-x64': 0.1.80 + '@napi-rs/canvas-linux-arm-gnueabihf': 0.1.80 + '@napi-rs/canvas-linux-arm64-gnu': 0.1.80 + '@napi-rs/canvas-linux-arm64-musl': 0.1.80 + '@napi-rs/canvas-linux-riscv64-gnu': 0.1.80 + '@napi-rs/canvas-linux-x64-gnu': 0.1.80 + '@napi-rs/canvas-linux-x64-musl': 0.1.80 + '@napi-rs/canvas-win32-x64-msvc': 0.1.80 + '@napi-rs/lzma-linux-x64-gnu@1.5.1': optional: true @@ -9629,6 +9790,8 @@ snapshots: optionalDependencies: node-fetch: 3.3.2 + '@xmldom/xmldom@0.8.15': {} + '@xterm/xterm@5.5.0': optional: true @@ -9873,6 +10036,8 @@ snapshots: inherits: 2.0.4 readable-stream: 3.6.2 + bluebird@3.4.7: {} + bn.js@4.12.5: {} body-parser@2.3.0: @@ -10251,6 +10416,8 @@ snapshots: dijkstrajs@1.0.3: {} + dingbat-to-unicode@1.0.1: {} + dir-glob@3.0.1: dependencies: path-type: 4.0.0 @@ -10322,6 +10489,10 @@ snapshots: kysely: 0.29.2 pg: 8.23.0 + duck@0.1.12: + dependencies: + underscore: 1.13.8 + dunder-proto@1.0.1: dependencies: call-bind-apply-helpers: 1.0.2 @@ -11389,6 +11560,12 @@ snapshots: js-tokens: 4.0.0 optional: true + lop@0.4.2: + dependencies: + duck: 0.1.12 + option: 0.2.4 + underscore: 1.13.8 + loupe@3.2.1: {} lru-cache@10.4.3: @@ -11402,6 +11579,19 @@ snapshots: dependencies: '@jridgewell/sourcemap-codec': 1.5.5 + mammoth@1.12.2: + dependencies: + '@xmldom/xmldom': 0.8.15 + argparse: 1.0.10 + base64-js: 1.5.1 + bluebird: 3.4.7 + dingbat-to-unicode: 1.0.1 + jszip: 3.10.1 + lop: 0.4.2 + path-is-absolute: 1.0.1 + underscore: 1.13.8 + xmlbuilder: 10.1.1 + markdown-it@14.3.0: dependencies: argparse: 2.0.1 @@ -11763,6 +11953,8 @@ snapshots: - tree-sitter - utf-8-validate + option@0.2.4: {} + os-paths@4.4.0: {} outdent@0.5.0: {} @@ -11833,6 +12025,8 @@ snapshots: path-exists@4.0.0: {} + path-is-absolute@1.0.1: {} + path-key@3.1.1: {} path-scurry@2.0.2: @@ -11850,10 +12044,19 @@ snapshots: pathval@2.0.1: {} + pdf-parse@2.4.5: + dependencies: + '@napi-rs/canvas': 0.1.80 + pdfjs-dist: 5.4.296 + pdfjs-dist@4.10.38: optionalDependencies: '@napi-rs/canvas': 0.1.100 + pdfjs-dist@5.4.296: + optionalDependencies: + '@napi-rs/canvas': 0.1.100 + pg-cloudflare@1.4.0: optional: true @@ -12821,6 +13024,8 @@ snapshots: uint8array-extras@1.5.0: {} + underscore@1.13.8: {} + undici-types@5.26.5: {} undici-types@6.21.0: {} @@ -13242,6 +13447,8 @@ snapshots: xml-name-validator@5.0.0: optional: true + xmlbuilder@10.1.1: {} + xmlchars@2.2.0: optional: true From 9c77d57a7f80c4b453e8da63705dfa4e53aaaf3e Mon Sep 17 00:00:00 2001 From: Peefy Date: Fri, 11 Sep 2026 21:34:58 +0800 Subject: [PATCH 6/8] feat(workspace): add Excel/Apple-Numbers spreadsheet support via SheetJS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - parsers-adapter: dynamic-import SheetJS, convert every sheet to CSV with a `# Sheet: ` header so the chunker preserves boundaries; .numbers goes through macOS `textutil -convert xlsx` then SheetJS - okf-backend: extend SUPPORTED_EXTENSIONS with .xlsx / .xls / .numbers and map them to `resource_type: spreadsheet` - fixture: add contract-terms.xlsx (2 sheets — Liability Caps + Indemnification) plus a regen script; demo now asserts ≥ 6 resources and the cross-file search hits the spreadsheet alongside the .md/.pdf/.docx - examples/package.json: add @melandlabs/workspace as a dev dep so the dynamic import in 22-workspace.ts resolves to the local symlink in dev (the npm-installed smoke test still falls through to skip cleanly) --- examples/fixtures/workspace-wiki.README.md | 21 +++- .../workspace-wiki/contract-terms.xlsx | Bin 0 -> 18335 bytes examples/package.json | 1 + examples/scripts/regen-xlsx-fixture.cjs | 41 +++++++ examples/src/simple/22-workspace.ts | 23 ++-- packages/workspace/package.json | 1 + packages/workspace/src/okf-backend.ts | 16 ++- packages/workspace/src/parsers-adapter.ts | 114 +++++++++++++++++- pnpm-lock.yaml | 75 ++++++++++++ 9 files changed, 276 insertions(+), 16 deletions(-) create mode 100644 examples/fixtures/workspace-wiki/contract-terms.xlsx create mode 100644 examples/scripts/regen-xlsx-fixture.cjs diff --git a/examples/fixtures/workspace-wiki.README.md b/examples/fixtures/workspace-wiki.README.md index 840fcf3c..6289a43f 100644 --- a/examples/fixtures/workspace-wiki.README.md +++ b/examples/fixtures/workspace-wiki.README.md @@ -1,6 +1,6 @@ # workspace-wiki fixture -A 5-file OKF folder used by `examples/src/simple/22-workspace.ts` to demo +A 6-file OKF folder used by `examples/src/simple/22-workspace.ts` to demo `@melandlabs/workspace` end-to-end. The demo copies these files into a tmp directory before each run so every run starts from a clean slate. @@ -13,12 +13,15 @@ directory before each run so every run starts from a clean slate. | `law-clause.md` | `.md` | `note` (front-matter `type: statute`) | Public law clause — matches `a.md`'s 12-month cap. | | `law-brief.pdf` | `.pdf` | `document` | Same public law clause, distributed as PDF (real binary). | | `signed-addendum.docx` | `.docx` | `document` | Same clause as Word docx (real binary). | +| `contract-terms.xlsx` | `.xlsx` | `spreadsheet` | Liability cap matrix + indemnification carve-outs (real binary, 2 sheets). | The 3 markdown files form a cites graph (`a → b → a`) that the cross-file search strategy walks. The PDF and DOCX are byte-identical re-statements of the same law — they exercise the workspace parsers-adapter multi-format path (`@melandlabs/rag`'s `parseFileToDocument` → -`PDFLoader` / `DocxLoader`). +`PDFLoader` / `DocxLoader`). The XLSX exercises the SheetJS path — +`@langchain/community` ships no standalone Excel loader (only +`CSVLoader`), so `.xlsx` / `.xls` go straight to SheetJS. ## Format coverage @@ -34,9 +37,11 @@ across formats. Supported extensions (from workspace deps) - `.pages` — `parseFileToDocument` (macOS only, via `AppleDocumentLoader`) +- `.xlsx` / `.xls` — SheetJS (`xlsx`) — each sheet converted to CSV + with `# Sheet: ` headers so the chunker can preserve boundaries +- `.numbers` — `textutil -convert xlsx` (macOS only) → SheetJS path -Not supported (not in fixture): `.xlsx`, `.numbers` (spreadsheets), -`.png`/`.jpg`/… (raster). +Not supported (not in fixture): `.png`/`.jpg`/… (raster). ## Generating the binary files @@ -48,10 +53,14 @@ pandoc law-clause.md -o law-brief.pdf echo "Public Law — Cap of Liability …" > /tmp/law-raw.txt textutil -convert docx -output signed-addendum.docx /tmp/law-raw.txt rm /tmp/law-raw.txt + +# contract-terms.xlsx — generated by examples/scripts/regen-xlsx-fixture.cjs +# (a small node script that uses SheetJS to write the workbook; see +# the script for sheet contents). ``` This README is intentionally placed **outside** `workspace-wiki/` so the -walker's `SUPPORTED_EXTENSIONS` filter does not pick it up — only the 5 +walker's `SUPPORTED_EXTENSIONS` filter does not pick it up — only the 6 indexable files above are scanned. ## Demo assertions @@ -59,7 +68,7 @@ indexable files above are scanned. The demo runs lexical, semantic, hybrid, and cross-file queries against this folder and asserts: -- 5 resources appear in `listWorkspaceResources` +- 6 resources appear in `listWorkspaceResources` - lexical search for "limitation" returns ≥ 1 hit - re-running `updateWorkspaceContext` reports every file as `unchanged` (sha256 dedup) diff --git a/examples/fixtures/workspace-wiki/contract-terms.xlsx b/examples/fixtures/workspace-wiki/contract-terms.xlsx new file mode 100644 index 0000000000000000000000000000000000000000..be8ee7d930343efac67c8d3362a068b609696a20 GIT binary patch literal 18335 zcmeHPU5q5xRh|$BAx9=yK?os%ibgU)&P-3w-|mcOI-2R3*&XlB?r3_vc8nn0uDab_ zJN4(P>Yiysq{NYsAqo<~pya_4Z+<`!OO8Yy%mX|m@Q~NMMB>Fi5d_IX;sulkzH@H< z-RkO|pKTe1cvih#b?-Uno_p>&=bm$a`smKbAN%FG^zT2#=5OzOd~S|DzlOqdr~ZXd zI#K=14^KP3f4Xq)Itx^?`8hlpLm5W4?`_o@3yU>FdZusLUT>@R+{ym@&6*L#!n1_q zdvdEbl2Pq*%_kn;JeH1#ffV)aATmJ3i?(Y0I1V=I^~mf?S40bbAU)J{{m>OLUc+8J z5ay}q$@u+QP(uTd8Aq1;-_9c{nuZgo1@RCmF1O1d3}=; z8%#?X+lIJ|Nm!vcgDJa?vIbY!Ud`}?3wyfH)?%FSz&AG&;(HGCz>6a)FrBbXmMk?4 z*s0jCP8;+UjiQzl8bjg0kZjNL&rakyjLQ*GjEb$=BL9&&)iWd~o`74x+vmRoFF~+Q zm$=Qa*U1`f?60if*v;rOfD@0y?d|Qg_8J*I1`wu+o+@67G*|X-Hg-~)1t9e}PP?_Z zy11gl7w9jK@w~pXv$MLM@S=z^ka}DhgTJ}Bwz9pX!84G8Umb&A+1YNj)--qqQt)eI z@cTE`*H(0R2I@PucRB`51G>MTP|rZRzH_@0{3gI}E+*hP6tp4JE^4Cdd+{V)bHzhH z+y?*w(uCm|F=m9WFrmdQ;dVk>Py=E^ib9PlGNZ8~YRibs&<^1KpA7^oBB%Gk+kgAP z+iw{ky#3}ouYCKRSN{H;uYBd5SH7iaX^6Ll*DE6a_>DjRkJrCs{QFyf{o~jEvKk() zEpO<(zx(RD|L_;pfUuw(@cqAf^GDx)^Zjr9*}whGYZc(zq39IBPi$94#yxpv9Q!W% z#)8o~m!0rx7$<#UYZ#&r+7+bt;=TrWZzP;5xE-kr??DLXQ3ZHw@KEEm-46$`UD5KL zzN-Nq`o6Q{hm``jLv0l#d@}HQ)s2UPBHXbM!>YDh!qbL!ZxFz1rb?;FzSWl+n@5iD zV#Gt9j18*uPo=Er-REsvlk1^vhJNICW8?F-u_NqCk(}5aO_6e>+qR2w8CCTeL#@f? z@PV=8J5{~jl|vm81|c|dQs$hQ*okyBG2Rk`Sh!UrPXt1*!ob`WaleYm_9!%qaC=dV zk?u*yH})(UMO8TWL-e;m`;LIWsvP!1XXNUTVQimP(Yz~szX-YOpSJqK4XR+^5QDGeU2sXxz^w8HWgx)a@!kakU4{Qo+bH8!1BY4j4X)NUmLN$Z1f~{N9*YvkN za=JpYmw`*yJhf|k)7-XW!*8n=K62YzwQWpdRmO9>tkZJMOppf>v`HoJRJdgmf=50v`)Vi(Fhun;J#zAFFJ0mGThf zIU*4s`j#qQrs6c+$jC5PJPn}#=1h&0ag%}EgAt6i@TvP$Wpdyei-+Cc%h(goBJW= zefAyD7Oc>s8p6k}4u6MoNZ-ol6b&|OB!cm=-7`WP(@ES9r96ULV(q(tSR)+)M=?s#-ZW2^O`XaHKJ36n z88bRniX}*YQ~75svb6NTMd34gI!6{eLKS(Gw31qdHu7c|om;0QScc z$4#jn;FXhAy(Z&ICFOXJBC<;`UoEoA%;&NqQ;Fi#fx3ps;`8YuOC$y1jVm5@I75Nd ztIhrwlaA7KR@u^a@N@t5*lRC7HaAC~C)9Ru#GBl-nSA|?V5ZTg?dr=PyoiEO3X6|8 z=)1vubYo)($9cUI7M5=wg?MTYvKgB`1DHA5W> zY55kG1D<+m;K$Fr`<*|1?`yBt-~G;4@nxv8eqVjR|IJs_gX^mA_g?*D^$q-|<6($1q8fX_s6ELL~1ap8c{pjqT~KfrKx5t-HIO!P zyG*!F7-+ZShZauu6bC{TH8&mECACFV*Qf8;5Ac7-k7M7(gJp}J?+FKw^%Ns{09jKe z(bWrGjrMuBFu<69NeDF7^azHMM@3J)5d9)GMf;J-8t2`lF%bmL=-$BXNNga}A*5t% zxmC1N=#18tjH%@0!#zUA)>KBgGq4?mSry#?P3t@t3{UZ~=s1!C(ySE-#Zr^mNmjX4 z%byRWWe?o-4C=@p`f5tss^!lLYK6x>cM2sZ^u`b0di_^EF*iq_UsX=1^3c=u{ZU44 zX6^{_#Pz&U`AC_C?Qo9{w;7@AVws@ zN5|SW;UHUn80b(fYGv4LEa}C&I&z!O6o!`3SeobF(W2334c87sA1*5xbR646mU@;^ z(RxBrnCb~m!4-rKvN3Z`vU(mz`;6aeWE4~Bp(7u7*s`}Vxg55)iT|u6 zjn|co*A+i}Rk` za+HjBhO4o?nz4-~$%2UQ?QUlXN}_uGfoI3k?CUTsqBd+U>ZLG6jk0fGhr44tc06KU zKZCZQ^`0}zRbKI^&T1+*0C?T8Mik--|#OS(6x7m;A) z`9aZ=FJ1raH-F=c29 zzG7boD|yrtL_IwjW-yx8PjZpBaNZ^|JlVrS77j?u3|pCi?%cK!(V2<+&t0*yR~_RCgne=E9(LvTbVC1^3+ zbo^)#<|CmoHpzRD%c$l?_eGP0r9&MZ92oK(JD1?Yc&ZmaV_>-kk%o!LJ0Ez(*vwp( zGP+BnT3kenKO0`MG@WS2Q?W5xw z4BKlDf&dHg(t40xvq&~^Og|X$d=yXs2CtF7JP^)6VnZC071r~6ykl-e@NeO zQk?PqAO6Ju^2g@p=#v<2nwvb@(osrfW=1=+OfBW?D1?1k@*ZDOz-$yz7*Cg$vQQ&U z!xA05o5hUCl_CUpv@k~58=1(4jR(F)lm;pZ)`4e9*RyeBh_4_O&@e-1XuQUwJa-xv z5-X+AdiLRy|9$u8=H}@0Nw`>b+3Kg{f)%=}mYqtD_ez(yGCi(i>XY+9W|jFpnsl9z zA=1k<#k(}=jW5I_mu2YU8(*+*Bs&rj%`}#r}A7nHYZo8hTbc6+o)3ZGhc4zs9zazvrVS|YOX2M*=e^ve0;Ah^-`~!IAwk=0E6vM?_K$@K7MS%j8O2y|K`!V-1+Dx^7q|5)n7Q@>)dTngXPY45)B}wFqjN zsxpHMtDLO3vM(o&k@`$uzOelf(Sj6&eKe$ zx(gjQD2WJ1*m2RD%BwK~I@PGLq%f$w8lz`-D>10N8bi9(80Yg;Q;l;r5r=B3aUKTM zeR&wjNuA-Us{T*Gi6;mE<&!w$s;XwD-~?7#YDF`l+G;6a^98o84QUavLXQ(4L7{T~ zO8`$NhIk*pOAule$)I^q-z-A|!@OMtX_%RS;18#emr{LQx>ozak&% z#{xclK6ul-_*(&w!U7Y!pyT-wimNHx^x@}o8Pi(M0_-%XxmxR~s+SU^OSK3q;+)`7 zBwG=UPiq^7mc~nu+;oMZrAsL(3v~`I5Y2oH>CEI`GC@dF_?!H@CS^M1-(|>&yo_a0 z&?zf%nLzm@Pfp-Zev=90N?ArN@4x4$zxWiRW(HsU36e6E^3GFE<;ABfsjxdJQz 100 records"], +]; +XLSX.utils.book_append_sheet(workbook, XLSX.utils.aoa_to_sheet(indemnity), "Indemnification"); + +XLSX.writeFile(workbook, outPath); +console.log(`wrote ${outPath}`); diff --git a/examples/src/simple/22-workspace.ts b/examples/src/simple/22-workspace.ts index dc9b2efb..fdb42bf3 100644 --- a/examples/src/simple/22-workspace.ts +++ b/examples/src/simple/22-workspace.ts @@ -102,8 +102,15 @@ function makeRuntimeContext(): RuntimeContext { // copies each file into a tmp directory before running so every run starts // from a clean slate — the same fixtures also back the workspace tutorial. // The set covers every format @melandlabs/workspace's parsers-adapter -// claims to support: .md / .pdf / .docx (and .pages on macOS). -const FIXTURE_FILES = ["a.md", "b.md", "law-clause.md", "law-brief.pdf", "signed-addendum.docx"] as const; +// claims to support: .md / .pdf / .docx / .xlsx (and .pages / .numbers on macOS). +const FIXTURE_FILES = [ + "a.md", + "b.md", + "law-clause.md", + "law-brief.pdf", + "signed-addendum.docx", + "contract-terms.xlsx", +] as const; function resolveFixtureDir(): string { // examples/src/simple/22-workspace.ts → examples/fixtures/workspace-wiki @@ -176,13 +183,13 @@ export default async function demoWorkspace() { } check( - "updateWorkspaceContext returns ok with files_scanned ≥ 5", - update.filesScanned >= 5, + "updateWorkspaceContext returns ok with files_scanned ≥ 6", + update.filesScanned >= 6, `filesScanned=${update.filesScanned}, filesAdded=${update.filesAdded}`, ); check( "updateWorkspaceContext flags the new files as added", - update.filesAdded >= 5, + update.filesAdded >= 6, `filesAdded=${update.filesAdded}`, ); check("updateWorkspaceContext returns a positive jobId", update.jobId > 0, `jobId=${update.jobId}`); @@ -197,8 +204,8 @@ export default async function demoWorkspace() { workspace_id: WORKSPACE_ID, }); check( - "listWorkspaceResources returns ≥ 5 resources for the fixture folder", - listed.resources.length >= 5, + "listWorkspaceResources returns ≥ 6 resources for the fixture folder", + listed.resources.length >= 6, `total=${listed.total}, resources=${listed.resources.map((r) => r.canonical_key).join(", ")}`, ); @@ -325,7 +332,7 @@ export default async function demoWorkspace() { }); check( "re-running update reports every file as `unchanged` (sha256 dedup)", - reUpdate.filesUnchanged >= 5 && reUpdate.filesAdded === 0, + reUpdate.filesUnchanged >= 6 && reUpdate.filesAdded === 0, `unchanged=${reUpdate.filesUnchanged}, added=${reUpdate.filesAdded}`, ); diff --git a/packages/workspace/package.json b/packages/workspace/package.json index b2409e06..aecee703 100644 --- a/packages/workspace/package.json +++ b/packages/workspace/package.json @@ -47,6 +47,7 @@ "mammoth": "^1.11.0", "pdf-parse": "^2.0.0", "sqlite-vec": "^0.1.9", + "xlsx": "^0.18.5", "zod": "^4.3.6" }, "peerDependencies": { diff --git a/packages/workspace/src/okf-backend.ts b/packages/workspace/src/okf-backend.ts index 5d9c70f0..19758afe 100644 --- a/packages/workspace/src/okf-backend.ts +++ b/packages/workspace/src/okf-backend.ts @@ -18,7 +18,17 @@ import { extractText } from "./parsers-adapter"; import type { SqliteWorkspaceStore } from "./sqlite"; import type { OkfFolderResource, UpdateWorkspaceContextResult, WorkspaceEdgeType } from "./types"; -const SUPPORTED_EXTENSIONS = new Set([".md", ".markdown", ".txt", ".pdf", ".docx", ".pages"]); +const SUPPORTED_EXTENSIONS = new Set([ + ".md", + ".markdown", + ".txt", + ".pdf", + ".docx", + ".xlsx", + ".xls", + ".numbers", + ".pages", +]); async function walk(dir: string): Promise { const out: string[] = []; @@ -55,6 +65,10 @@ function resourceTypeForExtension(ext: string): string { return "document"; case ".docx": return "document"; + case ".xlsx": + case ".xls": + case ".numbers": + return "spreadsheet"; case ".pages": return "document"; default: diff --git a/packages/workspace/src/parsers-adapter.ts b/packages/workspace/src/parsers-adapter.ts index 23ad43db..f786e0d9 100644 --- a/packages/workspace/src/parsers-adapter.ts +++ b/packages/workspace/src/parsers-adapter.ts @@ -8,16 +8,28 @@ * - `.pdf` — `parseFileToDocument` (`@melandlabs/rag`) * - `.docx` — `parseFileToDocument` * - `.pages` — `parseFileToDocument` (via `AppleDocumentLoader`) + * - `.xlsx` / `.xls` — SheetJS (`xlsx`) — round-trips every sheet to + * CSV. `@langchain/community` does not ship an + * Excel loader, so we go straight to SheetJS + * instead of paying the langchain dependency + * tax. + * - `.numbers` — `textutil -convert xlsx` (macOS only), + * then the `.xlsx` path above. * * Binary raster files (`.png`, `.jpg`, …) intentionally raise an explicit * "unsupported" error so the caller can fall back to manual transcription. */ -import { readFile } from "node:fs/promises"; +import { execFile } from "node:child_process"; +import { readFile, unlink } from "node:fs/promises"; +import { tmpdir } from "node:os"; import { extname } from "node:path"; +import { promisify } from "node:util"; import { parseFile, parseFileToDocument } from "@melandlabs/rag"; import { estimateTokens } from "@melandlabs/shared"; +const execFileAsync = promisify(execFile); + let _parsersConfigured = false; function ensureParsersConfigured(): void { @@ -49,6 +61,8 @@ const MIME_BY_EXTENSION: Record = { ".pdf": "application/pdf", ".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document", ".doc": "application/msword", + ".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + ".xls": "application/vnd.ms-excel", ".pages": "application/x-iwork-pages-sffpages", ".numbers": "application/x-iwork-numbers-sffnumbers", ".keynote": "application/x-iwork-keynote-sffkeynote", @@ -66,6 +80,68 @@ export function detectMimeType(sourcePath: string): string { return MIME_BY_EXTENSION[ext] ?? "application/octet-stream"; } +/** + * Render a SheetJS workbook buffer as one CSV block per sheet. Used for + * `.xlsx` / `.xls` (and `.numbers` after a one-shot `textutil` conversion). + * + * Sheets are separated by `\n\n# Sheet: \n` so the chunker can + * preserve sheet boundaries without any extra metadata plumbing. + */ +async function extractSpreadsheet( + buffer: Buffer, + mimeType: string, + sourcePath: string, +): Promise { + const xlsxModule = await import("xlsx"); + // SheetJS exposes its surface as a namespace; some build entries also + // re-export it under `.default`. Coalesce both shapes here so we don't + // care which one `pnpm install` resolved at runtime. + const XLSX = ((xlsxModule as unknown as { default?: typeof xlsxModule }).default ?? + xlsxModule) as typeof xlsxModule; + type SheetJSModule = typeof import("xlsx"); + type Workbook = ReturnType; + const workbook = XLSX.read(buffer, { type: "buffer" }) as Workbook; + const blocks: string[] = []; + const sheetNames = workbook.SheetNames as string[]; + for (const name of sheetNames) { + const sheet = workbook.Sheets[name]; + if (!sheet) continue; + const csv = XLSX.utils.sheet_to_csv(sheet, { blankrows: false }); + if (!csv.trim()) continue; + blocks.push(`# Sheet: ${name}\n${csv.replace(/\n+$/, "")}`); + } + return { + text: blocks.join("\n\n"), + mimeType, + metadata: { + source: sourcePath, + sheets: sheetNames, + sheet_count: sheetNames.length, + }, + }; +} + +/** + * Convert an Apple Numbers workbook to .xlsx via the bundled + * `textutil` (macOS only). Returns the converted file path or null + * if the host is not macOS or `textutil` is missing. + */ +async function convertNumbersToXlsx(sourcePath: string): Promise { + if (process.platform !== "darwin") return null; + const outDir = tmpdir(); + try { + await execFileAsync("textutil", ["-convert", "xlsx", "-output", outDir, sourcePath]); + return `${outDir}/${ + sourcePath + .split("/") + .pop() + ?.replace(/\.numbers$/, ".xlsx") ?? "fixture.xlsx" + }`; + } catch { + return null; + } +} + /** * Read the source file, dispatch to the appropriate loader, and return * the canonical text body used by both the chunker and the embedder. @@ -79,6 +155,26 @@ export async function extractText(sourcePath: string, mimeType?: string): Promis return { text, mimeType: mime }; } const buffer = await readFile(sourcePath); + // Spreadsheets: SheetJS handles .xlsx / .xls directly. .numbers + // gets a one-shot textutil conversion on macOS, then falls through + // to the SheetJS path. Other platforms reject .numbers explicitly. + if (ext === ".xlsx" || ext === ".xls") { + return extractSpreadsheet(buffer, mime, sourcePath); + } + if (ext === ".numbers") { + const converted = await convertNumbersToXlsx(sourcePath); + if (!converted) { + throw new Error( + `.numbers parsing requires macOS (textutil); not available on ${process.platform}. Convert the file to .xlsx first.`, + ); + } + try { + const xlsxBuffer = await readFile(converted); + return extractSpreadsheet(xlsxBuffer, MIME_BY_EXTENSION[".xlsx"], sourcePath); + } finally { + unlink(converted).catch(() => undefined); + } + } // The RAG layer exposes `parseFile` (returns `{text, metadata}`) and // `parseFileToDocument` (returns a LangChain `Document`). The latter // also surfaces page-level metadata; prefer it when callers can @@ -103,6 +199,22 @@ export async function extractTextRaw(sourcePath: string, mimeType?: string): Pro const text = await readFile(sourcePath, "utf8"); return { text, mimeType: mime }; } + if (ext === ".xlsx" || ext === ".xls") { + const buffer = await readFile(sourcePath); + return extractSpreadsheet(buffer, mime, sourcePath); + } + if (ext === ".numbers") { + const converted = await convertNumbersToXlsx(sourcePath); + if (!converted) { + throw new Error(`.numbers parsing requires macOS (textutil); not available on ${process.platform}.`); + } + try { + const xlsxBuffer = await readFile(converted); + return extractSpreadsheet(xlsxBuffer, MIME_BY_EXTENSION[".xlsx"], sourcePath); + } finally { + unlink(converted).catch(() => undefined); + } + } const buffer = await readFile(sourcePath); const { text, metadata } = await parseFile(buffer, mime); return { text, mimeType: mime, metadata }; diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 09d093ca..1379b537 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -266,6 +266,9 @@ importers: '@melandlabs/vsa': specifier: ^0.3.0 version: link:../packages/vsa + '@melandlabs/workspace': + specifier: workspace:* + version: link:../packages/workspace better-sqlite3: specifier: ^11.7.0 version: 11.10.0 @@ -1560,6 +1563,9 @@ importers: sqlite-vec: specifier: ^0.1.9 version: 0.1.9 + xlsx: + specifier: ^0.18.5 + version: 0.18.5 zod: specifier: ^4.3.6 version: 4.4.3 @@ -4128,6 +4134,10 @@ packages: engines: {node: '>=0.4.0'} hasBin: true + adler-32@1.3.1: + resolution: {integrity: sha512-ynZ4w/nUUv5rrsR8UUGoe1VC9hZj6V5hU9Qw1HlMDJGEJw5S7TfTErWTjMys6M7vr0YWcPqs3qAr4ss0nDfP+A==} + engines: {node: '>=0.8'} + agent-base@6.0.2: resolution: {integrity: sha512-RZNwNclF7+MS/8bDg70amg32dyeZGZxiDuQmZxKLAlQjr3jGyLx+4Kkk58UO7D2QdgFIQCovuSuZESne6RG6XQ==} engines: {node: '>= 6.0.0'} @@ -4416,6 +4426,10 @@ packages: resolution: {integrity: sha512-Gmy6FhYlCY7uOElZUSbxo2UCDH8owEk996gkbrpsgGtrJLM3J7jGxl9Ic7Qwwj4ivOE5AWZWRMecDdF7hqGjFA==} engines: {node: '>=10'} + cfb@1.2.2: + resolution: {integrity: sha512-KfdUZsSOw19/ObEWasvBP/Ac4reZvAGauZhs6S/gqNhXhI7cKwvlH7ulj+dOEYnca4bm4SGo8C1bTAQvnTjgQA==} + engines: {node: '>=0.8'} + chai@5.3.3: resolution: {integrity: sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==} engines: {node: '>=18'} @@ -4512,6 +4526,10 @@ packages: resolution: {integrity: sha512-eYm0QWBtUrBWZWG0d386OGAw16Z995PiOVo2B7bjWSbHedGl5e0ZWaq65kOGgUSNesEIDkB9ISbTg/JK9dhCZA==} engines: {node: '>=6'} + codepage@1.15.0: + resolution: {integrity: sha512-3g6NUTPd/YtuuGrhMnOMRjFc+LJw/bnMp3+0r/Wcz3IXUuCosKRJvMphm5+Q+bvTVGcJJuRvVLuYba+WojaFaA==} + engines: {node: '>=0.8'} + color-convert@2.0.1: resolution: {integrity: sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==} engines: {node: '>=7.0.0'} @@ -4599,6 +4617,11 @@ packages: resolution: {integrity: sha512-tJtZBBHA6vjIAaF6EnIaq6laBBP9aq/Y3ouVJjEfoHbRBcHBAHYcMh/w8LDrk2PvIMMq8gmopa5D4V8RmbrxGw==} engines: {node: '>= 0.10'} + crc-32@1.2.2: + resolution: {integrity: sha512-ROmzCKrTnOwybPcJApAA6WBWij23HVfGVNKqqrZpuyZOHqK2CwHSvpGuyt/UNNvaIjEd8X5IFGp4Mh+Ie1IHJQ==} + engines: {node: '>=0.8'} + hasBin: true + croner@10.0.1: resolution: {integrity: sha512-ixNtAJndqh173VQ4KodSdJEI6nuioBWI0V1ITNKhZZsO0pEMoDxz539T4FTTbSZ/xIOSuDnzxLVRqBVSvPNE2g==} engines: {node: '>=18.0'} @@ -5239,6 +5262,10 @@ packages: resolution: {integrity: sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==} engines: {node: '>= 0.6'} + frac@1.1.2: + resolution: {integrity: sha512-w/XBfkibaTl3YDqASwfDUqkna4Z2p9cFSr1aHDt0WoMTECnRfBOv2WArlZILlqgWlmdIlALXGpM2AOhEk5W3IA==} + engines: {node: '>=0.8'} + fresh@2.0.0: resolution: {integrity: sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A==} engines: {node: '>= 0.8'} @@ -6803,6 +6830,10 @@ packages: sqlite-vec@0.1.9: resolution: {integrity: sha512-L7XJWRIBNvR9O5+vh1FQ+IGkh/3D2AzVksW5gdtk28m78Hy8skFD0pqReKH1Yp0/BUKRGcffgKvyO/EON5JXpA==} + ssf@0.11.2: + resolution: {integrity: sha512-+idbmIXoYET47hH+d7dfm2epdOMUDjqcB4648sTZ+t2JwoyBFL/insLfB/racrDmsKB3diwsDA696pZMieAC5g==} + engines: {node: '>=0.8'} + sswr@2.2.0: resolution: {integrity: sha512-clTszLPZkmycALTHD1mXGU+mOtA/MIoLgS1KGTTzFNVm9rytQVykgRaP+z1zl572cz0bTqj4rFVoC2N+IGK4Sg==} peerDependencies: @@ -7397,6 +7428,14 @@ packages: resolution: {integrity: sha512-LZNJgPzfKR+/J3cHkxcpHKpKKvGfDZVPS4hfJCc4cCG0CgYzvlD6yE/S3CIL/Yt91ak327YCpiF/0MyeZHEHKA==} engines: {node: '>= 12.0.0'} + wmf@1.0.2: + resolution: {integrity: sha512-/p9K7bEh0Dj6WbXg4JG0xvLQmIadrner1bi45VMJTfnbVHsc7yIajZyoSoK60/dtVBs12Fm6WkUI5/3WAVsNMw==} + engines: {node: '>=0.8'} + + word@0.3.0: + resolution: {integrity: sha512-OELeY0Q61OXpdUfTp+oweA/vtLVg5VDOXh+3he3PNzLGG/y0oylSOC1xRVj0+l4vQ3tj/bB1HVHv1ocXkQceFA==} + engines: {node: '>=0.8'} + wordwrap@1.0.0: resolution: {integrity: sha512-gvVzJFlPycKc5dZN4yPkP8w7Dc37BtP1yczEneOb4uq34pXZcvrtRTmWV8W+Ume+XCxKgbjM+nevkyFPMybd4Q==} @@ -7461,6 +7500,11 @@ packages: resolution: {integrity: sha512-sqMMuL1rc0FmMBOzCpd0yuy9trqF2yTTVe+E9ogwCSWQCdDEtQUwrZPT6AxqtsFGRNxycgncbP/xmOOSPw5ZUw==} engines: {node: '>= 6.0'} + xlsx@0.18.5: + resolution: {integrity: sha512-dmg3LCjBPHZnQp5/F/+nnTa+miPJxUXB6vtk42YjBBKayDNagxGEeIdWApkYPOf3Z3pm3k62Knjzp7lMeTEtFQ==} + engines: {node: '>=0.8'} + hasBin: true + xml-name-validator@5.0.0: resolution: {integrity: sha512-EvGK8EJ3DhaHfbRlETOWAS5pO9MZITeauHKJyb8wyajUfQUenkIg2MvLDTZ4T/TgIcm3HU0TFBgWWboAZ30UHg==} engines: {node: '>=18'} @@ -9846,6 +9890,8 @@ snapshots: acorn@8.18.0: {} + adler-32@1.3.1: {} + agent-base@6.0.2: dependencies: debug: 4.4.3 @@ -10133,6 +10179,11 @@ snapshots: camelcase@6.3.0: {} + cfb@1.2.2: + dependencies: + adler-32: 1.3.1 + crc-32: 1.2.2 + chai@5.3.3: dependencies: assertion-error: 2.0.1 @@ -10214,6 +10265,8 @@ snapshots: clsx@2.1.1: {} + codepage@1.15.0: {} + color-convert@2.0.1: dependencies: color-name: 1.1.4 @@ -10292,6 +10345,8 @@ snapshots: object-assign: 4.1.1 vary: 1.1.2 + crc-32@1.2.2: {} + croner@10.0.1: {} croner@8.1.2: {} @@ -10916,6 +10971,8 @@ snapshots: forwarded@0.2.0: {} + frac@1.1.2: {} + fresh@2.0.0: {} fs-constants@1.0.0: {} @@ -12660,6 +12717,10 @@ snapshots: sqlite-vec-linux-x64: 0.1.9 sqlite-vec-windows-x64: 0.1.9 + ssf@0.11.2: + dependencies: + frac: 1.1.2 + sswr@2.2.0(svelte@5.56.8): dependencies: svelte: 5.56.8 @@ -13394,6 +13455,10 @@ snapshots: triple-beam: 1.4.1 winston-transport: 4.9.0 + wmf@1.0.2: {} + + word@0.3.0: {} + wordwrap@1.0.0: {} wordwrapjs@5.1.1: {} @@ -13444,6 +13509,16 @@ snapshots: dependencies: os-paths: 4.4.0 + xlsx@0.18.5: + dependencies: + adler-32: 1.3.1 + cfb: 1.2.2 + codepage: 1.15.0 + crc-32: 1.2.2 + ssf: 0.11.2 + wmf: 1.0.2 + word: 0.3.0 + xml-name-validator@5.0.0: optional: true From 378a3aadd9ce20f0996ce90d6aca53ff09eda3bb Mon Sep 17 00:00:00 2001 From: Peefy Date: Fri, 11 Sep 2026 21:35:20 +0800 Subject: [PATCH 7/8] chore(changeset): document spreadsheet support in workspace CLI --- .changeset/workspace-spreadsheet-support.md | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 .changeset/workspace-spreadsheet-support.md diff --git a/.changeset/workspace-spreadsheet-support.md b/.changeset/workspace-spreadsheet-support.md new file mode 100644 index 00000000..392b6e6f --- /dev/null +++ b/.changeset/workspace-spreadsheet-support.md @@ -0,0 +1,10 @@ +--- +"@melandlabs/workspace": minor +"@melandlabs/opencontext": minor +--- + +Add Excel / Apple Numbers spreadsheet parsing to `@melandlabs/workspace`'s `parsers-adapter`. `.xlsx` and `.xls` are converted via SheetJS (`xlsx`) — one CSV block per sheet, prefixed with `# Sheet: ` so the chunker preserves sheet boundaries. `.numbers` files are first converted with macOS `textutil -convert xlsx`, then routed through the SheetJS path. + +The OKF walker now picks up `.xlsx`, `.xls`, and `.numbers` (macOS) alongside the existing `.md`, `.markdown`, `.txt`, `.pdf`, `.docx`, and `.pages` formats, and tags them with `resource_type: "spreadsheet"`. + +The 22-workspace demo now ships a 6-file fixture folder (`.md` × 3, `.pdf`, `.docx`, `.xlsx`) and asserts `filesScanned ≥ 6`, `filesAdded ≥ 6`, `listWorkspaceResources ≥ 6`, and that the re-run reports every file as `unchanged` under sha256 dedup. From 9d4504667294c512ed788b2b86296595dc289c87 Mon Sep 17 00:00:00 2001 From: Peefy Date: Fri, 11 Sep 2026 21:46:55 +0800 Subject: [PATCH 8/8] =?UTF-8?q?fix(examples):=20drop=20direct=20@melandlab?= =?UTF-8?q?s/workspace=20dep=20=E2=80=94=20comes=20via=20opencontext=20tra?= =?UTF-8?q?nsitively?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The smoke test does `pnpm install --ignore-workspace` to verify the published packages resolve cleanly. Listing `@melandlabs/workspace` as a direct dep with `workspace:*` made that step fail before the demo could even try its skip-on-missing-impot fallback. In monorepo dev the symlink is created transitively through `@melandlabs/opencontext`'s workspace dep, so the dynamic import in 22-workspace.ts still resolves. The npm-installed smoke test simply hits the existing skip path the demo was already designed for. --- examples/package.json | 1 - pnpm-lock.yaml | 3 --- 2 files changed, 4 deletions(-) diff --git a/examples/package.json b/examples/package.json index 66bcff37..f0f939a7 100644 --- a/examples/package.json +++ b/examples/package.json @@ -52,7 +52,6 @@ "@melandlabs/opencontext": "^0.9.0", "@melandlabs/rag": "^0.3.1", "@melandlabs/rss": "^0.3.0", - "@melandlabs/workspace": "workspace:*", "@melandlabs/search": "^0.3.0", "@melandlabs/security": "^0.3.0", "@melandlabs/shared": "^0.4.1", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 1379b537..377c1bb7 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -266,9 +266,6 @@ importers: '@melandlabs/vsa': specifier: ^0.3.0 version: link:../packages/vsa - '@melandlabs/workspace': - specifier: workspace:* - version: link:../packages/workspace better-sqlite3: specifier: ^11.7.0 version: 11.10.0