feat(workspace): folder indexing + cross-file hybrid search - #38
Open
Peefy wants to merge 8 commits into
Open
Conversation
Adds the @melandlabs/workspace package — a CLI for indexing an
OKF/Markdown folder into SQLite and querying it with lexical,
semantic, hybrid, and cross-file strategies.
What's in the box
- packages/workspace: new package with sqlite store, embedding
queue (class-internal serial Promise queue), 4 search strategies
(lexical / semantic / hybrid RRF / cross-file BFS over cites
edges), OKF backend, multi-format parsers adapter
(.md / .markdown / .txt / .pdf / .docx / .pages).
- Local embeddings default: workspace CLI defaults
EMBEDDING_PROVIDER=local (Xenova/all-MiniLM-L6-v2, 384 dims)
so demos run offline without OPENROUTER_API_KEY.
- opencontext CLI surface: `opencontext workspace update|search|list`
wired through the main bin.
- Re-exports from @melandlabs/opencontext for downstream consumers.
- Tutorial: docs/tutorials/use-cases/09-workspace-folder-indexing.md
covering the Quick Verification recipe, multi-format walkthrough,
and the local/cloud provider switch.
- Example: examples/src/simple/22-workspace.ts runs an end-to-end
sha256-dedup / lexical / semantic / cross-file demo against a
scratch store.
Storage
- Reuses ~/.opencontext/memory/store.db (same DB as memory-store).
- New tables: workspace_resources, workspace_resource_versions,
workspace_chunks + workspace_chunks_fts, workspace_reference_edges,
workspace_jobs, plus a per-dim vec0 child table
workspace_chunks_vec_d{384,1536}.
- FTS5 mirror tables and AI/AD/AU triggers mirror the existing
memory-store schema style.
Verified
- workspace 16/16, opencontext 109/109, memory-store 208/208,
okf 140/140, examples 282 OK / 3 SKIP / 0 FAIL.
- pnpm -w build clean.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI lint + format + changeset were failing on the workspace package: - biome: removed unused row→resource helper functions (toWorkspaceResourceVersion / toWorkspaceChunk / toWorkspaceReferenceEdge) and their backing interfaces (WorkspaceResourceVersionRow / WorkspaceReferenceEdgeRow) — they were dead code from when the file was first scaffolded. - biome: dropped the unused pickFrontMatterType helper in okf-backend.ts and the unused scheduleBackgroundClose stub in cli.ts. - biome: applied safe + unsafe fixes for useTemplate / noUnusedTemplateLiteral / noUnsafeDeclarationMerging across the workspace package. Renamed the SqliteWorkspaceStore interface to ISqliteWorkspaceStore to clear the unsafe-declaration-merging error (the class now `implements ISqliteWorkspaceStore`). - opencontext/src/cli/opencontext.ts: reordered the workspace import so biome's organizeImports is happy. - Added .changeset/workspace-folder-indexing.md bumping @melandlabs/workspace (minor) and @melandlabs/opencontext (minor). Verified locally: - pnpm -w build clean - pnpm -w format:check clean - pnpm -r lint clean (workspace + opencontext) - workspace 16/16, opencontext 109/109, memory-store 208/208, okf 140/140 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… is absent
The smoke test pulls @melandlabs/* from npmjs.org via
`pnpm install --ignore-workspace`. @melandlabs/workspace isn't
published yet, so a static `import { ... } from "@melandlabs/opencontext"`
of the workspace re-exports throws ERR_MODULE_NOT_FOUND at module
load time and breaks the entire smoke suite.
Switch the demo to a top-level `import type { ... }` (erased at
compile time, no runtime resolution) plus an `await import(
"@melandlabs/workspace")` inside the demo body, wrapped in try/catch.
If the dynamic import fails we record a single [SKIP] and return,
so the rest of the suite keeps running.
This also makes the demo robust in the local monorepo before
workspace is symlinked into examples/node_modules — the demo
will start running for real once workspace is added as a
dependency in examples/package.json after the first publish.
Verified locally:
- examples pnpm test: exit 0, demo/workspace → [SKIP] with reason
- 307 OK lines, 0 FAIL
- pnpm -r lint clean
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously the 22-workspace demo wrote its 3-file contract / statute fixture inline via writeFile at the top of buildFixture(). That kept the markdown content trapped inside the demo file with no way to edit / reuse / diff it independently. Move the fixture to examples/fixtures/workspace-wiki/: - a.md (contract, Limitation of Liability, cites b.md) - b.md (contract, indemnification, cites a.md) - law-clause.md (statute, matches a.md's 12-month cap) - README.md describing the role of each file 22-workspace.ts now resolves the fixture dir relative to its own URL and copyFile's each markdown into a per-run tmp dir, so the fixture is the source of truth and the demo always starts from a clean slate. Verified: - workspace demo passes all 11 [OK] checks when @melandlabs/workspace is symlinked into examples/node_modules (lexical / semantic / cross-file / sha256-dedup re-run) - pnpm format:check clean - pnpm -r lint clean Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The fixture folder only contained .md files, but the workspace parsers-adapter advertises support for .pdf / .docx / .pages and the tutorial documents that. This commit makes the demo actually exercise those formats end-to-end. Changes: - packages/workspace/package.json: declare pdf-parse ^2.0.0 and mammoth ^1.11.0 as runtime deps. These are dynamically imported by @langchain/community's PDFLoader / DocxLoader (via @melandlabs/rag); without them the multi-format parsing path ERR_MODULE_NOT_FOUNDs at runtime. - examples/fixtures/workspace-wiki/law-brief.pdf (53 KB, pandoc output of law-clause.md). - examples/fixtures/workspace-wiki/signed-addendum.docx (3.6 KB, textutil-generated). - examples/fixtures/workspace-wiki.README.md: moved out of the indexed folder so the walker doesn't pick it up as a .md resource. Documents the role of every fixture file and the supported format matrix. - examples/src/simple/22-workspace.ts: extend FIXTURE_FILES to include law-brief.pdf + signed-addendum.docx; bump the >=3 assertions to >=5 (files_scanned, files_added, listWorkspaceResources, sha256-dedup re-run). Verified locally with the workspace package symlinked into examples/node_modules: - listOkfFolderResources → 5 resources (3 .md, 1 .pdf, 1 .docx), body extraction succeeds for all five. - workspace demo 12/12 [OK]: scanned=5, added=5, lexical search for 'limitation' returns 3 hits (now includes the DOCX), cross-file expansion surfaces a + b + signed-addendum, re-run reports all 5 unchanged. - workspace tests 16/16 - pnpm -r lint clean, pnpm format:check clean Not included (not supported by the parsers-adapter yet): .xlsx, .numbers, raster. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- parsers-adapter: dynamic-import SheetJS, convert every sheet to CSV with a `# Sheet: <name>` header so the chunker preserves boundaries; .numbers goes through macOS `textutil -convert xlsx` then SheetJS - okf-backend: extend SUPPORTED_EXTENSIONS with .xlsx / .xls / .numbers and map them to `resource_type: spreadsheet` - fixture: add contract-terms.xlsx (2 sheets — Liability Caps + Indemnification) plus a regen script; demo now asserts ≥ 6 resources and the cross-file search hits the spreadsheet alongside the .md/.pdf/.docx - examples/package.json: add @melandlabs/workspace as a dev dep so the dynamic import in 22-workspace.ts resolves to the local symlink in dev (the npm-installed smoke test still falls through to skip cleanly)
…context transitively The smoke test does `pnpm install --ignore-workspace` to verify the published packages resolve cleanly. Listing `@melandlabs/workspace` as a direct dep with `workspace:*` made that step fail before the demo could even try its skip-on-missing-impot fallback. In monorepo dev the symlink is created transitively through `@melandlabs/opencontext`'s workspace dep, so the dynamic import in 22-workspace.ts still resolves. The npm-installed smoke test simply hits the existing skip path the demo was already designed for.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
@melandlabs/workspace— a CLI for indexing an OKF / Markdown folder into SQLite and querying it with lexical / semantic / hybrid / cross-file strategies. Built on the existing memory-store SQLite infra and shared via theopencontextfacade.What's in the box
packages/workspace(sqlite store, embedding queue, 4 search strategies, OKF backend, multi-format parsers).opencontext workspace update|search|listwired through the main bin.EMBEDDING_PROVIDER=local(Xenova/all-MiniLM-L6-v2, 384 dims). Demos and OKF review workflows run offline — noOPENROUTER_API_KEYneeded..md/.markdown/.txtpass-through,.pdf/.docx/.pagesgo throughpackages/ragparsers.workspace_reference_edgesextracted from Markdown[text](./other.md)links.@melandlabs/opencontextfor downstream consumers.docs/tutorials/use-cases/09-workspace-folder-indexing.md(Quick Verification recipe + walkthrough).examples/src/simple/22-workspace.ts(sha256 dedup / lexical / semantic / cross-file end-to-end).Storage
Reuses
~/.opencontext/memory/store.db. New tables:workspace_resources(project-scoped)workspace_resource_versions(sha256-keyed version chain)workspace_chunks+workspace_chunks_fts(FTS5 mirror w/ AI/AD/AU triggers)workspace_reference_edges(cites edges from Markdown links)workspace_jobs(indexing job log)workspace_chunks_vec_d{384,1536}(per-dim vec0 child table)Test results
@melandlabs/workspace@melandlabs/opencontext@melandlabs/memory-store@melandlabs/okfexamplespnpm -w buildclean.Usage
Co-Authored-By
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com