[feature] Context ingestion integrations for third-party frameworks (e.g. LlamaIndex) across varied file inputs

[Peefy]

Problem

Teams who adopted retrieval frameworks like LlamaIndex (or LangChain) before — or alongside — OpenContext hold their corpus behind framework loaders/readers: PDFs, CSVs, DOCX, Markdown, HTML, notebooks, ZIP archives, images needing OCR, and so on. Today the path into OpenContext is DIY: pull documents out of the framework, reshape them into raw messages / knowledge records yourself, then call the memory API.

The building blocks already exist on our side — parsers (PDF/ZIP/text), chunking, embeddings, and the sqlite-vec / pgvector / Chroma adapters — but there is no supported bridge from common frameworks' document models into OpenContext records, so every integration reinvents the same glue.

Proposed solution

A small adapter layer per framework plus explicit multi-format file ingestion entry points.

1. Framework adapter (LlamaIndex first)

Map Document/Node objects straight into OpenContext knowledge / raw-message records:

import { llamaIndexAdapter } from "@melandlabs/opencontext/adapters";
import { SimpleDirectoryReader } from "llamaindex";

const documents = await new SimpleDirectoryReader().loadData("./knowledge-base");

// Each doc becomes a chunked, embedded, searchable record with provenance preserved
await llamaIndexAdapter().ingest(documents, {
	userId: "u-42",
	kind: "knowledge",
	source: (doc) => doc.metadata.file_path, // provenance: file://...
});

// Recall flows back through the unified search API
const hits = await llamaIndexAdapter().retrieve({
	userId: "u-42",
	query: "What did we decide about embedding providers?",
});

2. File-input coverage matrix

State and test which inputs each adapter handles, and route unsupported ones through the existing parser registry (with OCR as an optional opt-in extension):

Input	Source	Status
PDF, TXT	existing parsers	wire-through
DOCX, HTML, EPUB	new parsers	route via parser registry
CSV / Excel rows	new mapper	row → message per row
ZIP archives	existing parser	recursive walk
Images (screenshots, scans)	OCR extension	v2 / opt-in

3. CLI sugar

Belt-and-braces for non-framework users:

opencontext ingest ./docs --user u-42 --kind knowledge
opencontext ingest ./report.pdf --source "file://reports/q4.pdf"

Scope notes

• Start with LlamaIndex; LangChain-style loaders can follow the same adapter interface (ingest(records, opts) + retrieve(query, opts)).
• Adapters should be additive peer-dependency entrypoints so the core package stays dependency-light.
• Deterministic first: ingest runs without any LLM roundtrip; synthesis stays in the agent layer.

Alternatives considered

• Point users at each framework's own vector-store integrations: loses the temporal graph, provenance, corrections, and unified search — the actual reasons to bring data into OpenContext.
• Only expand the internal parser list (no framework adapters): helps greenfield projects, still leaves every LlamaIndex/LangChain team writing bespoke glue.

Where does this change live?

A new adapters/ module (or additional exports in the existing integrations / rag packages) plus an optional ingest subcommand in packages/opencontext/src/cli/.