fix(indexer): cap large document assets and skip vector data by default - #181
Merged
Conversation
Content-heavy repos (RAG corpora of decks, spreadsheets, PDFs, and dataset shards) could OOM the indexer: every document and data artifact was admitted into the read+extract pipeline with no per-class cap, so a few hundred large non-source files pulled gigabytes into memory (#120). Add a walk-time corpus-admission gate. Non-code extractors now declare an AssetClass (document / data / image) via a marker interface; the indexer builds a language->class map from the registry once per walk and drops artifacts before they are read: - documents (pdf/pptx/xlsx/text) over a per-file cap (default 10 MiB) -> skip_reason "large_document" - binary/vector data (parquet/npy/npz/lance/arrow/feather) skipped entirely by default -> skip_reason "vector_data" Skipped files become synthetic file nodes (skipped_due_to_content) so they stay listable and roll up in index_health, mirroring the existing size/timeout skip telemetry. The gate runs on both the cold IndexCtx walk and the incremental indexFile path. All behaviour is configurable under index.content (max_document_bytes, index_data, max_data_bytes). Caps are tri-state (>0 cap / 0 default / <0 no cap), so a zero-valued config still yields the correct defaults; users who want the prior behaviour set max_document_bytes: -1 and index_data: true.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the content-repo OOM in #120. A RAG corpus (the reporter's repo: ~2,045 admitted files / ~4.42 GiB, ~85% pptx/xlsx/pdf, ~99% non-source documents + vector data) could exhaust memory during indexing because every document and data artifact was admitted into the read+extract pipeline with no per-class cap and no opt-out — so a few hundred large non-source files pulled gigabytes into memory.
Prior fixes bounded the symptom (#132 large-read gate, #137 bytes-in-flight semaphore + shadow-bytes guard + content/code split) but not the cause: there was no corpus admission control. This adds one.
What changed
A walk-time corpus-admission gate drops non-source artifacts before they are read and extracted:
AssetClass(document/data/image) via a newparser.AssetExtractormarker interface. The indexer builds alanguage -> classmap from the registry once per walk (inert for all-code repos).skip_reason: large_document.skip_reason: vector_data.Skipped files become synthetic
KindFilenodes (skipped_due_to_content) so they stay listable and roll up inindex_health, mirroring the existing size/timeout skip telemetry. The gate runs on both the coldIndexCtxwalk and the incrementalindexFile(watcher) path, so the live path can't re-admit what the cold path skipped.Markdown prose and code are never gated.
Configuration
All behaviour is configurable under
index.content:Caps are tri-state (
>0cap /0built-in default /<0no cap), so a zero-valued config still yields the correct defaults regardless of how it was constructed. Users who want the prior behaviour setmax_document_bytes: -1andindex_data: true.Behaviour change
This changes defaults for existing users: documents over 10 MiB and all vector/data artifacts are no longer indexed unless opted in. This is the intended outcome per the issue discussion — the multimodal feature stays intact for normal-size documents.
Validation
go build ./...(CGO) — successgo test -race ./internal/config/... ./internal/parser/...— 1942 passedgo test -race ./internal/indexer/— 668 passed (incl. newcontent_admission_test.go)go vet+golangci-linton changed packages — cleancmd/gortexwire-contract golden — unaffected (no graph node/edge field change)eval/.venv/)Notes
The in-memory
gortex initpath (graph.New, nil content sink) still retains full section text; the admission gate shrinks how much it accumulates, but unconditional content-node leaning was intentionally skipped to avoid regressing in-memory content search on the--backend memorycase. Left as a possible follow-up.Closes #120