Skip to content

fix(indexer): cap large document assets and skip vector data by default - #181

Merged
zzet merged 1 commit into
mainfrom
fix/index-content-admission-caps
Jun 27, 2026
Merged

fix(indexer): cap large document assets and skip vector data by default#181
zzet merged 1 commit into
mainfrom
fix/index-content-admission-caps

Conversation

@zzet

@zzet zzet commented Jun 27, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes the content-repo OOM in #120. A RAG corpus (the reporter's repo: ~2,045 admitted files / ~4.42 GiB, ~85% pptx/xlsx/pdf, ~99% non-source documents + vector data) could exhaust memory during indexing because every document and data artifact was admitted into the read+extract pipeline with no per-class cap and no opt-out — so a few hundred large non-source files pulled gigabytes into memory.

Prior fixes bounded the symptom (#132 large-read gate, #137 bytes-in-flight semaphore + shadow-bytes guard + content/code split) but not the cause: there was no corpus admission control. This adds one.

What changed

A walk-time corpus-admission gate drops non-source artifacts before they are read and extracted:

  • Non-code extractors declare an AssetClass (document / data / image) via a new parser.AssetExtractor marker interface. The indexer builds a language -> class map from the registry once per walk (inert for all-code repos).
  • Documents (pdf/pptx/xlsx/text) over a per-file cap (default 10 MiB) are skipped with skip_reason: large_document.
  • Binary/vector data (parquet/npy/npz/lance/arrow/feather) are skipped entirely by default with skip_reason: vector_data.

Skipped files become synthetic KindFile nodes (skipped_due_to_content) so they stay listable and roll up in index_health, mirroring the existing size/timeout skip telemetry. The gate runs on both the cold IndexCtx walk and the incremental indexFile (watcher) path, so the live path can't re-admit what the cold path skipped.

Markdown prose and code are never gated.

Configuration

All behaviour is configurable under index.content:

index:
  content:
    max_document_bytes: 10485760   # >0 cap | 0 default | <0 no cap
    index_data: false              # opt in to admit data assets
    max_data_bytes: 67108864       # cap applied only when index_data is on

Caps are tri-state (>0 cap / 0 built-in default / <0 no cap), so a zero-valued config still yields the correct defaults regardless of how it was constructed. Users who want the prior behaviour set max_document_bytes: -1 and index_data: true.

Behaviour change

This changes defaults for existing users: documents over 10 MiB and all vector/data artifacts are no longer indexed unless opted in. This is the intended outcome per the issue discussion — the multimodal feature stays intact for normal-size documents.

Validation

  • go build ./... (CGO) — success
  • go test -race ./internal/config/... ./internal/parser/... — 1942 passed
  • go test -race ./internal/indexer/ — 668 passed (incl. new content_admission_test.go)
  • go vet + golangci-lint on changed packages — clean
  • cmd/gortex wire-contract golden — unaffected (no graph node/edge field change)
  • Confirmed no test fixtures regress (data-asset files only exist under eval/.venv/)

Notes

The in-memory gortex init path (graph.New, nil content sink) still retains full section text; the admission gate shrinks how much it accumulates, but unconditional content-node leaning was intentionally skipped to avoid regressing in-memory content search on the --backend memory case. Left as a possible follow-up.

Closes #120

Content-heavy repos (RAG corpora of decks, spreadsheets, PDFs, and
dataset shards) could OOM the indexer: every document and data artifact
was admitted into the read+extract pipeline with no per-class cap, so a
few hundred large non-source files pulled gigabytes into memory (#120).

Add a walk-time corpus-admission gate. Non-code extractors now declare an
AssetClass (document / data / image) via a marker interface; the indexer
builds a language->class map from the registry once per walk and drops
artifacts before they are read:

  - documents (pdf/pptx/xlsx/text) over a per-file cap (default 10 MiB)
    -> skip_reason "large_document"
  - binary/vector data (parquet/npy/npz/lance/arrow/feather) skipped
    entirely by default -> skip_reason "vector_data"

Skipped files become synthetic file nodes (skipped_due_to_content) so
they stay listable and roll up in index_health, mirroring the existing
size/timeout skip telemetry. The gate runs on both the cold IndexCtx
walk and the incremental indexFile path.

All behaviour is configurable under index.content (max_document_bytes,
index_data, max_data_bytes). Caps are tri-state (>0 cap / 0 default /
<0 no cap), so a zero-valued config still yields the correct defaults;
users who want the prior behaviour set max_document_bytes: -1 and
index_data: true.
@zzet
zzet merged commit 7dd56e4 into main Jun 27, 2026
9 checks passed
@zzet
zzet deleted the fix/index-content-admission-caps branch June 27, 2026 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gortex init OOM

1 participant