feat(indexer): git-aware untracked-asset skip + bounded init store - #182
Merged
Conversation
`.gitignore` is honoured already, but it can't catch files that are merely untracked — never `git add`ed and not ignored — which is exactly how uncommitted RAG corpora, downloaded datasets, and build outputs get admitted and blow up the index (#120). Add `index.skip_untracked_assets` (default off). When on, a full-index walk builds the repo's `git ls-files` set once and drops asset-class files (document/data/image) git does not track, with an `untracked_asset` skip node. Untracked CODE is still indexed, so new/unsaved source keeps working. Inert on a non-git repo or when git is unavailable.
`gortex init` indexed into an all-in-memory graph (graph.New()), so a content-heavy repo accumulated the whole post-parse graph — including full document/section text, which the in-memory store never leans — and could OOM during onboarding (#120). Index into a temporary sqlite store instead: nodes persist per file and the content sink leans section text to disk, so init inherits the same shadow / byte-budget guards the daemon already uses. The store and its temp dir are removed once skill generation / coverage have read the graph. Both consumers (genskills.Build, parity.CoverageOf) already take graph.Store, so the change is confined to indexRepoForInit and its two callers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two follow-ups to #181 (content-admission caps), both targeting the #120 OOM from a different angle. Stacked on #181 — base is
fix/index-content-admission-caps; review/merge after it (retarget tomainonce #181 lands).Motivation came from the reporter's actual
intake.json: the 4.42 GiB admitted corpus is untracked RAG scratch data, and the OOM happens ongortex init, which indexed entirely in RAM.A — opt-in skip of untracked non-source assets (
index.skip_untracked_assets)respect_gitignoreis already on, but.gitignorecan't catch files that are merely untracked (nevergit added, not ignored) — which is exactly how uncommitted RAG corpora / datasets / build outputs get admitted.index.skip_untracked_assets(default off, per request — current behaviour is unchanged unless enabled).git ls-filesset once and drops asset-class files (document/data/image) git doesn't track, with anuntracked_assetskip node.git ls-filesis unavailable (everything admitted as before). Applies to the cold full-index walk; the incremental watcher path keeps the size/class caps only.For the reporter: flipping this on removes the entire untracked 4.42 GiB with zero per-file tuning.
D —
gortex initindexes into a temporary on-disk storeindexRepoForInitusedgraph.New()(all-in-memory), with no content sink — so document/section text was never leaned and the whole post-parse graph was pinned in RAM. That's the path the reporter's OOM most likely hit (parsing got 2040 files … killed).genskills.Build,parity.CoverageOf) already takegraph.Store, so the change is confined toindexRepoForInitand its two callers.Validation
go build ./...(CGO) — successgo test -race ./internal/indexer/ ./internal/config/...— 768 passed (incl. new untracked-asset tests with a real git repo)go test ./cmd/gortex/ -run 'Init|Parity|Wire|Intake'— 47 passedgo vet+golangci-linton changed packages — cleangortex initon a temp git repo → succeeds, 0 leftover temp stores (cleanup verified)Relates to #120.