Skip to content

feat(indexer): git-aware untracked-asset skip + bounded init store - #182

Merged
zzet merged 2 commits into
mainfrom
feat/index-git-aware-untracked-and-init-store
Jun 27, 2026
Merged

feat(indexer): git-aware untracked-asset skip + bounded init store#182
zzet merged 2 commits into
mainfrom
feat/index-git-aware-untracked-and-init-store

Conversation

@zzet

@zzet zzet commented Jun 27, 2026

Copy link
Copy Markdown
Owner

Summary

Two follow-ups to #181 (content-admission caps), both targeting the #120 OOM from a different angle. Stacked on #181 — base is fix/index-content-admission-caps; review/merge after it (retarget to main once #181 lands).

Motivation came from the reporter's actual intake.json: the 4.42 GiB admitted corpus is untracked RAG scratch data, and the OOM happens on gortex init, which indexed entirely in RAM.

A — opt-in skip of untracked non-source assets (index.skip_untracked_assets)

respect_gitignore is already on, but .gitignore can't catch files that are merely untracked (never git added, not ignored) — which is exactly how uncommitted RAG corpora / datasets / build outputs get admitted.

  • New index.skip_untracked_assets (default off, per request — current behaviour is unchanged unless enabled).
  • When on, a full-index walk builds the repo's git ls-files set once and drops asset-class files (document/data/image) git doesn't track, with an untracked_asset skip node.
  • Untracked CODE is still indexed, so new/unsaved source keeps working.
  • Inert on a non-git repo or when git ls-files is unavailable (everything admitted as before). Applies to the cold full-index walk; the incremental watcher path keeps the size/class caps only.

For the reporter: flipping this on removes the entire untracked 4.42 GiB with zero per-file tuning.

D — gortex init indexes into a temporary on-disk store

indexRepoForInit used graph.New() (all-in-memory), with no content sink — so document/section text was never leaned and the whole post-parse graph was pinned in RAM. That's the path the reporter's OOM most likely hit (parsing got 2040 files … killed).

  • Init now indexes into a temporary sqlite store: nodes persist per file and the content sink leans section text to disk, so init inherits the daemon's shadow / byte-budget guards.
  • The store + temp dir are removed once skill generation / coverage have read the graph.
  • Both consumers (genskills.Build, parity.CoverageOf) already take graph.Store, so the change is confined to indexRepoForInit and its two callers.

Validation

  • go build ./... (CGO) — success
  • go test -race ./internal/indexer/ ./internal/config/... — 768 passed (incl. new untracked-asset tests with a real git repo)
  • go test ./cmd/gortex/ -run 'Init|Parity|Wire|Intake' — 47 passed
  • go vet + golangci-lint on changed packages — clean
  • Smoke: built the binary, ran gortex init on a temp git repo → succeeds, 0 leftover temp stores (cleanup verified)

Relates to #120.

zzet added 2 commits June 27, 2026 12:56
`.gitignore` is honoured already, but it can't catch files that are
merely untracked — never `git add`ed and not ignored — which is exactly
how uncommitted RAG corpora, downloaded datasets, and build outputs get
admitted and blow up the index (#120).

Add `index.skip_untracked_assets` (default off). When on, a full-index
walk builds the repo's `git ls-files` set once and drops asset-class
files (document/data/image) git does not track, with an `untracked_asset`
skip node. Untracked CODE is still indexed, so new/unsaved source keeps
working. Inert on a non-git repo or when git is unavailable.
`gortex init` indexed into an all-in-memory graph (graph.New()), so a
content-heavy repo accumulated the whole post-parse graph — including
full document/section text, which the in-memory store never leans — and
could OOM during onboarding (#120).

Index into a temporary sqlite store instead: nodes persist per file and
the content sink leans section text to disk, so init inherits the same
shadow / byte-budget guards the daemon already uses. The store and its
temp dir are removed once skill generation / coverage have read the
graph. Both consumers (genskills.Build, parity.CoverageOf) already take
graph.Store, so the change is confined to indexRepoForInit and its two
callers.
Base automatically changed from fix/index-content-admission-caps to main June 27, 2026 11:10
@zzet
zzet merged commit 51beef3 into main Jun 27, 2026
1 check passed
@zzet
zzet deleted the feat/index-git-aware-untracked-and-init-store branch June 27, 2026 11:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant