feat: index freshness, staleness & watch robustness - #91
Merged
Conversation
Mix a per-language extractor version into each Merkle leaf so a file whose content is unchanged but whose extractor logic was upgraded re-extracts on the next reconcile — surgically, by language, without re-reading content. An empty salt reproduces the content-only leaf exactly, so the registry is dormant (zero behaviour change) until a language version is deliberately bumped.
Persist a repo_index_state row at the end of every (re)index: the git SHA + dirty flag the graph reflects, the Merkle workspace fingerprint, node/ edge counts for plausibility baselining, and the per-language extractor versions that produced the graph. Optional capability — backends without durable state skip the write, like the file-mtime ledger.
Report db_bytes / wal_bytes / wal_db_ratio in the daemon_health snapshot so a runaway WAL high-water mark is observable instead of silently filling the disk. Optional capability via a DBStatReporter type-assertion — in-memory backends contribute nothing. (The global-pass short-circuit on an unchanged workspace already exists via the warmup changedRepos / ReconcileAll guards; the workspace fingerprint digest is now persisted in repo_index_state.)
…eason Previously a plain extraction error (not a panic, timeout, or size cap) was dropped silently — the file vanished from the graph with no trace. Capture it as a synthetic KindFile skip node, in both the in-process and subprocess extraction paths, and stamp a uniform skip_reason (size / timeout / minified / parse_failed / parse_panic) on every skip-node shape so the reasons roll up. The Merkle reconcile already retries a failed file when its content or extractor version changes, so no separate retry ledger is needed.
index_health now reports a skipped:{size,timeout,minified,parse_failed,
parse_panic} breakdown so an agent can see WHY a symbol is missing instead
of guessing, plus a nodes_per_file density with a soft warning when a
populated graph has almost no symbols beyond file shells (a broken grammar
or aborted reindex) — complementing the existing zero-edge sanity check.
…ed lines A corrupt or DLP-encrypted .gitignore previously fed garbage lines straight to the exclude matcher and could silently truncate the pattern list on the default 64 KiB scanner limit. Skip non-UTF-8 lines, raise the scanner buffer, and keep the patterns already read on any error — exclusion loading is best-effort and must never abort or mis-scan.
The inline parse-failed skip node broke the load-bearing invariant that a transient mid-edit parse failure must keep a file's prior symbols (the live watcher path must not zero a file's nodes). Mirror the size-skip mechanism instead: the live/re-extract paths keep prior nodes on a parse failure (unchanged), and the full-index path collects failed files and emits skip nodes in a post-pass — visible telemetry without the destructive replace.
When a file-reading tool (read_file / get_symbol_source / get_editing_context / get_file_summary / get_symbol) returns content for a file that changed on disk since it was indexed, attach a small structured freshness block so the agent sees, inline, that the graph view may lag the working tree — with the indexed SHA and the dirty-at-index flag from repo_index_state. Omitted (zero extra tokens) for the common fresh case and in multi-repo mode; the check is O(1) via a new IsTrackedStale that never false-positives on untracked paths.
Add typed Phase constants (discover/parse/resolve/link/persist), a human duration formatter (45s / 5m 12s / 1h 20m), and an ETA/throughput computation (items/sec + estimated remaining), surfaced on the warmup parse phase in the workspace-readiness payload so index progress is legible and estimable.
Add a fast (250ms) notify-file loop to the adaptive poller: touching <root>/.gortex/reindex.notify (or $GORTEX_NOTIFY_FILE) forces an immediate HEAD+filesystem reconcile, giving an agent or hook sub-second re-index latency instead of waiting out the adaptive interval. Register a post-checkout git hook that touches the notify file, so a branch switch reconciles the working tree at once. Poller teardown moved to a WaitGroup to join both loops.
…clude Add a dedicated `include` list to .gortex.yaml: each entry is appended last as a gitignore !pattern re-include, so it wins over the builtin / .gitignore / global / repo exclude layers. The readable counterpart to hand-writing negations, for a vendored or generated tree (Pods/, a per-customer dir) you want in the graph despite .gitignore.
Detect a Windows drive surfaced into WSL2 via 9p/drvfs (or an SMB share) by probing /proc/version + statfs, and on such a mount skip the native fsnotify backend — where inotify events arrive late or never and confirmWatchActive would hang ~5s per path — relying on the adaptive poller + git hooks instead. GORTEX_FORCE_FSNOTIFY=1 overrides. No-op on non-Linux and normal mounts.
When GORTEX_AUTOINDEX=1, the first tool call in a session background-indexes the current working directory if it is an untracked git repo under a tracked file-count limit. Off by default (auto-indexing is expensive — the user opts in), bounded (an oversized tree is left for an explicit track), and fully backgrounded so the request path pays only a getenv + a sync.Once.
…onciles The implements/override inference passes are already scoped to the changed- affected type set (runScopedInferencePasses, add-parity with the full pass). The remaining whole-graph passes — capability-edge and framework-dispatch synthesis — derive only from code structure, so a reconcile that touched only non-code files (a README, a config) and removed nothing now skips them entirely. They stay whole-graph on a genuine code change: sound cross-file scoping needs an affected-set those synthesizers do not expose, and an unsound scope would silently diverge the graph from a full pass.
When the working directory is a linked git worktree that the indexed graph does not cover, read-tool responses now carry a worktree_mismatch flag — the agent is in one checkout while the graph reflects another, so results may not match the files on disk. Computed once per session and folded into the existing inline freshness rider (single-repo; multi-repo owns its own worktree routing).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A self-describing freshness/provenance layer for the graph: every file knows how it was extracted and whether the graph still matches the working tree; the index records what it could not see. 15 commits, feature-by-feature, each built +
go vet+ golangci-lint +-raceclean.What's in it
Freshness provenance (the spine)
repo_index_staterow written at index end: the git SHA + dirty flag the graph reflects, the Merkle workspace fingerprint, node/edge counts, and the per-language extractor versions. DB and WAL on-disk sizes now surface indaemon_health.Skip-ledger (honest accounting for indexing)
skip_reasonfile nodes instead of vanishing — in both the in-process and subprocess extraction paths, full-index only so a transient mid-edit parse failure never zeroes a file's prior symbols (a regression the full suite caught and is fixed here).index_healthrolls skips up by reason (size/timeout/minified/parse_failed/parse_panic) and reports anodes_per_filedensity plausibility check..gitignoreparsing hardened against non-UTF-8 / oversized lines.Inline freshness rider
read_file/get_symbol_source/get_editing_context/get_file_summary/get_symbol) attach a smallfreshnessblock (stale / indexed-sha / dirty-at-index / worktree-mismatch), omitted entirely when fresh (zero extra tokens). O(1) per-target-file staleness that never false-positives on an untracked path.Reconcile triggers & watch robustness
.gortex/reindex.notify(or$GORTEX_NOTIFY_FILE) to force a sub-second reconcile; a post-checkout git hook uses it.GORTEX_FORCE_FSNOTIFY=1overrides.GORTEX_AUTOINDEX=1), bounded, fully-backgrounded zero-config auto-index of an untracked cwd.Progress, config & incremental soundness
index.includeforce-include config to index a.gitignore-excluded tree.Deliberately deferred (PART, not in this PR)
Simultaneous multi-branch indexing and worktree-overlay indexing — the overlay-per-branch architecture. The substrate exists (branch-keyed snapshot cache, the overlay manager, worktree detection) but a sound implementation is a dedicated design cycle; a hasty version would risk silent graph divergence, the exact failure this work prevents.
Validation
go build ./...+go vet+ golangci-lint green. Each feature-race-tested; the full touched-package-racesuite is the final gate.