lsp: gate the file sweep on real dispatch relevance - #615
Conversation
The definition fallback's rebind arm rewrites an edge's target and tags rebound_from, then counted the correction into edges_confirmed. Split it into EnrichResult/EnrichmentStatus EdgesRebound and surface it beside the existing counters (enrichment logs, index_health sums, zero-yield guard).
On a degraded pass the serial fallback loop can be the only source of progress - every other usefulYield site is unreachable - so the productivity checkpoint read a pass that settles thousands of edges there as zero-yield and cancelled it.
The dispatch half of the demand-gated file sweep admitted any file that declared a type or interface - in C# that is every file. Gate it on the same evidence the per-callable incoming gate uses: dispatch-relevant callables, interfaces, and types with implements/extends adjacency in either direction (edge kinds survive unresolved targets, so an unresolvable base list still qualifies). A bare data type no longer admits its file; the demand half is untouched.
zzet
left a comment
zzet
left a comment
There was a problem hiding this comment.
The tightening is right for C#, but the admission test is circular in the languages where the sweep is the only source of hierarchy edges.
The gate admits a class only when it already carries an extends / implements edge. I drove the real extractors on idiomatic sources to check where that edge exists before enrichment:
csharp,typescript,python,php— extractor emits it (unresolved::X). Premise holds.java,kotlin,ruby— extractor emits nothing for a base list (onlyreferences); the edges arrive from thetstypessupplemental lane at resolve time. Premise holds indirectly.c,cpp,objc,objcpp,swift— nothing, from anywhere. Notstypeslane for them.
class Derived : public Base → 0 hierarchy edges. class Circle: Shape (Swift) → 0. C++ also emits no KindInterface node at all (cpp.go classes are KindType), so neither the unconditional-interface branch nor the resolver's method-set EdgeImplements inference (internal/resolver/inference_scale.go:376) can pick it up.
For clangd and sourcekit-lsp the sweep's typeHierarchy/supertypes hop (provider.go:1603) is the only producer of base-class edges. Under this gate a C++ class-only file drops out of the demand sweep and its hierarchy is never recovered — the gate requires as input exactly what the sweep exists to produce. The Swift line in the description ("admits conformance-declaring files at a similar rate") does not hold: only protocol declarations admit, a conforming class has no edge.
Suggested fix, keeps the measured C# win: fall back to the permissive check when the language's edge set carries zero extends / implements. One pass over repoEdges, self-tuning, no per-language table.
Minor: docs/lsp.md:248 still documents the old gate — the demand row of the sweep-mode table is now wrong.
Rest checks out. Build, lint, and -race over ./internal/semantic/..., ./internal/mcp/..., ./internal/indexer/... are clean here, and both new tests fail on the right assertions when typeIsDispatchRelevant is reverted to the kind check — they are real pins.
The strict type gate is circular in languages whose extractor emits no base-list edges and that have no supplemental type lane (c, cpp, objc, swift): there the sweep's typeHierarchy hop is the only producer of extends / implements edges, so requiring such an edge for admission demands as input exactly what the sweep exists to produce, and a class-only file's hierarchy is never recovered. Probe the language's edge set once per pass: any extends / implements edge a non-LSP lane minted (extractor, tstypes, resolver inference) keeps the strict gate; zero such edges falls the type check back to admitting every type. Sweep-recovered edges (lsp_resolved / lsp_dispatch) are excluded from the evidence - counting the sweep's own output would flip a hierarchy-blind language onto the strict gate one run later and silently drop every class added after that. One pass over the projected repo edges, attributed per language via the source node - no per-language table.
|
Fixed in 80b32a0. The gate now self-tunes per language: one pass over the projected repo edges looks for any extends / implements edge a non-LSP lane produced for that language. Zero such edges means the type check falls back to admitting every type, so c / cpp / objc / swift keep the pre-gate behaviour and the typeHierarchy hop still recovers their bases. One detail beyond the suggestion: the probe ignores lsp_resolved / lsp_dispatch origins. Counting the sweep's own recoveries would flip a hierarchy-blind language onto the strict gate one run later, and a class added after that would carry no edge and never be admitted. Extractor edges with unresolved targets still count (empty origin), so the measured C# behaviour is unchanged. Pinned by three sweep-level tests (zero-evidence fallback, other-language edges do not count, LSP-origin edges are not evidence) plus unit coverage on the probe. The demand row in docs/lsp.md is updated too. |
|
@pbednarcik, there is one more inefficiency Scenario:
It's self-reinforcing: the gate admits hierarchy-involved types, sweeps them, confirms their edges — decaying precisely the evidence it depends on. A fresh reindex re-mints unresolved stubs and restores evidence, so the behaviour oscillates between reindexes. Considering you've made follow-up PRs, I wonder if it's an issue or if it was handled? |
|
It's real, and no — the follow-up PRs didn't touch it (they were MCP-side). Reproduced your exact scenario end-to-end and fixed it on this branch. The mechanism is precisely as you traced it: Fix, in
Tests, red first against the pre-fix code:
Edge meta rides the SQLite projection ( One honest limit: edges confirmed by builds without this commit have already lost their provenance, so on an existing store a language whose entire hierarchy edge set was confirmed can sit on the permissive fallback until its next reindex re-mints the extractor edges. The fix stops the decay from here on; it can't reconstruct provenance that was already erased. |
2 participants
Problem
№2 from the #605 discussion. Under the demand default, the dispatch half of the per-file sweep gate admits a file when it declares any type or interface —
enrichNodeIsDispatchRelevantwas a bare kind check. In C# that is every file, so the gate admitted the whole repo: on my production monorepo, 5,715 of 5,715 symbol-bearing C# files (100%). The per-callable incoming-calls gate is already discriminating (callableIsDispatchRelevant); the file gate never caught up.Change
The file gate now uses the same evidence the per-callable gate does, plus a type-level sibling:
nodeDispatchmap (abstract / overrides / parent-implements callables) intoft.dispatchinstead of the kind check.lspGraphView.typeIsDispatchRelevant: an interface is always in — it is the dispatch surface by definition, and its implementers' AST edges may be exactly what failed to resolve, so the adjacency-less case is the case that needs the sweep most. A class qualifies through hierarchy involvement: animplements/extendsedge in either direction. Edge KINDS survive even when the AST could not resolve the target, so a class with an unresolvable base list still qualifies — recovering those edges is the sweep's whole value for types.enrichNodeIsDispatchRelevantis deleted; the sweep-mode docs now describe the discriminating meaning.Measured on a real graph
Projected onto my production monorepo's settled store (interface-heavy DI conventions — most services implement an interface, entities extend bases):
So the gate trims ~20% of the sweep — worth having on every pass, and worth more on first index where the deadline bites. I'd read ~20% as representative for enterprise C# and Java, where DI-for-testability makes interface-heavy the norm. Other cultures shift the ratio in both directions: Kotlin (final-by-default classes, data classes) should cut deeper, while Swift's protocol-oriented style admits conformance-declaring files at a similar rate for different reasons. A DTO-heavy codebase cuts deeper anywhere. The honest conclusion the ledger + these numbers point at: on an interface-conventioned repo the file gate alone is not a multiple, and the remaining volume lives per-callable (prepare/outgoing run for every callable in an admitted file). Happy to discuss that as a follow-up — outgoing-on-every-callable is deliberate today, it is how plain call edges get recovered.
Caveats kept in view
scaleEnrichTimeouton the node count), so tightening the gate leaves the budget generous rather than shrinking it — the pass gets closer to finishing instead of being cut.Tests
TestEnrichTypeIsDispatchRelevantFromView: interface unconditional; implementer; unresolvable base still qualifies; supertype side; bare data type and callables excluded.TestLSP_Enrich_SweepGate: one pass over three files under the demand default — the POCO file is skipped, the hierarchy file swept, and a type-less file with unresolved demand still swept.Stacked on #613 — the rebound ledger is the measuring instrument for this change's accuracy effect on the next full pass.