fix(groupalgo): in-memory compute-once memo guard for group Pass-4 (bounds persist-failure recompute spin)#5908
Merged
Conversation
…ss-4 The group-scope Pass-4 sweep (Louvain + PageRank + O(V*E) betweenness over the assembled ~32k-node group union) reused results only via the on-disk <group>-algo.json overlay. When that overlay could not be persisted (read-only ~/.grafel/groups, disk-full, EPERM) the disk-skip path could never engage, so RunGroupAlgorithmsIncremental re-ran the full sweep on EVERY trigger, pinning the daemon at sustained high CPU with spikes on a large multi-repo group -- the group-scope analog of the per-repo #50 compute->evict spin. Add a process-local guard keyed on the group-version (community input hash): the heavy pass now runs at most once per version regardless of whether the overlay reached disk. The result is recorded BEFORE the caller's overlay write, so a persist failure cannot reopen the spin. A real re-index bumps the input hash -> exactly one recompute (correctness preserved). Guard access is mutex-protected; the cached result is treated read-only, mirroring the overlay reconstitution contract. Tests (>=2-repo group, exercising the combined union path): - compute-once-per-version across repeated loads with no persist - persist failure (read-only overlay dir) does not recompute forever - structural re-index bumps the version -> one recompute, then reuse
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Recovered from an abandoned prior-session branch (verified genuinely unmerged, not subsumed by the existing disk-overlay guard). Refs #5850.
The failure mode it fixes: main's group Pass-4 recompute-avoidance is entirely disk-overlay-gated (
readOverlayUnconditional/OverlayNeedsRecompute, keyed onInputHash). When the overlay can't persist — read-only~/.grafel/groups, disk-full, EPERM (the "sidecars=0" symptom) — the overlay never lands, so every trigger falls through to a fullgraph.RunAlgorithms(O(V·E) betweenness over the ~32k-node group union). This adds an in-memory memo (memoMu) layered below the disk-overlay skip: it fires only when the disk skip did not, closing that recompute spin. It's the group-scope analog of the already-merged per-repo in-memory guard.Reviewed (independent, mutation-proven):
CommunityInputHash(full node set + weighted edges); stores the literal*AlgorithmResults; deep-equal-verified. A structural change invalidates the key (mutation confirmed).-race -count=20→ 60 pass;memoMusole lock, no lock-order cycle; map bounded to one entry per group (replaced on version change), no leak.Process-local by design → helps the in-process path (
GRAFEL_SUBPROCESS_INDEXER=0); the default subprocess path retains nothing. Bounded per-group retention on that path is the deliberate CPU-spin-vs-RAM tradeoff (possible follow-up: free the memo entry after a successful overlay write).🤖 Generated with Claude Code
https://claude.ai/code/session_017quGgaqK7NRoxGT6BTqV2o