feat(#5901): gen-dir read substrate — manifest, segment descriptor, key-range routing (dark)#5913
Merged
Merged
Conversation
…5901 dark slice 1/5) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017quGgaqK7NRoxGT6BTqV2o
…g on MultiReader (#5901 dark slice 2/5) - GraphView: the common read surface satisfied by both *Reader (single-file) and *MultiReader (segment-set), so segment-aware call sites hold "a graph" without caring how it is backed. - KeyRange + OpenSegmentsWithRanges: LookupEntityByID skips segments whose manifest [Min,Max] window cannot hold the key (decision 4), and pure- relationship segments (HasEntities=false). SegmentSearchHook is the test seam proving out-of-range segments are pruned. OpenSegments == no routing (legacy). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017quGgaqK7NRoxGT6BTqV2o
…s segment-set vs absent (#5901 dark slice 3/5) - GraphDescriptor{Kind,Dir,Path,GenDir,Manifest,Segments} + CurrentGraphDescriptor(dir). Resolves `current` naming graph.<gen>.fb (single-file), graph.<gen>/ dir or graph.<gen>/manifest.json (segment-set → reads+validates manifest), else legacy flat graph.fb / absent. Corrupt segment-set manifest surfaces an error (loud); missing gen file/dir + hostile pointer fall back to flat (never an error). - CurrentGraphPath is UNCHANGED — the ~11 #5891 single-file/legacy callers keep working; descriptor is the additive segment-aware entry point. Parity asserted. - Format primitives for the future writer + fixtures: GenDirName, SegmentFileName (zero-padded seg-NNNN.fb), WriteCurrentPointerRaw (accepts gen-file OR gen-dir/ manifest pointer). No producer calls them yet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017quGgaqK7NRoxGT6BTqV2o
…oad (#5901 dark slice 4a/5) - reader_open.go: ReaderForDir(dir) → GraphView. single-file/legacy/absent → fbreader.Open (byte-identical common path); segment-set → OpenSegmentsWithRanges threading manifest [Min,Max] key routing. Path-based OpenSegmentReader(genDir) for callers holding the resolved gen dir. - load.go: LoadGraphFromDir + LoadGraphHeaderOnlyFromDir route segment-sets through a shared materializeGraphView core (extracted from loadFBDoc) that runs identically over *Reader and *MultiReader — single-file path unchanged. The fbversion gate stays inside the core (no fbversion bump). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017quGgaqK7NRoxGT6BTqV2o
…te (#5901 dark slice 4b/5) - daemon/mcp graph_cache: Entry.Reader + Get/GetForRepoRef now hold fbreader.GraphView. openReader detects a gen DIR and opens a MultiReader via graph.OpenSegmentReader (openSegmentSetAndGate), composing the SAME #5907 crash-recover + fbversion gate as the single-file openAndGate. GetForRepoRef keys segment-sets on the gen dir. Single-file/.fb path byte-identical. - daemon/state_path findGraphFileInDir: resolves a segment-set to its gen dir + manifest.json mtime, so the internal/mcp reload loop discovers it and serves the segment-aware Document (readDocumentFromDir → LoadGraphFromDir). - internal/mcp state.go:1395: documented the segment-set boundary — a gen dir has no .fb ext so the zero-copy mmap cutover declines it (newRdr nil) and the repo serves from the materialized Document. The mmap-cutover *MultiReader is a later serve slice. NO fbversion bump; NO producer changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017quGgaqK7NRoxGT6BTqV2o
…ice 5/5) GCStaleGens now sweeps BOTH single-file gens (graph.<gen>.fb, unlink) and multi-segment gen dirs (graph.<gen>/, os.RemoveAll — every seg-NNNN.fb + the manifest). The keep-window (current + immediately-previous) is computed over the union of gen files and gen dirs, so a mixed-history state dir still retains exactly the two newest generations. Still strictly best-effort / fail-soft: a still-mapped segment on Windows fails to delete, is swallowed, and swept on the next write — GC never fails the caller. parseGenDir added (segment-set counterpart to parseGen). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017quGgaqK7NRoxGT6BTqV2o
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part (b) of #5890. The read-side substrate for the segmented graph format, landing DARK (no fbversion bump, no writer, no enrichment). Single-file/legacy graphs stay byte-identical. Closes #5901.
internal/graph/manifest.go):Manifest{FormatVersion, Segments[]SegmentMeta{File,Kind,EntityCount,RelCount,MinKey,MaxKey}},ManifestFormatVersion=1(independent of fbversion). Validated read (known version, ≥1 segment, bareseg-NNNN.fbonly — path-traversal hardened) + atomic validated write. Entity/relationship streams listable independently.CurrentGraphPathUNCHANGED (single-file only → the ~11 bug(win): resident mmap reader blocks reindex rewrite of graph.fb on Windows (file-locking) #5891 callers untouched); new additiveCurrentGraphDescriptor(dir)classifies SINGLE-FILE | SEGMENT-SET | ABSENT.currentmay name a gen dir/manifest. Corrupt segment-set manifest → loud error; missing gen / hostile pointer → safe flat fallback.fbreader.GraphViewunifies*Reader+*MultiReader;OpenSegmentsWithRangesthreads per-segment min/max keys soLookupEntityByIDskips segments that can't hold the key (inclusive bounds; empty range never skips = safe fallback).materializeGraphViewcore run by both single-fileloadFBDoc(byte-identical) andloadSegmentSetDocument; fbversion gate inside the core.openReaderdetects a gen dir → MultiReader through the #5890(g): graceful update/install/reinstall migration for the breaking on-disk format (force-reindex OK) #5907 crash+version gate;findGraphFileInDirresolves segment-sets; state.go's.fb-ext guard declines a gen dir cleanly (Doc-backed serve). Zero-copy mmap serve for segments is deferred to #5890(h): zero-copy mmap serve for segment-sets (MultiReader cutover) — required before segmented writes ship by default #5912 (required before the writer ships by default — else large graphs regress read-RSS).GCStaleGensrm -rfs stale gen dirs + unlinks flat gens, keep-window (current+previous) over the union, fail-soft (Windows).Independently reviewed, mutation-proven: single-file parity, routing-skip correctness, manifest validation, GC keep-window, serve routing composition. 2346 tests pass.
🤖 Generated with Claude Code
https://claude.ai/code/session_017quGgaqK7NRoxGT6BTqV2o