feat(#5900): additive segment-aware fbreader MultiReader (streaming-write foundation)#5906
Merged
Merged
Conversation
…t a) Additive-only slice of the bounded-memory streaming-write epic: OpenSegments opens N finalized graph.fb-shaped segment files and presents them as one unified graph view. - EntityCount/RelationshipCount sum across segments. - LookupEntityByID tries each segment's own FlatBuffers `(key)` binary search in turn (relationship endpoints are stable strings per graph.fbs, so cross-segment resolution needs no forward-ref bookkeeping). - IterateEntities/IterateRelationships/FilterEntitiesByKind/ IterateRelationshipsFromID/IterateRelationshipsToID chain the per-segment scans in segment order. - EntityAt/RelationshipAt index into the segment concatenation; Version/LoadGraphMeta/CommunityCount/CommunityAt/LoadAlgoStats delegate to the first segment (header-ownership across segments is a later gen-dir-layout decision). - Close releases every segment's mmap, closing all of them even if one errors, and OpenSegments unwinds already-opened segments on a partial failure. No producer writes segments yet and no gen-dir layout changes; the existing single-file Open/Reader path is untouched. Tests cover sum-across-segments counts, any-segment lookup, explicit cross-segment relationship resolution in both directions, in-order iteration chaining (including early-stop across the whole chain), global EntityAt/ RelationshipAt indexing, 1-segment parity against the existing Reader, and clean multi-segment Close (t.Cleanup registered before TempDir teardown, per the v0.1.9 Windows mapped-file-unlink lesson).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part (a) of the #5890 bounded-memory streaming-write epic. Adds
fbreader.MultiReader(OpenSegments([]string)) presenting N FlatBuffers segment files as one unified graph view — the foundation for segmented graph storage. Closes #5900.Purely additive: two new files (
multireader.go,multireader_test.go); no existing file or caller touched; dead-until-wired by later slices. Single-segment behavior is byte-identical to the currentReader.LookupEntityByIDfans out per-segment(key)binary search (first hit);EntityAt/RelationshipAtindex the concatenation; iterate/filter/rels-from-to chain per-segment scans with cross-boundary early-stop; header fields delegate to segment 0 (multi-segment header ownership deferred to the gen-dir-layout slice).Closereleases every mapping, returns first error, idempotent;OpenSegmentsunwinds on mid-open failure (Windows mmap-lifecycle correctness).Independently reviewed with mutation proofs: single-segment order-parity, cross-segment lookup completeness, concatenation-index boundaries, early-stop propagation, and Close-releases-all each confirmed caught by a test. 272
internal/graphtests pass.🤖 Generated with Claude Code
https://claude.ai/code/session_017quGgaqK7NRoxGT6BTqV2o