Add remote URL support to multi-repo hydration#2827
Merged
steveyegge merged 4 commits intosteveyegge:mainfrom Mar 27, 2026
Merged
Add remote URL support to multi-repo hydration#2827steveyegge merged 4 commits intosteveyegge:mainfrom
steveyegge merged 4 commits intosteveyegge:mainfrom
Conversation
mrmaxsteel
added a commit
to mrmaxsteel/beads
that referenced
this pull request
Mar 25, 2026
- Fix lock file race: remove os.Remove in releaseLock to prevent TOCTOU race where another process's lock gets deleted; stale lock cleanup in acquireLock handles orphaned files - Add debug.Logf to writeMeta for error visibility - Rename cacheErr to idiomatic err in repo sync remote block - Escalate remote push failure to FatalError in create (silent warning meant the remote never received the issue) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Multi-repo hydration (`bd repo sync`) reads issues.jsonl from local filesystem paths, predating beads' dolt remote support. This adds remote URL support so hydration, `--repo`, and `bd repo add` can accept dolt remote URLs (dolthub://, https://, s3://, etc.) alongside local paths. New internal/remotecache package manages cached dolt clones at ~/.cache/beads/remotes/<hash>/ with clone/pull/push lifecycle and file-based locking for concurrent access. Closes steveyegge#2826 Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Fix gosec G306: tighten meta file permissions from 0644 to 0600 - Make lock polling context-aware (respects Ctrl+C / ctx cancellation) - Replace defer cache.Push with explicit call (FatalError calls os.Exit, which skips deferred functions — silent data loss on remote creates) - Log remoteStore.Close() errors as warnings instead of discarding - Quantify CacheKey birthday-bound collision risk in comment Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Fix lock file race: remove os.Remove in releaseLock to prevent TOCTOU race where another process's lock gets deleted; stale lock cleanup in acquireLock handles orphaned files - Add debug.Logf to writeMeta for error visibility - Rename cacheErr to idiomatic err in repo sync remote block - Escalate remote push failure to FatalError in create (silent warning meant the remote never received the issue) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Replace defer-in-loop with eager close in repo sync (defer inside a for loop leaks connections until function return) - Add FreshFor TTL to Cache (default 30s) — Ensure() skips pull when last pull is within the window, FreshFor=0 preserves always-pull - Add TestPush: full round-trip integration test (clone → insert → push → re-clone → verify data) - Add TestEnsureFreshFor: validates TTL skip and FreshFor=0 bypass - Add concurrency doc comment to OpenStore Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
f5098f7 to
168069e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
internal/remotecachepackage: cached dolt clones for remote URLs (~/.cache/beads/remotes/<hash>/) with clone/pull/push lifecycle and file-based lockingbd repo add/bd repo syncnow accept dolt remote URLs (dolthub://, https://, s3://, etc.) alongside local paths — sync reads from cached SQL store instead of JSONLbd create --repo <remote-url>pulls into cache, creates issue, pushes back (two network round-trips, no git checkout)isValidRemoteURLin config.go now delegates to sharedremotecache.IsRemoteURLCloses #2826
Changes by commit
1.
/{cmd,internal}: add remote URL support to multi-repo hydrationinternal/remotecachepackage withCachestruct managing local dolt clones at~/.cache/beads/remotes/<sha256-prefix>/url.go:IsRemoteURL()recognizes dolthub://, https://, s3://, gs://, file://, ssh://, git+ssh://, git+https://, and SCP-style URLscache.go:Ensure()(clone or pull),Push(),OpenStore(),Evict()with file-based lockingcmd/bd/config.go:isValidRemoteURLdelegates toremotecache.IsRemoteURL, removing duplicated scheme list and regexcmd/bd/repo.go:bd repo addaccepts remote URLs (skips local.beadsvalidation);bd repo removeevicts cache;bd repo synchydrates from remote SQL storecmd/bd/create.go:--repo <remote-url>pulls into cache, creates issue in cached store, pushes back2.
fix: address review issues in remote hydration PRdefer cache.Pushwith explicit call (FatalErrorcallsos.Exit, which skips defers)3.
fix(remotecache): address code review findings from #2827os.RemoveinreleaseLockto prevent TOCTOU race — stale lock cleanup handled byacquireLock's age check insteaddebug.LogftowriteMetafor error visibilityFatalErrorin create (silent warning meant the remote never received the issue)4.
fix(remotecache): defer store close, add Push test, add freshness TTLcmd/bd/repo.go: Use idiomaticdefer func() { _ = remoteStore.Close() }()instead of inline close afterSearchIssues— matches project convention and guards against future code between open and closeinternal/remotecache/cache.go: AddFreshForfield toCachestruct (default 30s viaDefaultCache()).Ensure()skips pulling on warm start when the last pull is within the TTL window.FreshFor: 0preserves always-pull behavior.internal/remotecache/cache_test.go: AddTestPush— full round-trip integration test (clone → insert row → push → re-clone → verify data). AddTestEnsureFreshFor— validates TTL skip andFreshFor=0bypass.Test plan
go test ./internal/remotecache/ -v— 8 tests passing (URL detection, cache key, cold clone, warm pull, push round-trip, freshness TTL, eviction, default cache path) usingfile://protocolgo test ./cmd/bd/ -short— all existing tests passgo build ./cmd/bd/— server build cleango build -tags embeddeddolt ./cmd/bd/— embedded build cleanTestEmbeddedInit/database(dolt lock contention unrelated to this PR)🤖 Generated with Claude Code