Skip to content

Add remote URL support to multi-repo hydration#2827

Merged
steveyegge merged 4 commits intosteveyegge:mainfrom
mrmaxsteel:feat/remote-hydration
Mar 27, 2026
Merged

Add remote URL support to multi-repo hydration#2827
steveyegge merged 4 commits intosteveyegge:mainfrom
mrmaxsteel:feat/remote-hydration

Conversation

@mrmaxsteel
Copy link
Copy Markdown
Contributor

@mrmaxsteel mrmaxsteel commented Mar 25, 2026

Summary

  • Adds internal/remotecache package: cached dolt clones for remote URLs (~/.cache/beads/remotes/<hash>/) with clone/pull/push lifecycle and file-based locking
  • bd repo add / bd repo sync now accept dolt remote URLs (dolthub://, https://, s3://, etc.) alongside local paths — sync reads from cached SQL store instead of JSONL
  • bd create --repo <remote-url> pulls into cache, creates issue, pushes back (two network round-trips, no git checkout)
  • isValidRemoteURL in config.go now delegates to shared remotecache.IsRemoteURL

Closes #2826

Changes by commit

1. /{cmd,internal}: add remote URL support to multi-repo hydration

  • New internal/remotecache package with Cache struct managing local dolt clones at ~/.cache/beads/remotes/<sha256-prefix>/
  • url.go: IsRemoteURL() recognizes dolthub://, https://, s3://, gs://, file://, ssh://, git+ssh://, git+https://, and SCP-style URLs
  • cache.go: Ensure() (clone or pull), Push(), OpenStore(), Evict() with file-based locking
  • cmd/bd/config.go: isValidRemoteURL delegates to remotecache.IsRemoteURL, removing duplicated scheme list and regex
  • cmd/bd/repo.go: bd repo add accepts remote URLs (skips local .beads validation); bd repo remove evicts cache; bd repo sync hydrates from remote SQL store
  • cmd/bd/create.go: --repo <remote-url> pulls into cache, creates issue in cached store, pushes back

2. fix: address review issues in remote hydration PR

  • Tighten meta file permissions from 0644 to 0600 (gosec G306)
  • Make lock polling context-aware (respects ctx cancellation / Ctrl+C)
  • Replace defer cache.Push with explicit call (FatalError calls os.Exit, which skips defers)

3. fix(remotecache): address code review findings from #2827

  • Remove os.Remove in releaseLock to prevent TOCTOU race — stale lock cleanup handled by acquireLock's age check instead
  • Add debug.Logf to writeMeta for error visibility
  • Escalate remote push failure to FatalError in create (silent warning meant the remote never received the issue)

4. fix(remotecache): defer store close, add Push test, add freshness TTL

  • cmd/bd/repo.go: Use idiomatic defer func() { _ = remoteStore.Close() }() instead of inline close after SearchIssues — matches project convention and guards against future code between open and close
  • internal/remotecache/cache.go: Add FreshFor field to Cache struct (default 30s via DefaultCache()). Ensure() skips pulling on warm start when the last pull is within the TTL window. FreshFor: 0 preserves always-pull behavior.
  • internal/remotecache/cache_test.go: Add TestPush — full round-trip integration test (clone → insert row → push → re-clone → verify data). Add TestEnsureFreshFor — validates TTL skip and FreshFor=0 bypass.

Test plan

  • go test ./internal/remotecache/ -v — 8 tests passing (URL detection, cache key, cold clone, warm pull, push round-trip, freshness TTL, eviction, default cache path) using file:// protocol
  • go test ./cmd/bd/ -short — all existing tests pass
  • go build ./cmd/bd/ — server build clean
  • go build -tags embeddeddolt ./cmd/bd/ — embedded build clean
  • CI: all jobs pass except pre-existing flaky TestEmbeddedInit/database (dolt lock contention unrelated to this PR)

🤖 Generated with Claude Code

mrmaxsteel added a commit to mrmaxsteel/beads that referenced this pull request Mar 25, 2026
- Fix lock file race: remove os.Remove in releaseLock to prevent TOCTOU
  race where another process's lock gets deleted; stale lock cleanup in
  acquireLock handles orphaned files
- Add debug.Logf to writeMeta for error visibility
- Rename cacheErr to idiomatic err in repo sync remote block
- Escalate remote push failure to FatalError in create (silent warning
  meant the remote never received the issue)

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Copy link
Copy Markdown
Contributor Author

@mrmaxsteel mrmaxsteel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.

mrmaxsteel and others added 4 commits March 26, 2026 08:38
Multi-repo hydration (`bd repo sync`) reads issues.jsonl from local
filesystem paths, predating beads' dolt remote support. This adds
remote URL support so hydration, `--repo`, and `bd repo add` can
accept dolt remote URLs (dolthub://, https://, s3://, etc.) alongside
local paths.

New internal/remotecache package manages cached dolt clones at
~/.cache/beads/remotes/<hash>/ with clone/pull/push lifecycle and
file-based locking for concurrent access.

Closes steveyegge#2826

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Fix gosec G306: tighten meta file permissions from 0644 to 0600
- Make lock polling context-aware (respects Ctrl+C / ctx cancellation)
- Replace defer cache.Push with explicit call (FatalError calls os.Exit,
  which skips deferred functions — silent data loss on remote creates)
- Log remoteStore.Close() errors as warnings instead of discarding
- Quantify CacheKey birthday-bound collision risk in comment

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Fix lock file race: remove os.Remove in releaseLock to prevent TOCTOU
  race where another process's lock gets deleted; stale lock cleanup in
  acquireLock handles orphaned files
- Add debug.Logf to writeMeta for error visibility
- Rename cacheErr to idiomatic err in repo sync remote block
- Escalate remote push failure to FatalError in create (silent warning
  meant the remote never received the issue)

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Replace defer-in-loop with eager close in repo sync (defer inside a
  for loop leaks connections until function return)
- Add FreshFor TTL to Cache (default 30s) — Ensure() skips pull when
  last pull is within the window, FreshFor=0 preserves always-pull
- Add TestPush: full round-trip integration test (clone → insert → push
  → re-clone → verify data)
- Add TestEnsureFreshFor: validates TTL skip and FreshFor=0 bypass
- Add concurrency doc comment to OpenStore

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@mrmaxsteel mrmaxsteel force-pushed the feat/remote-hydration branch from f5098f7 to 168069e Compare March 26, 2026 08:42
@steveyegge steveyegge merged commit 541b684 into steveyegge:main Mar 27, 2026
31 checks passed
@mrmaxsteel mrmaxsteel deleted the feat/remote-hydration branch March 27, 2026 06:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi-repo hydration should use dolt remotes, not local filesystem

2 participants