feat(sync): add incremental Obsidian DataPlug - #92
Open
vcvyg wants to merge 4 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
Adds first-class incremental sync support for Obsidian vaults by introducing an ObsidianVaultPlug with a persisted cursor and extending the generic plug materialization path to support stable upserts, typed links, and soft deletes. This routes Obsidian vaults through the existing contextseek sync command and validates behavior with an integration test + fixture vault.
Changes:
- Add
ObsidianVaultPlugthat tracks per-file state (mtime/size/hash) and emits add/update/deleteRawEvents with stableitem_ids and wikilink-derivedLinkType.related_toedges. - Extend
RawEventandContextSeek.plug()to support incremental operations (upsert + soft delete) and attach typed links. - Update CLI/sync reporting to include added/updated/deleted/skipped counts and add integration coverage using a fixture vault.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/integration_tests/test_obsidian_sync.py | Integration tests covering incremental add/update/delete/skip behavior, stable IDs, and wikilink materialization. |
| tests/fixtures/obsidian_vault/Projects/Alpha.md | Fixture note used to validate incremental updates/deletes. |
| tests/fixtures/obsidian_vault/Home.md | Fixture note containing a wikilink to validate link materialization. |
| tests/fixtures/obsidian_vault/.obsidian/app.json | Minimal marker to ensure vault format detection via .obsidian/. |
| src/contextseek/plugs/obsidian/plug.py | New incremental Obsidian vault DataPlug with cursor persistence, hashing, and wikilink extraction. |
| src/contextseek/plugs/obsidian/init.py | Exports Obsidian plug types for import/use via contextseek.plugs. |
| src/contextseek/plugs/core/protocols.py | Extends RawEvent with operation, item_id, and links for incremental materialization. |
| src/contextseek/plugs/init.py | Adds lazy export of ObsidianVaultPlug in the plugs package. |
| src/contextseek/daemon/sync_cmd.py | Detects Obsidian vaults and routes them through ObsidianVaultPlug, populating full SyncReport counts. |
| src/contextseek/client/contextseek.py | Implements incremental upsert/delete handling in plug() via _upsert_raw_event. |
| src/contextseek/cli/main.py | Updates sync output formatting to report added/updated/deleted/skipped for all formats. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+80
to
+104
| for rel, path in markdown_files.items(): | ||
| stat = path.stat() | ||
| old = old_files.get(rel) | ||
| if ( | ||
| old is not None | ||
| and int(old.get("mtime_ns", -1)) == stat.st_mtime_ns | ||
| and int(old.get("size", -1)) == stat.st_size | ||
| ): | ||
| next_files[rel] = dict(old) | ||
| skipped += 1 | ||
| continue | ||
|
|
||
| raw = path.read_text(encoding="utf-8", errors="replace") | ||
| digest = hashlib.sha256(raw.encode("utf-8")).hexdigest() | ||
| record = { | ||
| "mtime_ns": stat.st_mtime_ns, | ||
| "size": stat.st_size, | ||
| "content_hash": digest, | ||
| "item_id": item_ids[rel], | ||
| } | ||
| next_files[rel] = record | ||
| if old is not None and old.get("content_hash") == digest: | ||
| skipped += 1 | ||
| continue | ||
|
|
| @@ -66,6 +68,15 @@ class RawEvent: | |||
| metadata: dict = field(default_factory=dict) | |||
| """Extra key-value pairs the plug can supply (passed to provenance context).""" | |||
Comment on lines
+227
to
+236
| def _markdown_files(vault: Path) -> dict[str, Path]: | ||
| files: dict[str, Path] = {} | ||
| for path in vault.rglob("*"): | ||
| if not path.is_file() or path.suffix.casefold() != ".md": | ||
| continue | ||
| rel_path = path.relative_to(vault) | ||
| if any(part.startswith(".") for part in rel_path.parts[:-1]): | ||
| continue | ||
| files[rel_path.as_posix()] = path | ||
| return dict(sorted(files.items())) |
Comment on lines
+76
to
+77
| events: list[RawEvent] = [] | ||
| next_files: dict[str, dict[str, Any]] = {} |
Contributor
Author
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/contextseek/client/contextseek.py:573
- In
_upsert_raw_event, update events rebuild a freshContextItemand only copy a few lifecycle fields fromexisting. This currently resetsimportanceback to its default (1.0) on every update unless the plug explicitly provides animportancehint later, which can unintentionally discard user- or pipeline-tuned importance values.
item.created_at = existing.created_at
item.updated_at = datetime.now(timezone.utc)
item.relevance_boost = existing.relevance_boost
item.access_count = existing.access_count
item.lineage_access_count = existing.lineage_access_count
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.
Suppressed comments (2)
src/contextseek/plugs/obsidian/plug.py:114
- The cursor
content_hashis computed from the raw file text, but the materializedRawEvent.contentuses_indexable_markdown(raw)(which strips frontmatter and rewrites wikilinks). This can trigger unnecessaryupdatewrites when only non-indexed parts change (e.g., YAML frontmatter), even though the stored content would be identical. Compute the digest from the same indexable content you materialize.
raw = path.read_text(encoding="utf-8", errors="replace")
digest = hashlib.sha256(raw.encode("utf-8")).hexdigest()
record = {
src/contextseek/daemon/sync_cmd.py:544
on_progressis passed through toObsidianVaultPlug, but that plug reports the first argument as "changed/processed" (adds+updates+deletes), while the existingSyncProgress.update()UI treats the first argument as strictly "added" (it printsadded={added}and computesdone = added + skipped). This makes progress output misleading for Obsidian syncs. Consider updating the progress callback contract (andSyncProgress.update) to accept/labelprocessedand optionally surface updated/deleted counts.
plug = ObsidianVaultPlug(
p,
sync_key=scope,
persist_state=not dry_run,
on_progress=on_progress,
Comment on lines
+566
to
+574
| if existing is not None: | ||
| action = "plug_update" | ||
| item.created_at = existing.created_at | ||
| item.updated_at = datetime.now(timezone.utc) | ||
| item.importance = existing.importance | ||
| item.relevance_boost = existing.relevance_boost | ||
| item.access_count = existing.access_count | ||
| item.lineage_access_count = existing.lineage_access_count | ||
| item.last_accessed_at = existing.last_accessed_at |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ObsidianVaultPlugthat persists per-scope file mtime, size, content hash, and stable item identityRawEventmaterialization with stable upserts, typed links, and soft deletescontextseek synccommand and report add/update/delete/skip counts[[wikilinks]]toLinkType.related_toedgesWhy
The generic sync path deduplicated content, but it could not update a changed source, remove a vanished source, or preserve Obsidian relationships. Stable per-file identities and a committed-after-success cursor make repeated vault syncs incremental and recoverable.
Checks
uv run ruff format --check src testsuv run ruff check src testsuv run pytest tests/ -q --ignore=tests/unit_tests/test_powermem_plug.py(445 passed, 8 skippedon Windows)The excluded PowerMem test file has existing Windows-specific executable/path assumptions (
.EXE, backslashes); all observed full-suite failures were isolated to that file.Closes #48