Add Notion datasource sync foundation

[schickling-assistant]

Problem

@overeng/notion-datasource-sync needed production-grade coverage around the live Notion adapter boundary before it could be treated as more than a fail-closed foundation.

Goal

Make datasource-sync prove the important Notion integration paths end-to-end: page-property pagination, canonical query/high-watermark mapping, safe schema updates, NotionMD body push, bounded daemon soak behavior, OTEL trace shape, and an automated visible demo.

Decisions

• Keep unsupported Notion capabilities fail-closed and document those boundaries in VRS docs.
• Implement only the safe schema update subset: add property, rename property, and additive select/multi-select options.
• Reuse local Notion packages for page-property schemas/client support and NotionMD body synchronization instead of duplicating those contracts inside datasource-sync.
• Use a two-tier soak strategy: deterministic daemon soak for repeatability plus credentialed live fixture soak for real Notion lifecycle proof.

Verification

Latest landed milestone commit: d793d9bb.

Delegated final verification from Worker E:

• CI=1 pnpm --dir packages/@overeng/notion-datasource-sync exec tsc -p tsconfig.json --pretty false passed.
• Focused non-live suite passed: 9 files, 207 passed, 7 skipped.
• Credentialed live Notion suite passed: 6 passed, 25 skipped.
• Final live run id: notion-ds-sync-d5ecbe90-fd73-41da-89af-f2806685984c.
• Remote ledger and demo pages were visible and sanitized.
• Worker F staged diff check passed before commit.

Earlier package validation on this branch also covered datasource-sync unit/E2E, notion-effect-client page-property tests, and package TypeScript checks.

Complexity

The added complexity is mostly test harness and adapter boundary code. That is intentional: the live Notion API has pagination, schema mutation, body sync, cleanup, and eventual-consistency behavior that cannot be represented safely by unit tests alone.

Concerns

• Full devenv check:all, oxlint, and stale FOD validation were intentionally skipped per explicit user direction. Do not treat this PR as repo-wide gate complete yet.
• Full live daemon mutation remains deferred; current soak coverage is deterministic daemon soak plus bounded live fixture soak.
• Some Notion API surfaces remain intentionally unsupported/fail-closed where the API is computed, generated, destructive, or not safely patchable.

Follow-ups

• User-owned oxlint/FOD cleanup.
• Optional broader/nightly live daemon soak before declaring datasource-sync production-ready.

References

Draft PR. Auto-merge is not enabled.

Posted on behalf of @schickling

field	value
agent_name	🔌 co1-surge
agent_session_id	f7bf968c-3162-4f22-ac15-8d1b11f2b66b
agent_tool	Codex CLI
agent_tool_version	0.130.0
agent_runtime	Codex CLI 0.130.0
agent_model	unknown
worktree	effect-utils/schickling/2026-05-25-notion-md-db
machine	dev3
tooling_profile	dotfiles@unknown-dirty

[schickling-assistant]

Worker AP pushed 588edce8 fix: harden datasource replica cdc settlement.

Summary:

• Public replica CDC statuses are now settled from planner decisions + durable outbox state after non-dry-run sync.
• Body CDC validates local_body_content against local_body_hash before planning remote writes.
• Safe conflict-resolution CDC actions route through the store-backed conflict command path.
• Public metadata/schema CDC is downgraded to explicit fail-closed until verified post-write hash reconciliation is modeled; command/gateway paths remain separate.

Verification:

• git diff --check
• datasource-sync TypeScript
• vitest run src/replica/replica.e2e.test.ts src/e2e/cli.e2e.test.ts -> 43 passed

Live disposable writes were not run locally because this shell has a Notion token but lacks NOTION_DATASOURCE_SYNC_LIVE=1 and NOTION_DATASOURCE_SYNC_PARENT_PAGE_ID.

Posted on behalf of @schickling

field	value
agent_name	🏖️ co3-beech
agent_session_id	02b4f40d-2292-411a-ac1a-1dcdd7889d3e
agent_tool	Codex CLI
agent_tool_version	0.131.0
agent_runtime	Codex CLI 0.131.0
agent_model	unknown
worktree	effect-utils/schickling/2026-05-25-notion-md-db
machine	dev3
tooling_profile	dotfiles@4db6783

[schickling-assistant]

Worker AS update:

• Added CLI-level fake E2E coverage for public SQLite CDC settlement through sync <workspace>. The test mutates notion.sqlite directly via notion_rows.in_trash, runs CLI sync, verifies a guarded restore command executed, and asserts both notion_row_changes and notion_local_changes settle to applied.
• Tightened row-create fail-closed behavior/docs to state the core safety invariant: Notion create-page has no idempotency key, so execution remains blocked until durable returned page_id reconciliation makes retry duplicate-safe.
• Left metadata/schema public CDC, database metadata, files/upload staging, Notion views, and destructive schema workflows explicitly fail-closed.

Verification:

• git diff --check
• CI=1 pnpm --dir packages/@overeng/notion-datasource-sync exec tsc -p tsconfig.json --pretty false
• CI=1 pnpm --dir packages/@overeng/notion-datasource-sync exec vitest run src/replica/replica.e2e.test.ts src/e2e/cli.e2e.test.ts --config vitest.config.ts -> 44 passed

Posted on behalf of @schickling

field	value
agent_name	🏖️ co3-beech
agent_session_id	02b4f40d-2292-411a-ac1a-1dcdd7889d3e
agent_tool	Codex CLI
agent_tool_version	0.131.0
agent_runtime	Codex CLI 0.131.0
agent_model	unknown
worktree	effect-utils/schickling/2026-05-25-notion-md-db
machine	dev3
tooling_profile	dotfiles@4db6783

[schickling-assistant]

Worker AV row-create milestone is implemented and locally/live verified in commit 7bb5cabc.

Highlights:

• Added explicit notion_row_creates public SQLite create surface while keeping notion_rows confirmed remote-observed only.
• Added guarded CreatePageCommand planning/execution, live/fake gateways, durable remote_page_id settlement, and needs_reconciliation handling when a create settlement lacks a page id.
• Added notion_rows_effective / notion_cells_effective for pending local creates.
• Updated README, CLI docs, capability/safety/testing docs, VRS, and changelog.

Verification completed:

• git diff --check
• datasource-sync TypeScript
• focused replica/CLI E2E
• nearby gateway/core/store/planner/executor unit tests
• OTEL E2E/unit tests
• live Notion scratch-parent E2E: 33 passed / 1 skipped; ledger packages/@overeng/notion-datasource-sync/tmp/notion-datasource-sync-live/worker-av-20260527T194515.json; cleanup reached verified-cleaned.

Epic update: #698 (comment)

Posted on behalf of @schickling

field	value
agent_name	🏖️ co3-beech
agent_session_id	02b4f40d-2292-411a-ac1a-1dcdd7889d3e
agent_tool	Codex CLI
agent_tool_version	0.131.0
agent_runtime	Codex CLI 0.131.0
agent_model	unknown
worktree	effect-utils/schickling/2026-05-25-notion-md-db
machine	dev3
tooling_profile	dotfiles@4db6783

[schickling-assistant]

Worker AY update: promoted public SQLite data-source metadata CDC for title/description.

Highlights:

• notion_data_sources now projects metadata_json, title_plain_text, description_plain_text, and metadata_hash.
• notion_metadata_changes with resource_type=data_source now plans guarded PatchDataSourceMetadataCommand intents, computes expected post-write metadata hashes, and settles from outbox/read-after-write state.
• Live authority is documented: the adapter patches owning database title/description where needed, then verifies by retrieving the data source and matching the canonical metadata hash.
• Separate database metadata authority, files/uploads, Notion views, relation/people writes, and schema migrations remain fail-closed.

Verification:

• git diff --check
• datasource-sync TypeScript
• focused replica/CLI E2E: 48 passed
• daemon + OTel tests: 23 passed
• gateway/planner/executor/core unit tests: 120 passed
• full disposable live Notion E2E with local-only ledger: 33 passed, 1 skipped; cleanup failures: 0

Posted on behalf of @schickling

field	value
agent_name	🏖️ co3-beech
agent_session_id	02b4f40d-2292-411a-ac1a-1dcdd7889d3e
agent_tool	Codex CLI
agent_tool_version	0.131.0
agent_runtime	Codex CLI 0.131.0
agent_model	unknown
worktree	effect-utils/schickling/2026-05-25-notion-md-db
machine	dev3
tooling_profile	dotfiles@4db6783

[schickling-assistant]

Worker BA milestone: promoted database metadata CDC for the public SQLite replica.\n\nSummary:\n- Added notion_databases as a separate database/container metadata projection.\n- Added executable notion_metadata_changes(resource_type='database') for title/description edits with database_id, base-hash guards, expected post-write metadata hash settlement, and fake/CLI/live coverage.\n- Database icon/cover/parent/trash/lock remain fail-closed. Files/uploads, Notion view CDC, relation/people writes, destructive schema migrations, and unsupported conflict-resolution actions remain explicit follow-ups.\n\nVerification:\n- git diff --check\n- datasource-sync TypeScript\n- focused replica/CLI E2E: 48 passed\n- gateway/planner/executor/core tests: 120 passed\n- daemon fake + observability tests: 22 passed\n- full disposable live Notion E2E: 33 passed, 1 skipped; cleanup failures: 0\n\nLive ledger: packages/@overeng/notion-datasource-sync/tmp/notion-datasource-sync-live/worker-ba-final-20260527T212544Z.json

Posted on behalf of @schickling

field	value
agent_name	🏖️ co3-beech
agent_session_id	02b4f40d-2292-411a-ac1a-1dcdd7889d3e
agent_tool	Codex CLI
agent_tool_version	0.131.0
agent_runtime	Codex CLI 0.131.0
agent_model	unknown
worktree	effect-utils/schickling/2026-05-25-notion-md-db
machine	dev3
tooling_profile	dotfiles@4db6783

[schickling-assistant]

Worker BC update: pushed fccc9858 with read-only Notion UI view inventory projection in notion_views. This keeps Notion UI views separate from generated local SQLite views, does not use view query results as row-membership authority, and keeps view write CDC fail-closed. Verified with TypeScript, git diff --check, focused replica/CLI/gateway/core/OTel tests, and full disposable live Notion E2E using the scratch parent with local ledger only.

Posted on behalf of @schickling

field	value
agent_name	🏖️ co3-beech
agent_session_id	02b4f40d-2292-411a-ac1a-1dcdd7889d3e
agent_tool	Codex CLI
agent_tool_version	0.131.0
agent_runtime	Codex CLI 0.131.0
agent_model	unknown
worktree	effect-utils/schickling/2026-05-25-notion-md-db
machine	dev3
tooling_profile	dotfiles@4db6783

[schickling-assistant]

Worker BD update: promoted guarded public SQLite relation removals/reorders from fully paginated relation bases. Relation writes now require complete base values, preserve all non-target relation entries through replacement-shaped writes, enforce the 100 item Notion cap, and keep relation additions fail-closed until target accessibility is modeled. Verified with fake E2E, focused replica/CLI/gateway/executor coverage, and a full disposable live Notion suite using a local ledger only.

Posted on behalf of @schickling

field	value
agent_name	🏖️ co3-beech
agent_session_id	02b4f40d-2292-411a-ac1a-1dcdd7889d3e
agent_tool	Codex CLI
agent_tool_version	0.131.0
agent_runtime	Codex CLI 0.131.0
agent_model	unknown
worktree	effect-utils/schickling/2026-05-25-notion-md-db
machine	dev3
tooling_profile	dotfiles@4db6783

[schickling-assistant]

Worker BE update: pushed 5b743c64 fix: guard complex datasource replica cell edits. This pass keeps the remaining high-risk non-schema surfaces fail-closed unless already promoted: direct public SQLite people/files current-state edits now abort before visible replica mutation or CDC append; external URL file attachments still use explicit staging; local uploads, people writes, relation additions, view write CDC, and broader conflict actions remain documented fail-closed until their identity/reconciliation models are proven. Verification: git diff --check, datasource-sync TypeScript, focused replica/CLI E2E (54 passed), OTel tests (7 passed), full disposable live Notion E2E against scratch parent (33 passed / 1 skipped) with local ledger tmp/notion-datasource-sync-live/worker-be-20260527T232651Z.json and 0 cleanup failures.

Posted on behalf of @schickling

field	value
agent_name	🏖️ co3-beech
agent_session_id	02b4f40d-2292-411a-ac1a-1dcdd7889d3e
agent_tool	Codex CLI
agent_tool_version	0.131.0
agent_runtime	Codex CLI 0.131.0
agent_model	unknown
worktree	effect-utils/schickling/2026-05-25-notion-md-db
machine	dev3
tooling_profile	dotfiles@4db6783

[schickling-assistant]

Worker BF update pushed in 8816357b: guarded public SQLite relation additions are now supported when the relation base is fully paginated and each added target is already observed in notion_relation_targets for the same data source/property. Unobserved targets still fail closed; local uploads, people writes, view write CDC, broader conflict actions, and destructive schema workflows remain fail-closed. Verification: diff check, TS, focused replica/CLI E2E 55 passed, gateway/planner/executor/core 120 passed, daemon/OTel 23 passed, full disposable live Notion E2E 33 passed / 1 skipped; local ledger had 20 entries and 0 cleanup failures.

Posted on behalf of @schickling

field	value
agent_name	🏖️ co3-beech
agent_session_id	02b4f40d-2292-411a-ac1a-1dcdd7889d3e
agent_tool	Codex CLI
agent_tool_version	0.131.0
agent_runtime	Codex CLI 0.131.0
agent_model	unknown
worktree	effect-utils/schickling/2026-05-25-notion-md-db
machine	dev3
tooling_profile	dotfiles@4db6783

[schickling-assistant]

Worker BH pushed c918c70 (fix: model remaining datasource replica guard surfaces). Remaining high-risk product surfaces stay fail-closed unless their identity/reconciliation model is proven. Changes: added local-upload lifecycle fields to notion_file_assets; added typed fail-closed notion_view_changes CDC with compatibility projection; added E2E coverage for local upload and view-write requests staying unsupported/no-intent; corrected capabilities docs so notion_views read-only inventory is supported while view writes/query membership are not, and schema writes refer to the dedicated command path rather than public SQLite schema CDC. Verification: git diff --check; datasource-sync TypeScript; focused replica/CLI E2E 55 passed; daemon/OTel 17 passed; full disposable live Notion E2E 33 passed / 1 skipped using the Effect Utils scratch parent with local-only ledger worker-bh-20260528T000014Z.json and 0 cleanup failures. Did not stage unrelated genie/ci-workflow.ts drift or .claude/.

Posted on behalf of @schickling

field	value
agent_name	🏖️ co3-beech
agent_session_id	02b4f40d-2292-411a-ac1a-1dcdd7889d3e
agent_tool	Codex CLI
agent_tool_version	0.131.0
agent_runtime	Codex CLI 0.131.0
agent_model	unknown
worktree	effect-utils/schickling/2026-05-25-notion-md-db
machine	dev3
tooling_profile	dotfiles@4db6783