From 39e3b0f2c82a8aaf897e9fd694997f17dd8a4530 Mon Sep 17 00:00:00 2001 From: Nene7ko_ <1604009816@qq.com> Date: Fri, 28 Aug 2026 15:18:09 +0800 Subject: [PATCH 1/5] docs: define source snapshot integration shape --- docs/en/rfcs/1240_source_integration_shape.md | 223 ++++++++++++++++++ docs/zh/rfcs/1240_source_integration_shape.md | 201 ++++++++++++++++ 2 files changed, 424 insertions(+) create mode 100644 docs/en/rfcs/1240_source_integration_shape.md create mode 100644 docs/zh/rfcs/1240_source_integration_shape.md diff --git a/docs/en/rfcs/1240_source_integration_shape.md b/docs/en/rfcs/1240_source_integration_shape.md new file mode 100644 index 000000000..83c7b0bdb --- /dev/null +++ b/docs/en/rfcs/1240_source_integration_shape.md @@ -0,0 +1,223 @@ +# Source Integration Shape + +- Proposal Name: source_integration_shape +- Status: Proposed +- Start Date: 2026-08-25 +- Tracking Issue: [oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240) +- Related RFCs: [RFC 0019](0019_local_source_memory_runtime.md), [RFC 0020](0020_runtime_backed_memory_remote_access.md), and [RFC 0051](0051_experience_skill_artifact_families.md) + +# Summary + +PowerContext needs a Source contract for external systems whose objects change over time. A GitHub issue, Notion page, +Slack message, or Linear issue has a stable provider identity, but its current value is allowed to change. The value +already used by an Artifact must remain exact. This RFC therefore separates a mutable Source head from immutable +snapshots of the values observed at particular points in time. + +The proposed model has two levels: + + Source: identifies the external object and points to its latest snapshot. + Source Snapshot: preserves the exact value observed at one provider revision and capture time. + +Artifacts, Candidates, Handoff records, and other exact-evidence consumers MUST cite an immutable Source Snapshot. +A provider URL or logical object ID alone is not sufficient evidence unless the provider guarantees that the referenced +revision is immutable and re-readable. + +The default read path may use the latest snapshot for search and ingestion. Historical Artifact lineage MUST continue +to use the snapshot that was present when the Artifact was produced. This RFC defines the PowerContext boundary only; +connectors remain responsible for discovery, synchronization, checkpoints, retries, credentials, and provider-specific +change handling. + +# Motivation + +The current Source implementation uses one identity for both the logical object and its captured value. The relational +primary key is effectively (scope_id, source_type, source_id), and a second payload for that key is a conflict. This is +correct for an immutable capture, but it cannot represent a mutable Source head plus retained historical snapshots. + +The current SourceRef also contains only source_type and source_id. Artifact lineage therefore identifies a logical key, +not the exact provider revision or payload that produced the Artifact. A later provider update could make the same +reference resolve to different content or become impossible to resolve. + +The design must preserve these properties: + +- the latest Source can change and be re-ingested without rewriting historical evidence; +- snapshots referenced by an Artifact remain readable after the provider changes or becomes unavailable; +- repeated delivery of the same snapshot is idempotent; +- connectors can use provider-native revisions such as a Git commit SHA, Notion edit marker, Slack message timestamp, + or file SHA-256; +- existing captured text remains useful without forcing every integration through an untyped payload; +- Source-specific adapters can participate in persistence, Runtime, transport, and evidence projection; +- connector synchronization does not become part of the Source model. + +# Proposed model + +## Source head and snapshots + +The public model should distinguish a mutable Source head from an immutable snapshot reference. Names are illustrative: + + { + "source_type": "github-issue", + "source_id": "oceanbase/powercontext#1240", + "snapshot_id": "snap_01J..." + } + +The mutable Source head contains at least: + +- source_type and source_id: stable logical identity; +- latest_snapshot_id: the snapshot used by the current read and ingest path; +- locator and provider metadata needed by a connector. + +A snapshot contains at least: + +- snapshot_id: PowerContext identity of the immutable snapshot; +- source_type and source_id: the stable logical object identity; +- provider_revision: provider-native revision, when available; +- materialization: captured or referenced; +- content_hash: hash of the canonical observed value; +- captured_at: PowerContext capture timestamp; +- payload: canonical captured value when materialized; +- locator: provider URL or provider-specific locator when useful. + +snapshot_id MUST identify one immutable payload. Reusing it with a different canonical payload is a conflict. A changed +provider revision creates a new snapshot and advances latest_snapshot_id; it MUST NOT update the payload of a snapshot +already referenced by an Artifact. Equal canonical payloads MAY reuse a snapshot when provider provenance is equivalent. + +SnapshotRef is the exact-evidence citation boundary. During migration, a legacy SourceRef without snapshot_id may resolve +only to the one immutable payload represented by an existing legacy row. Newly created exact evidence MUST include +snapshot_id. A two-part reference MUST NOT silently resolve to the latest snapshot when that could change historical +Artifact lineage. + +## Capture, Ref, and Hybrid + +Capture is the default for evidence that enters an Artifact lineage. It stores the canonical value and hash locally as a +snapshot, so provider availability and future edits cannot change historical evidence. Connectors may capture every +observed change or only changes admitted to the PowerContext ingestion boundary; once a snapshot is referenced by an +Artifact, it is retained according to the retention policy. + +Ref is allowed only when a provider revision has a documented immutable and re-readable contract. A mutable URL, object ID, +or current updated_at value is not enough. If a Ref cannot be resolved with the recorded revision, the snapshot is +unavailable and MUST NOT be silently replaced by the provider current value. An unavailable Ref cannot satisfy an +Artifact's exact-evidence requirement unless it is first materialized as a retained snapshot. + +Hybrid stores both the provider locator/revision and the canonical captured value in the snapshot. It is preferred when +external traceability matters, provider reads are expensive, or a connector may reconcile later changes. + +The resulting flow is: + + connector discovers source + -> new snapshot with provider revision and hash when the value changes + -> advance Source.latest_snapshot_id + -> Artifact lineage cites the snapshot used for generation + +## ContentSource and the content API + +ContentSource remains one concrete built-in captured-text Source. Its current behavior is effectively a single immutable +snapshot per source ID; the future model should allow a new snapshot when the same logical ID receives new content. +POST /v1/sources/content remains the compatibility and minimum-ingestion path for callers that already have text. It is +not the universal representation of GitHub, Notion, Slack, Linear, or other provider objects. + +Additional integrations should define typed Source and capture models. They may reuse common snapshot persistence, +hashing, idempotency, and citation machinery while keeping provider-specific fields in their typed payload. + +## Connector boundary + +Connectors own discovery, provider authentication, cursors, checkpoints, polling, webhooks, retries, rate limits, and +conversion of provider responses into typed snapshots. + +PowerContext owns Source and snapshot identity validation, durable snapshot storage and idempotency, exact reads by +snapshot reference, scope isolation, Artifact foreign-key integrity, evidence projection, and citation validation. + +This division follows the useful part of TencentDB-Agent-Memory's ISourceFetcher and SourceFetcherRegistry: fetchers route +provider protocols and return a provider version, while the core stores metadata and serves memory operations. PowerContext +additionally retains immutable evidence because Artifact lineage needs stronger replay guarantees. + +# TencentDB-Agent-Memory research + +This RFC is informed by [TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory): + +- MemoryKnowledge routes source protocols through ISourceFetcher and SourceFetcherRegistry. +- Git synchronization returns a commit hash and stores repo_url, branch, version, and last_sync_at. +- Wiki source files use filename plus sha256 for incremental change detection and track ingest status separately. +- Wiki and CodeGraph maintain asset-level version counters and audit rows for sync lifecycle events. +- MemoryCore separates knowledge metadata from content/indexing and uses versioned records for memory evolution. + +These patterns are useful for connector boundaries, provider revision metadata, content hashing, incremental sync, and +operational audit. TencentDB-Agent-Memory treats a changed hash as a reason to pull and ingest the new value; retaining +the prior source text is not its primary contract. PowerContext has a different requirement: when an Artifact already +cites that text, the prior payload must remain available after the current Source changes. PowerContext therefore adopts +the metadata and fetcher separation, but adds a retained immutable snapshot for Artifact citations. + +References: + +- [SourceFetcher types](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/source-fetcher/types.ts) +- [SourceFetcher registry](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/source-fetcher/registry.ts) +- [Git fetcher](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/source-fetcher/git-fetcher.ts) +- [Wiki source index](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/index-db.ts) +- [Knowledge metadata schema](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/db/client.ts) +- [CodeGraph service](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/store/code-graph-service.ts) + +# Persistence and Runtime impact + +The implementation is expected to introduce a mutable Source/current-head representation alongside an immutable snapshot +table. Artifact lineage should reference the immutable snapshot key rather than only source_type/source_id. The exact +migration shape is left to implementation, but it MUST preserve old captured Sources and reject payload replacement under +an existing snapshot identity. + +A new typed Source must register with Source resolution and exact reads, persistence encoding and decoding, Runtime +composition where applicable, HTTP/client mapping if exposed, evidence projection, citation validation, and focused +persistence and end-to-end tests. Evidence projectors SHOULD use adapter capabilities or a registry instead of accumulating +ContentSource-specific branches in Runtime composition. + +Existing ContentSource captures remain readable. Existing POST /v1/sources/content remains idempotent for its current +identity/payload contract. Legacy two-part references require an explicit migration rule and MUST NOT silently mean the +latest snapshot when that could change historical Artifact lineage. OpenAPI changes follow agreement on this identity +and legacy behavior. + +# Alternatives considered + +## Keep the current stable Source key + +This keeps the smallest API, but provider changes remain conflicts and multiple snapshots cannot be cited. It is suitable +only for already-immutable captures. + +## Use only provider references + +This minimizes storage, but exact evidence depends on provider retention, permissions, availability, and historical-read +semantics. It cannot satisfy replay for arbitrary integrations. + +## Make every provider value a ContentSource + +This reduces public types but moves provider semantics into untyped metadata, weakens validation, and makes evidence +projection provider-blind. It is not a durable extension boundary. + +## Store only a local revision + +A local revision helps ordering and cursors but does not prove which provider state was observed. It complements, rather +than replaces, provider revision and content hash. + +# Rollout and validation + +The first bounded validation should cover one mutable provider object and one immutable provider revision. A GitHub +issue/commit or Git repository commit is a practical candidate. It should demonstrate: + +1. repeated delivery of one provider revision is idempotent; +2. a later provider revision creates a new snapshot and advances the current head without replacing the old snapshot; +3. current search and ingestion use the new snapshot; +4. an Artifact still cites and reads the old snapshot after the provider changes; +5. a missing or unverifiable Ref is rejected rather than silently refreshed; +6. evidence projection and HTTP/client mapping preserve the exact snapshot reference. + +# Open questions + +- Should snapshot_id be opaque and generated by PowerContext, content-addressed, or expose both forms? +- Should SourceRef grow a snapshot_id, or should a separate SnapshotRef be introduced? +- Which provider revision guarantees qualify for Ref-only materialization? +- What retention and garbage-collection policy applies to snapshots no longer referenced by an Artifact? +- Should large captures use an external blob store while retaining a canonical hash and durable locator? +- Should logical Source heads be public API, or remain connector/persistence metadata initially? + +# Decision requested + +Approve the mutable Source head plus immutable snapshot model, snapshots as the exact-evidence boundary, Capture as the +default materialization, Hybrid as the preferred traceable form, and ContentSource as one concrete capture type rather +than a universal provider model. After approval, implementation can define the concrete schema, migration, OpenAPI +fields, retention rules, and one bounded connector validation. diff --git a/docs/zh/rfcs/1240_source_integration_shape.md b/docs/zh/rfcs/1240_source_integration_shape.md new file mode 100644 index 000000000..2bdc69989 --- /dev/null +++ b/docs/zh/rfcs/1240_source_integration_shape.md @@ -0,0 +1,201 @@ +# Source 集成形态 + +- 提案名称：source_integration_shape +- 状态：Proposed +- 开始日期：2026-08-25 +- Tracking Issue：[oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240) +- 相关 RFC：[RFC 0019](0019_local_source_memory_runtime.md)、[RFC 0020](0020_runtime_backed_memory_remote_access.md)、[RFC 0051](0051_experience_skill_artifact_families.md) + +# 摘要 + +PowerContext 需要为会随时间变化的外部系统定义 Source contract。GitHub issue、Notion page、Slack message 或 +Linear issue 具有稳定的 provider identity，但当前值允许变化；已经被 Artifact 使用的原文必须保持精确。本 RFC +因此将可变的 Source head 与某个时间点观察到的不可变 snapshot 分开。 + +建议的模型分为两层： + + Source：标识外部对象，并指向最新 snapshot。 + Source Snapshot：保存某个 provider revision 和捕获时间下的确切值。 + +Artifact、Candidate、Handoff 以及其他 exact-evidence 消费者必须引用不可变 Source Snapshot。只有 provider URL 或 +逻辑对象 ID 不是充分证据，除非 provider 保证该 revision 不可变且可重新读取。 + +默认读取路径可以使用最新 snapshot 进行检索和 ingest；历史 Artifact lineage 必须继续使用 Artifact 生成时采用的 +snapshot。本 RFC 只定义 PowerContext 的边界；Connector 继续负责 discovery、同步、checkpoint、重试、凭证和 +provider-specific change handling。 + +# 动机 + +当前 Source 实现用同一个 identity 同时表示逻辑对象和捕获值。关系数据库主键实际上是 +(scope_id, source_type, source_id)，同一 key 写入第二个 payload 会冲突。这对不可变 capture 是正确的，但无法表示 +可变 Source head 以及保留的历史 snapshot。 + +当前 SourceRef 也只有 source_type 和 source_id。因此 Artifact lineage 只能标识逻辑 key，不能标识生成 Artifact 时 +实际使用的 provider revision 或 payload。provider 后续更新后，同一个引用可能解析到不同内容，或者无法再解析。 + +设计必须保持：最新 Source 可以变化并重新 ingest 而不改写历史证据；被 Artifact 引用的 snapshot 在 provider 变化或 +不可用后仍可读取；同一个 snapshot 的重复投递幂等；connector 可以 +使用 Git commit SHA、Notion edit marker、Slack message timestamp 或文件 SHA-256；现有 captured text 仍然有用； +typed Source 可以接入 persistence、Runtime、transport 和 evidence projection；connector 同步逻辑不进入 Source model。 + +# 设计说明 + +## Source head 与 snapshot + +公共模型应区分可变 Source head 与不可变 snapshot 引用。名称仅为示意： + + { + "source_type": "github-issue", + "source_id": "oceanbase/powercontext#1240", + "snapshot_id": "snap_01J..." + } + +可变 Source head 至少包含： + +- source_type、source_id：稳定逻辑身份； +- latest_snapshot_id：当前读取和 ingest 使用的 snapshot； +- connector 所需的 locator 与 provider metadata。 + +Snapshot 至少包含： + +- snapshot_id：PowerContext 为不可变 snapshot 生成的身份； +- source_type、source_id：稳定逻辑对象身份； +- provider_revision：provider-native revision（如果有）； +- materialization：captured 或 referenced； +- content_hash：canonical observed value 的 hash； +- captured_at：PowerContext 捕获时间； +- payload：materialized 时的 canonical captured value； +- locator：需要时保存 provider URL 或 provider-specific locator。 + +snapshot_id 必须只对应一份不可变 payload。相同 snapshot identity 写入不同 payload 必须冲突。provider revision +变化时生成新 snapshot 并推进 latest_snapshot_id；不得更新已经被 Artifact 引用的 snapshot payload。provider +provenance 等价时，相同 canonical payload 可以复用 snapshot。 + +SnapshotRef 是 exact-evidence 的 citation boundary。迁移期间，缺少 snapshot_id 的旧 SourceRef 只能解析到现有旧行 +代表的那份不可变 payload。新创建的 exact evidence 必须带 snapshot_id；如果会改变历史 Artifact lineage 的含义， +两段式引用不得静默解析为 latest snapshot。 + +## Capture、Ref 与 Hybrid + +进入 Artifact lineage 的证据默认使用 Capture。它将 canonical value 和 hash 作为 snapshot 保存在本地，因此 +provider 可用性和未来编辑不会改变历史证据。Connector 可以捕获每次变化，也可以只把进入 PowerContext ingestion +boundary 的变化保存下来；一旦 snapshot 被 Artifact 引用，就必须按 retention policy 保留。 + +只有在 provider 明确保证 revision 不可变且可重新读取时，才允许使用 Ref。可变 URL、对象 ID 或当前 updated_at 不够。 +如果 Ref 无法按记录的 revision 解析，snapshot 必须标记为 unavailable，不能静默替换为 provider 当前值。变得 +unavailable 的 Ref 不能满足 Artifact 的 exact-evidence 要求，除非先物化为保留的 snapshot。 + +Hybrid 在 snapshot 中同时保存 provider locator/revision 和 canonical captured value。当需要外部追溯、provider +读取成本较高，或 connector 需要协调后续变更时，优先使用 Hybrid。 + +流程为： + + connector 发现 source + -> 内容变化时生成带 provider revision 和 hash 的新 snapshot + -> 推进 Source.latest_snapshot_id + -> Artifact lineage 引用生成时采用的 snapshot + +## ContentSource 与 content API + +ContentSource 保持为一个具体的内置 captured-text Source。它当前实际上是每个 source ID 只有一份不可变 snapshot；未来 +同一逻辑 ID 收到新内容时应允许生成新 snapshot。POST /v1/sources/content 继续作为已有调用方的兼容入口和 +最小 ingestion path，不升级为 GitHub、Notion、Slack、Linear 等 provider 对象的统一表示。 + +新增集成应定义 typed Source 和 capture model。它们可以复用通用 snapshot persistence、hash、幂等和 citation 机制， +但 provider-specific 字段应放在 typed payload 中。 + +## Connector 边界 + +Connector 负责 discovery、provider authentication、cursor、checkpoint、polling、webhook、retry、rate limit，以及将 +provider response 转换为 typed snapshot。 + +PowerContext 负责校验 Source 和 snapshot identity、durable snapshot storage 与幂等、按 snapshot reference +精确读取、scope isolation、Artifact foreign-key integrity、evidence projection 和 citation validation。 + +这部分借鉴 TencentDB-Agent-Memory 的 ISourceFetcher 和 SourceFetcherRegistry：fetcher 路由 provider protocol 并返回 +provider version，core 保存 metadata 并提供 memory operation。PowerContext 还需要额外保留不可变 evidence，因为 +Artifact lineage 需要更强的 replay guarantee。 + +# TencentDB-Agent-Memory 调研参考 + +本设计参考了 [TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)： + +- MemoryKnowledge 通过 ISourceFetcher 和 SourceFetcherRegistry 路由 source protocol； +- Git 同步返回 commit hash，并保存 repo_url、branch、version、last_sync_at 等 metadata； +- Wiki source file 使用 filename 加 sha256 做增量变化检测，并独立跟踪 ingest status； +- Wiki 和 CodeGraph 使用 asset-level version counter 与 audit row 记录同步生命周期； +- MemoryCore 将 knowledge metadata 与 content/indexing service 分离，并用 versioned record 管理 memory 演进。 + +这些做法适合借鉴 connector 边界、provider revision metadata、content hashing、增量同步和运维审计。 +TencentDB-Agent-Memory 把 hash 变化视为重新拉取并 ingest 最新值的信号，保留旧 source 原文不是它的主要 contract。 +PowerContext 的要求不同：当 Artifact 已经引用这份原文时，当前 Source 变化后旧 payload 仍必须可用。因此 PowerContext +采用 metadata 与 fetcher 分离，同时为 Artifact citation 增加保留的 immutable snapshot。 + +上游关键参考文件： + +- [SourceFetcher types](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/source-fetcher/types.ts) +- [SourceFetcher registry](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/source-fetcher/registry.ts) +- [Git fetcher](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/source-fetcher/git-fetcher.ts) +- [Wiki source index](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/index-db.ts) +- [Knowledge metadata schema](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/db/client.ts) +- [CodeGraph service](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/store/code-graph-service.ts) + +# Persistence 与 Runtime 影响 + +实现预计需要在可变 Source/current-head 表示之外增加不可变 snapshot 表。Artifact lineage 应引用不可变 snapshot key， +而不是只保存 source_type/source_id。具体迁移结构留给实现阶段，但必须保留旧 captured Source，并拒绝在已有 +snapshot identity 下替换 payload。 + +新增 typed Source 必须接入 Source resolve 与 exact read、persistence encoding 与 decoding、Runtime composition、远程暴露时 +的 HTTP/client mapping、evidence projection、citation validation，以及聚焦的 persistence 和 end-to-end tests。Evidence +projector 应通过 adapter capability 或 registry 路由，避免 Runtime composition 不断累积 ContentSource 特判。 + +现有 ContentSource capture 必须继续可读；现有 POST /v1/sources/content 在当前 identity/payload contract 下继续幂等。旧的 +两段式引用需要显式迁移规则，不能在会改变历史 Artifact lineage 含义时静默解释为 latest snapshot。OpenAPI 变更应在 +identity 和 legacy behavior 达成共识后进行。 + +# 备选方案 + +## 保持当前 stable Source key + +API 最小，但 provider 对象变化仍然冲突，也无法引用多次 snapshot，只适合已经不可变的 capture。 + +## 只使用 provider reference + +存储成本低，但 exact evidence 依赖 provider retention、权限、可用性和历史读取语义，无法满足任意集成的 replay 要求。 + +## 所有 provider value 都使用 ContentSource + +类型更少，但 provider semantics 进入无类型 metadata，校验能力变弱，evidence projection 也无法感知 provider，不是持久的 +扩展边界。 + +## 只保存本地 revision + +本地 revision 有助于排序和 cursor，但不能证明观察到哪一份 provider state，只能补充而不能替代 provider revision 和 hash。 + +# 落地与验证 + +首个有界验证应覆盖一个会变化的 provider object 和一个不可变 provider revision。GitHub issue/commit 或 Git repository +commit 都是候选。验证至少应证明： + +1. 同一个 provider revision 的重复投递幂等； +2. 后续 provider revision 产生新 snapshot 并推进当前 head，而不替换旧 snapshot； +3. 当前检索和 ingest 使用新 snapshot； +4. provider 变化后 Artifact 仍能引用并读取旧 snapshot； +5. 缺失或无法验证的 Ref 被拒绝，而不是静默刷新； +6. evidence projection 与 HTTP/client mapping 保留精确 snapshot reference。 + +# 未决问题 + +- snapshot_id 应由 PowerContext 生成 opaque ID、使用 content-addressed ID，还是同时提供两者？ +- SourceRef 是否增加 snapshot_id，还是引入独立 SnapshotRef？ +- 哪些 provider revision guarantee 可以允许 Ref-only materialization？ +- 不再被 Artifact 引用的 snapshot 应采用什么 retention 和 garbage-collection policy？ +- 大型 capture 是否使用外部 blob store，同时保留 canonical hash 和 durable locator？ +- logical Source head 是否应成为首版 public API，还是初期只作为 connector/persistence metadata？ + +# 请求决策 + +请评审并批准：可变 Source head 加不可变 snapshot 模型；snapshot 作为 exact-evidence boundary；Capture 作为默认 +materialization；Hybrid 作为可追溯场景的优先形态；ContentSource 作为一个具体 capture 类型而不是 universal provider +model。批准后再定义具体 schema、migration、OpenAPI 字段、retention 规则和一个有界 connector validation。 From 76247e3b39cd11cce4fffdc4afe749808efdfbea Mon Sep 17 00:00:00 2001 From: Nene7ko_ <1604009816@qq.com> Date: Sat, 29 Aug 2026 01:34:29 +0800 Subject: [PATCH 2/5] docs(rfc): fix source integration metadata --- ...on_shape.md => 1388_source_integration_shape.md} | 13 +++++++------ ...on_shape.md => 1388_source_integration_shape.md} | 13 +++++++------ zensical.toml | 2 ++ 3 files changed, 16 insertions(+), 12 deletions(-) rename docs/en/rfcs/{1240_source_integration_shape.md => 1388_source_integration_shape.md} (94%) rename docs/zh/rfcs/{1240_source_integration_shape.md => 1388_source_integration_shape.md} (94%) diff --git a/docs/en/rfcs/1240_source_integration_shape.md b/docs/en/rfcs/1388_source_integration_shape.md similarity index 94% rename from docs/en/rfcs/1240_source_integration_shape.md rename to docs/en/rfcs/1388_source_integration_shape.md index 83c7b0bdb..62275d9a3 100644 --- a/docs/en/rfcs/1240_source_integration_shape.md +++ b/docs/en/rfcs/1388_source_integration_shape.md @@ -3,6 +3,7 @@ - Proposal Name: source_integration_shape - Status: Proposed - Start Date: 2026-08-25 +- RFC PR: [oceanbase/powercontext#1388](https://github.com/oceanbase/powercontext/pull/1388) - Tracking Issue: [oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240) - Related RFCs: [RFC 0019](0019_local_source_memory_runtime.md), [RFC 0020](0020_runtime_backed_memory_remote_access.md), and [RFC 0051](0051_experience_skill_artifact_families.md) @@ -148,12 +149,12 @@ the metadata and fetcher separation, but adds a retained immutable snapshot for References: -- [SourceFetcher types](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/source-fetcher/types.ts) -- [SourceFetcher registry](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/source-fetcher/registry.ts) -- [Git fetcher](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/source-fetcher/git-fetcher.ts) -- [Wiki source index](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/index-db.ts) -- [Knowledge metadata schema](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/db/client.ts) -- [CodeGraph service](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/store/code-graph-service.ts) +- [SourceFetcher types](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/source-fetcher/types.ts) +- [SourceFetcher registry](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/source-fetcher/registry.ts) +- [Git fetcher](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/source-fetcher/git-fetcher.ts) +- [Wiki source index](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/engines/wiki/index-db.ts) +- [Knowledge metadata schema](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/db/client.ts) +- [CodeGraph service](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/store/code-graph-service.ts) # Persistence and Runtime impact diff --git a/docs/zh/rfcs/1240_source_integration_shape.md b/docs/zh/rfcs/1388_source_integration_shape.md similarity index 94% rename from docs/zh/rfcs/1240_source_integration_shape.md rename to docs/zh/rfcs/1388_source_integration_shape.md index 2bdc69989..18bfa7acc 100644 --- a/docs/zh/rfcs/1240_source_integration_shape.md +++ b/docs/zh/rfcs/1388_source_integration_shape.md @@ -3,6 +3,7 @@ - 提案名称：source_integration_shape - 状态：Proposed - 开始日期：2026-08-25 +- RFC PR：[oceanbase/powercontext#1388](https://github.com/oceanbase/powercontext/pull/1388) - Tracking Issue：[oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240) - 相关 RFC：[RFC 0019](0019_local_source_memory_runtime.md)、[RFC 0020](0020_runtime_backed_memory_remote_access.md)、[RFC 0051](0051_experience_skill_artifact_families.md) @@ -133,12 +134,12 @@ PowerContext 的要求不同：当 Artifact 已经引用这份原文时，当前 上游关键参考文件： -- [SourceFetcher types](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/source-fetcher/types.ts) -- [SourceFetcher registry](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/source-fetcher/registry.ts) -- [Git fetcher](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/source-fetcher/git-fetcher.ts) -- [Wiki source index](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/index-db.ts) -- [Knowledge metadata schema](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/db/client.ts) -- [CodeGraph service](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/store/code-graph-service.ts) +- [SourceFetcher types](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/source-fetcher/types.ts) +- [SourceFetcher registry](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/source-fetcher/registry.ts) +- [Git fetcher](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/source-fetcher/git-fetcher.ts) +- [Wiki source index](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/engines/wiki/index-db.ts) +- [Knowledge metadata schema](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/db/client.ts) +- [CodeGraph service](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/store/code-graph-service.ts) # Persistence 与 Runtime 影响 diff --git a/zensical.toml b/zensical.toml index 61999ad39..6e8f0e84e 100644 --- a/zensical.toml +++ b/zensical.toml @@ -53,6 +53,7 @@ nav = [ { "0080 Memory Search Reranking" = "en/rfcs/0080_memory_search_reranking.md" }, { "0081 End-to-end Evaluation Architecture" = "en/rfcs/0081_end_to_end_evaluation_architecture.md" }, { "1229 Unified Workloads and Long-Horizon Memory Evaluation" = "en/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md" }, + { "1388 Source Integration Shape" = "en/rfcs/1388_source_integration_shape.md" }, { "0072 Scoped Statistics and Usage" = "en/rfcs/0072_scoped_statistics_and_usage.md" }, ] }, { "Meetings" = [ @@ -105,6 +106,7 @@ nav = [ { "0080 Memory 搜索 Rerank" = "zh/rfcs/0080_memory_search_reranking.md" }, { "0081 端到端评估架构" = "zh/rfcs/0081_end_to_end_evaluation_architecture.md" }, { "1229 统一工作负载与长程 Memory 评估" = "zh/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md" }, + { "1388 Source 集成形态" = "zh/rfcs/1388_source_integration_shape.md" }, { "0072 Scoped Statistics 与 Usage" = "zh/rfcs/0072_scoped_statistics_and_usage.md" }, ] }, { "会议纪要" = [ From a794c35db06d994cafe02c0207f2fd3c65b8ec55 Mon Sep 17 00:00:00 2001 From: Nene7ko_ <1604009816@qq.com> Date: Sat, 29 Aug 2026 02:41:13 +0800 Subject: [PATCH 3/5] docs(rfc): establish source observation model baseline --- docs/en/rfcs/1388_source_integration_shape.md | 686 ++++++++++++++---- docs/zh/rfcs/1388_source_integration_shape.md | 633 ++++++++++++---- 2 files changed, 1031 insertions(+), 288 deletions(-) diff --git a/docs/en/rfcs/1388_source_integration_shape.md b/docs/en/rfcs/1388_source_integration_shape.md index 62275d9a3..76f69e65f 100644 --- a/docs/en/rfcs/1388_source_integration_shape.md +++ b/docs/en/rfcs/1388_source_integration_shape.md @@ -1,137 +1,558 @@ -# Source Integration Shape - -- Proposal Name: source_integration_shape +- Proposal Name: `source_integration_shape` - Status: Proposed - Start Date: 2026-08-25 - RFC PR: [oceanbase/powercontext#1388](https://github.com/oceanbase/powercontext/pull/1388) - Tracking Issue: [oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240) -- Related RFCs: [RFC 0019](0019_local_source_memory_runtime.md), [RFC 0020](0020_runtime_backed_memory_remote_access.md), and [RFC 0051](0051_experience_skill_artifact_families.md) +- Related Discussion: [oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240), + [oceanbase/powercontext#1363](https://github.com/oceanbase/powercontext/issues/1363) +- Related Design: [oceanbase/powercontext#1345](https://github.com/oceanbase/powercontext/pull/1345) +- Related RFCs: [RFC 0002](0002_core_sdk_product_model.md), [RFC 0014](0014_memory_layer_design.md), + [RFC 0019](0019_local_source_memory_runtime.md), [RFC 0048](0048_handoff_artifact.md) # Summary -PowerContext needs a Source contract for external systems whose objects change over time. A GitHub issue, Notion page, -Slack message, or Linear issue has a stable provider identity, but its current value is allowed to change. The value -already used by an Artifact must remain exact. This RFC therefore separates a mutable Source head from immutable -snapshots of the values observed at particular points in time. +This RFC defines the standard Source model and the contract for defining additional Source types. + +A Source belongs to exactly one Scope. Within that Scope, a `SourceKey` identifies one logical source and a +`SourceRef` identifies one immutable observation of that source. Advancing the current observation, observing a +deletion, changing an external locator, or disconnecting a Connector does not alter an earlier observation or move +it to another Scope. + +A Source Definition gives one stable Source type its value schema, provenance schema, identity rules, observation +rules, materialization contract, canonicalization, and compatibility policy. Definitions are registered explicitly +and remain fixed for the lifetime of a composed Runtime. Persistence, transport, and Artifact consumers route by the +stable definition name and version rather than by a concrete Python class. + +A Definition may advertise named projection capabilities for consumers that do not understand its native value. +Each projection has an independently versioned schema and deterministic meaning over one exact observation. A +consumer selects a projection by capability name and version, never by inspecting a concrete Source class. -The proposed model has two levels: +A Connector lifecycle binds provider acquisition to a Scope, resolves definition-native inputs in its worker, +submits materialized observations, records per-item outcomes, and advances an opaque checkpoint only after accepted +observations are durable. Connector runs distinguish complete discovery from incomplete discovery so that absence +is not silently converted into deletion. - Source: identifies the external object and points to its latest snapshot. - Source Snapshot: preserves the exact value observed at one provider revision and capture time. +Materialization identifies the authority used to resolve an exact observation. A captured observation is resolved +from the canonical value retained by PowerContext. A referenced observation is resolved from an immutable external +revision. An external locator, modification time, ETag, or current-provider read does not by itself satisfy the +referenced contract. -Artifacts, Candidates, Handoff records, and other exact-evidence consumers MUST cite an immutable Source Snapshot. -A provider URL or logical object ID alone is not sufficient evidence unless the provider guarantees that the referenced -revision is immutable and re-readable. +`ContentSource` remains a simple captured-text Source. Its caller-stable identity and immutable-payload conflict rule +make it useful for one-shot content capture, but it is not the general external integration model. -The default read path may use the latest snapshot for search and ingestion. Historical Artifact lineage MUST continue -to use the snapshot that was present when the Artifact was produced. This RFC defines the PowerContext boundary only; -connectors remain responsible for discovery, synchronization, checkpoints, retries, credentials, and provider-specific -change handling. +This RFC defines Source, projection, Connector lifecycle, and the remote ingestion boundary between a worker and the +PowerContext Server. It does not define plugin discovery, a scheduler, credential transport, a concrete Source +family, or a Connector implementation. # Motivation -The current Source implementation uses one identity for both the logical object and its captured value. The relational -primary key is effectively (scope_id, source_type, source_id), and a second payload for that key is a conflict. This is -correct for an immutable capture, but it cannot represent a mutable Source head plus retained historical snapshots. +`ContentSource` and `POST /v1/sources/content` provide captured-text ingestion. The caller +chooses one `source_id`; replaying an identical payload is idempotent, while reusing that identity with a different +payload is a conflict. This gives exact evidence only when the caller treats the identity as immutable. + +External systems usually expose a different lifecycle. A wiki page, issue, object, message, or file has one logical +identity but may produce several values over time. The external object can be renamed, revised, deleted, restored, +or become temporarily unreadable. Artifacts that used an earlier value must continue to cite that exact evidence. +The two-part `(source_type, source_id)` Source reference cannot express both the stable logical object and its immutable observation +without making every integration invent a composite `source_id`. + +The extension boundary is also incomplete. A Source adapter binds a native input class to a concrete +Source class and a read result, while the built-in Runtime and relational persistence assemble a fixed adapter set. +This does not state the durable rules an independently defined Source type must follow across identity, persistence, +transport, and Artifact evidence. + +The standard model must answer six questions without assigning them to one identifier: + +1. Which Scope owns this evidence? +2. Which logical external or internal source does it describe? +3. Which exact observed value did an Artifact use? +4. Where does PowerContext read that exact value from? +5. Which definition gives the value and provenance their meaning? +6. Which declared view may a consumer use without understanding the native value? + +Connector concerns are adjacent but distinct. Discovery, credentials, filtering, checkpoints, retries, provider +change handling, and deletion detection decide which observations are submitted. They do not define Source identity, +weaken exact evidence, or change Scope ownership. + +# Guide-level explanation + +## Domain model + +Read the model by establishing ownership first, then logical identity, exact observation, materialization authority, +and type semantics: + +| Concept | Representation | Question answered | +| --- | --- | --- | +| Ownership | Scope | Where does the Source belong? | +| Logical identity | `SourceKey` | Which continuing source is this? | +| Exact evidence | `SourceRef` | Which immutable observation is cited? | +| Read authority | materialization | Where is that exact value resolved? | +| Type semantics | Source Definition | How are value, provenance, and identity interpreted? | +| Consumer view | named projection | Which declared representation may a consumer use? | +| Acquisition | Connector or direct caller | How are new observations found and submitted? | + +These responsibilities form one direction of dependency: + +```text +Connector or direct caller + | + v +Source Definition + | + v +Scope-owned Source history + | + +---- mutable head selection + | + `---- exact SourceRef ----> Artifact evidence +``` + +A Connector can use one Source Definition, several Connectors can use the same Definition, and a direct caller can +submit a Source without a Connector. Connector identity therefore does not become Source type identity. + +## Scope ownership + +Every SourceKey and observation belongs to exactly one Scope. Scope ownership is not inferred from an external +workspace, path, repository, provider account, Connector instance, or Source locator. Those values may contribute to +binding or provenance, but they do not allocate or replace `scope_id`. + +The fully qualified logical identity is: + +```text +SourceKey = (scope_id, source_type, source_id) +``` + +The fully qualified exact identity is: + +```text +SourceRef = (scope_id, source_type, source_id, observation_id) +``` + +A scope-bound operation may obtain `scope_id` from its fixed request binding instead of accepting it as an arbitrary +argument. The durable resolved reference still retains the owner Scope so that evidence remains unambiguous after +publication, reporting, or export. + +Changing a Scope Parent, Context References, an Agent binding, or an observation selection changes no SourceKey or +SourceRef. Publishing an Artifact across Scopes preserves the original Scope and exact SourceRef in provenance. It +does not move or implicitly copy the Source history. + +## Logical Source and immutable observation + +`source_id` names a logical source within one `(scope_id, source_type)` namespace. Its meaning is defined by the +Source Definition. It may correspond to a provider object ID, a stable import identity, or another normalized key. +It must not silently change when a new value is observed. + +`observation_id` names one immutable observation under a SourceKey. It is opaque to generic PowerContext components. +It may be derived from a provider revision, a canonical value digest, or a definition-specific combination. It does +not imply an integer sequence, timestamp order, or ancestry. + +The following invariants apply: + +- one `(SourceKey, observation_id)` identifies one canonical observation forever; +- re-observing the same canonical observation is idempotent; +- a different canonical observation cannot reuse an observation ID; +- one SourceKey may have several observations with the same value digest when their identity-bearing provenance is + different; +- observations with the same value digest are not automatically the same logical Source; and +- an Artifact cites an exact SourceRef, never a moving SourceKey or `latest` observation. + +For example, updating one logical Source retains its SourceKey and produces another SourceRef: + +```text +SourceKey(scope-a, record, provider-object-42) +|-- SourceRef(..., observation-1) "Initial value" +`-- SourceRef(..., observation-2) "Revised value" +``` + +An Artifact derived from `observation-1` continues to cite it after `observation-2` becomes current. + +## Source Definition + +A Source Definition is the durable semantic contract for one `source_type`. It declares: + +- a stable definition name and version; +- the Source value and typed provenance shapes; +- Source ID normalization and equality; +- observation ID normalization and equality; +- identity-bearing fields and non-identifying annotations; +- canonical bytes and the value digest algorithm; +- supported materialization modes and exact-read requirements; +- limits and validation failures; and +- compatibility rules for older definition versions. + +A Definition resolves definition-native input into a canonical observation and reads the definition-owned value from +an exact persisted observation. Resolution does not select a Scope, mutate a catalog, advance a head, or discover +external objects. Reading does not resolve `latest` or substitute another observation. + +Definitions are explicit and typed. A new integration must not simulate a new Source type by placing an undocumented +schema inside `ContentSource.metadata`. Provider-specific provenance may extend a Definition's declared schema, but +fields that affect identity, exactness, or compatibility must be named by the Definition. + +## Named projection capabilities + +A named projection is an optional, Definition-owned view of one exact observation. It allows an Artifact family or +another consumer to use a declared representation without knowing the native Source value or concrete Python class. + +A projection is selected by a stable name and version. Its Definition declares the output schema, canonicalization, +digest rules, and failures. The projection is evaluated against an exact SourceRef and cannot resolve a head, +`latest`, or a current provider value. For the same Definition version, projection version, and exact observation, it +returns the same canonical result. + +Projection capability is explicit. A consumer that requires a projection rejects a Source that does not advertise a +compatible capability; it does not infer content from metadata or fall back to a similarly shaped Source class. A +projection can be cached or persisted as a derivative, but its authority remains the exact Source observation and +its lineage retains that SourceRef. + +This contract does not prescribe a catalog of standard projection names or payload schemas. A projection becomes a +shared standard only after interoperating definitions and consumers demonstrate that its semantics are stable. Until +then, a Definition may expose namespaced projections without making them mandatory for other Source types. + +## Materialization authority + +Materialization answers where the value returned for an exact SourceRef comes from: + +| Materialization | Authority | Required guarantee | +| --- | --- | --- | +| `captured` | Canonical value retained by PowerContext | The retained value matches the observation digest | +| `referenced` | Immutable external revision | Re-reading the reference returns the same canonical value and digest | + +A captured Source may retain an external locator, provider revision, and digest as provenance. It remains captured +because the retained value is the read authority. This covers the useful part of a hybrid design without creating a +third mode with ambiguous fallback semantics. + +A Definition can use referenced materialization only when the external system and its reader can address immutable +historical values. Reading the current value at a path, page ID, issue ID, or URL is not sufficient. Modification +times and ETags may contribute to provenance or conflict detection, but a Definition must state whether the provider +guarantees that they address an immutable value. + +When the referenced value is unavailable or its digest differs, exact resolution fails. PowerContext does not return +the current provider value, a stale cache entry, or another observation. A provider that cannot satisfy this rule +must use captured materialization or reject the observation. + +## Current head and deletion + +A Source history is immutable; its current head is a mutable catalog selection. The head can select one exact +SourceRef or record that the logical Source was positively observed as deleted. The head is useful for current-state +queries and later acquisition, but it is not evidence and cannot appear in an Artifact citation. + +Advancing or deleting a head changes no observation. A timeout, permission failure, incomplete listing, unavailable +Connector, or disconnect is not positive deletion evidence and does not change the head. A Source Definition may +define a tombstone value only when deletion itself is meaningful Source evidence; a generic head deletion does not +fabricate one. + +## ContentSource + +`ContentSource` remains the neutral captured-text path defined by RFC 0019. Its caller chooses an identity that can be +committed once with one canonical payload. The persistence conflict rule makes an accepted ContentSource exact, but +it does not provide a separate logical Source lifecycle. + +The standard model treats this as a valid single-observation Source implementation: + +- the existing identity remains immutable; +- an identical replay remains idempotent; +- different content under the same identity remains a conflict; +- references that resolve ContentSource remain exact and unchanged; and +- no mutable head or multi-observation behavior is inferred from metadata. + +ContentSource is suitable for prompts, explicit text capture, import records, and other cases where the caller +already owns an immutable identity. Integrations that observe one logical object over time should define or reuse a +multi-observation Source type instead. + +# Reference-level explanation -The current SourceRef also contains only source_type and source_id. Artifact lineage therefore identifies a logical key, -not the exact provider revision or payload that produced the Artifact. A later provider update could make the same -reference resolve to different content or become impossible to resolve. +## Source identity contract -The design must preserve these properties: +`scope_id` is the ownership boundary defined by the Scope organization design. `source_type` is the stable Source +Definition name. `source_id` is a non-empty, normalized identifier whose equality and bounds are declared by that +Definition. -- the latest Source can change and be re-ingested without rewriting historical evidence; -- snapshots referenced by an Artifact remain readable after the provider changes or becomes unavailable; -- repeated delivery of the same snapshot is idempotent; -- connectors can use provider-native revisions such as a Git commit SHA, Notion edit marker, Slack message timestamp, - or file SHA-256; -- existing captured text remains useful without forcing every integration through an untyped payload; -- Source-specific adapters can participate in persistence, Runtime, transport, and evidence projection; -- connector synchronization does not become part of the Source model. +Source identity is Scope-local. Two Scopes may contain equivalent external material without sharing ownership or +identity. A Definition may include a stable external instance or connection discriminator in its `source_id` rules +when required to prevent collisions, but the discriminator does not replace `scope_id`. -# Proposed model +Renames are definition-specific. A provider object ID may preserve SourceKey across locator changes. A path-derived +identity normally treats a rename as one logical deletion and one creation. A Definition must not claim rename-stable +identity when its provider and acquisition path cannot prove it. -## Source head and snapshots +## Observation contract -The public model should distinguish a mutable Source head from an immutable snapshot reference. Names are illustrative: +An observation contains these standard fields: - { - "source_type": "github-issue", - "source_id": "oceanbase/powercontext#1240", - "snapshot_id": "snap_01J..." - } +```text +SourceObservation +|-- source_key +|-- observation_id +|-- definition_version +|-- materialization +|-- value_digest +|-- provenance +`-- definition-owned value or exact external reference +``` -The mutable Source head contains at least: +`value_digest` uses SHA-256 over the canonical bytes declared by the Definition and is encoded as +`sha256:`. For structured values, the Definition specifies a deterministic canonicalization. The +digest verifies value equality; it does not replace SourceKey or observation identity. -- source_type and source_id: stable logical identity; -- latest_snapshot_id: the snapshot used by the current read and ingest path; -- locator and provider metadata needed by a connector. +The canonical observation contains every field that the Definition says affects identity or exact meaning. +Operational facts such as a retry count, last scan time, or processing status are not Source value and do not change +observation identity. If a timestamp or provider attribute affects provenance meaning, the Definition must classify +and canonicalize it explicitly. -A snapshot contains at least: +## Source reference contract -- snapshot_id: PowerContext identity of the immutable snapshot; -- source_type and source_id: the stable logical object identity; -- provider_revision: provider-native revision, when available; -- materialization: captured or referenced; -- content_hash: hash of the canonical observed value; -- captured_at: PowerContext capture timestamp; -- payload: canonical captured value when materialized; -- locator: provider URL or provider-specific locator when useful. +A SourceRef identifies an exact observation and includes its owner Scope. It never accepts an absent observation ID, +`latest`, a head version, or a current provider locator. -snapshot_id MUST identify one immutable payload. Reusing it with a different canonical payload is a conflict. A changed -provider revision creates a new snapshot and advances latest_snapshot_id; it MUST NOT update the payload of a snapshot -already referenced by an Artifact. Equal canonical payloads MAY reuse a snapshot when provider provenance is equivalent. +Within a scope-bound operation, a compact local representation may omit a repeated `scope_id` only while the current +Scope is fixed and the resolved durable value restores it. Any reference that crosses a Scope boundary, leaves the +Runtime, or enters durable cross-Scope provenance carries the owner Scope explicitly. -SnapshotRef is the exact-evidence citation boundary. During migration, a legacy SourceRef without snapshot_id may resolve -only to the one immutable payload represented by an existing legacy row. Newly created exact evidence MUST include -snapshot_id. A two-part reference MUST NOT silently resolve to the latest snapshot when that could change historical -Artifact lineage. +Reference resolution verifies all four identity components and the stored observation's definition version and +digest. Failure to resolve the exact observation is distinct from the logical Source being deleted, the head having +advanced, or the Connector being unavailable. -## Capture, Ref, and Hybrid +## Definition registration contract -Capture is the default for evidence that enters an Artifact lineage. It stores the canonical value and hash locally as a -snapshot, so provider availability and future edits cannot change historical evidence. Connectors may capture every -observed change or only changes admitted to the PowerContext ingestion boundary; once a snapshot is referenced by an -Artifact, it is retained according to the retention policy. +Executable Definitions belong to the worker that resolves definition-native inputs, canonicalizes Source values, +and computes named projections. The Server does not import Connector or Definition packages and does not execute +their Python classes. -Ref is allowed only when a provider revision has a documented immutable and re-readable contract. A mutable URL, object ID, -or current updated_at value is not enough. If a Ref cannot be resolved with the recorded revision, the snapshot is -unavailable and MUST NOT be silently replaced by the provider current value. An unavailable Ref cannot satisfy an -Artifact's exact-evidence requirement unless it is first materialized as a retained snapshot. +Before submitting an observation, the worker registers an immutable declarative manifest containing the stable +Definition name and version, the canonical Source JSON Schema, every projection key and output JSON Schema, and a +fingerprint over the complete declaration. The fingerprint is SHA-256 over RFC 8785 canonical JSON. Registration is +idempotent for an identical manifest and rejects a different declaration for an existing `(source_type, +definition_version)`. -Hybrid stores both the provider locator/revision and the canonical captured value in the snapshot. It is preferred when -external traceability matters, provider reads are expensive, or a connector may reconcile later changes. +The Server validates the manifest's schemas and any named projection it recognizes as a shared standard. A manifest +does not transfer executable identity rules, canonicalization code, read behavior, credentials, or provider +configuration. Those remain worker-owned. The registered manifest is sufficient for the Server to validate and +retain an opaque canonical observation without loading plugin code. -The resulting flow is: +Definition discovery and registration are separate. A package entry point or another discovery mechanism may report +available Definitions, but installation does not imply activation. This RFC does not select entry points, a central +settings format, pluggy, or a Connector marketplace. - connector discovers source - -> new snapshot with provider revision and hash when the value changes - -> advance Source.latest_snapshot_id - -> Artifact lineage cites the snapshot used for generation +## Remote worker ingestion contract -## ContentSource and the content API +A Connector runs in an independent worker process. The worker owns provider access and all executable Definition +behavior. The Server owns durable Source history, Artifact consumption, and checkpoint comparison. Their data-plane +interaction consists of four generic operations: -ContentSource remains one concrete built-in captured-text Source. Its current behavior is effectively a single immutable -snapshot per source ID; the future model should allow a new snapshot when the same logical ID receives new content. -POST /v1/sources/content remains the compatibility and minimum-ingestion path for callers that already have text. It is -not the universal representation of GitHub, Notion, Slack, Linear, or other provider objects. +1. register an immutable Source Definition manifest; +2. read the opaque checkpoint for one Connector binding; +3. submit a worker-materialized Source observation with all declared projections; and +4. compare-and-swap the binding checkpoint from the value read at run start. -Additional integrations should define typed Source and capture models. They may reuse common snapshot persistence, -hashing, idempotency, and citation machinery while keeping provider-specific fields in their typed payload. +The observation envelope carries the Definition name, version and fingerprint, canonical Source payload, and one +value for every projection declared by the manifest. The Server validates envelope identity, payload schema, +projection-key equality, projection schemas, and standard projection invariants before durable acceptance. Provider +names, storage services, paths, credentials, or other Connector-specific configuration do not appear in this API +unless a Definition deliberately includes them in its canonical Source schema. -## Connector boundary +The Server returns a durable Source receipt before the worker may commit a checkpoint. The checkpoint operation uses +optimistic comparison so concurrent runs of the same binding cannot silently overwrite each other. Submission is +idempotent for an identical Source identity and payload; conflicting content for an accepted identity is rejected. -Connectors own discovery, provider authentication, cursors, checkpoints, polling, webhooks, retries, rate limits, and -conversion of provider responses into typed snapshots. +## Definition compatibility contract -PowerContext owns Source and snapshot identity validation, durable snapshot storage and idempotency, exact reads by -snapshot reference, scope isolation, Artifact foreign-key integrity, evidence projection, and citation validation. +The Definition name remains stable across compatible schema evolution. Each persisted observation records the +Definition version used to validate and canonicalize it. A newer Definition version must either declare how it reads +an older observation without changing its canonical meaning or coexist with a reader for the older version. -This division follows the useful part of TencentDB-Agent-Memory's ISourceFetcher and SourceFetcherRegistry: fetchers route -provider protocols and return a provider version, while the core stores metadata and serves memory operations. PowerContext -additionally retains immutable evidence because Artifact lineage needs stronger replay guarantees. +A Definition change is incompatible when it changes SourceKey equality, observation equality, canonical value bytes, +provenance meaning, or materialization guarantees for an accepted observation. Such a change requires a new +Definition version and cannot rewrite existing SourceRefs. + +A projection change is incompatible when it changes the output schema, canonical bytes, or meaning for an accepted +observation. Such a change requires a new projection version. It does not require a new Source Definition version +when the Source value and observation semantics remain unchanged. + +Renaming a Definition creates a new `source_type`. Reclassifying an existing observation under another Definition is +an explicit derivation with provenance, not an in-place migration of identity. -# TencentDB-Agent-Memory research +## Connector lifecycle contract + +A Connector owns provider interaction: discovery, credentials, filtering, checkpoints, retries, rate limits, +provider change handling, and positive deletion detection. It submits definition-native inputs against a Scope +binding and receives exact accepted SourceRefs. + +A Source Definition owns semantic normalization: logical identity, observation identity, canonical value, +provenance, materialization validity, and exact read. A Connector cannot override those rules. If the intersection of +provider capabilities, Connector behavior, and Definition requirements cannot satisfy a selected materialization, +the observation is rejected or captured under a valid mode. + +```text +provider capabilities + intersect Connector behavior + intersect Source Definition requirements + = valid Source observation +``` + +A Connector type declares a stable name and version, its configuration schema, the Source Definitions it can submit, +and the acquisition capabilities it provides. Capabilities are optional and explicit. Typical capabilities include a +complete snapshot, a change feed, checkpoint resume, and authoritative deletion events. A Connector cannot advertise +a capability that its provider and acquisition path cannot enforce. + +A Connector binding activates one Connector configuration for exactly one Scope. The binding has a stable identity +for checkpoint and provider-namespace continuity, but it does not own Sources and does not replace `scope_id` or +`source_type`. Credentials are resolved by the hosting environment and do not become Source value or provenance. + +A Connector run begins from an opaque binding checkpoint, resolves zero or more definition-native inputs inside the +worker, and submits their materialized observations. It records an outcome for every item. An accepted or +idempotently replayed observation returns its exact SourceRef. A rejected or failed item remains visible in the run +outcome and cannot be hidden by advancing the checkpoint past work that is not safely replayable. + +A run finishes as complete or incomplete. A complete snapshot may produce positive deletion evidence for previously +known provider objects that are absent. An incomplete listing, timeout, permission failure, cancellation, or lost +connection produces no absence-based deletion evidence. An authoritative provider deletion event may produce +positive deletion evidence independently of snapshot completeness when its binding and object identity are verified. + +The completed checkpoint advances only after its accepted observations and deletion evidence are durable. Retrying +from an earlier checkpoint is valid because Source observation submission is idempotent. Connector checkpoint, +health, retry, and run-status records are operational state rather than Source observations or Artifact evidence. + +Installation, discovery, activation, and execution are separate concerns. Installing a Connector package does not +activate a binding. A Connector package executes outside the PowerContext Server and uses the remote worker +ingestion contract; scheduling and process supervision belong to the deployment environment. + +## Artifact evidence and cross-Scope delivery + +An Artifact revision records exact SourceRefs used directly by its computation. Advancing a Source head does not +change existing Artifact lineage. Recalculation against a newer observation produces a new Artifact revision rather +than rewriting prior evidence. + +Sources remain in their producing Scope. A Context Reference may expand a read selection according to the Scope +organization contract, but it does not change Source ownership. Exact Artifact publication across Scopes retains the +origin Scope and exact SourceRef in lineage. Publishing an Artifact does not publish every Source in its origin Scope. + +If an application deliberately captures the same external value into another Scope, the target receives a new +Scope-owned Source observation. Its provenance may cite the origin Scoped SourceRef, but the original Source is not +moved and the two SourceKeys are not made identical. + + +## Persistence, retention, migration, and operation freezing + +The persistence model MUST represent Scope-owned Source history separately from the mutable current-head selection. +Each accepted observation receives one immutable SourceRef. Head advancement, head deletion, and locator changes MUST +not delete or rewrite observations. Artifact, Candidate, and Handoff lineage MUST retain the exact SourceRefs used by +their computation. + +An observation referenced by a durable Artifact, Candidate, or Handoff MUST be protected from ordinary garbage +collection. Head deletion MUST NOT delete historical observations. Legal or user-requested hard deletion is a separate +audited operation; if it removes cited evidence, the resulting lineage break MUST be explicit and observable. + +An operation that consumes `latest` MUST resolve and freeze exact SourceRefs (or an equivalent Source high-watermark) +at operation start. Later head advancement MUST NOT change the operation's inputs, and its output lineage MUST record +the observations actually used. Exact reads MUST distinguish missing history, unavailable referenced content, definition +incompatibility, and a deleted current head; none may be silently replaced by a newer observation or current provider value. + +Migration from the current two-part Source rows MUST be one-to-one and idempotent: each legacy row receives a stable +SourceKey/head and deterministic observation identity, and existing Artifact, Candidate, and Handoff references continue +to resolve to that same payload. Legacy two-part reads remain an explicit compatibility path and MUST NOT resolve to a +newer head. Old and new workers may coexist only while this rule is enforced. An interrupted migration MUST resume +without duplicating observations or changing payload-conflict semantics; rollback MUST NOT rewrite accepted evidence. + +## Conformance + +A Source Definition can be supported only after its mandatory contract passes conformance scenarios for: + +- identity normalization and collision rejection; +- identical observation replay; +- conflicting payload rejection for one observation ID; +- several immutable observations under one SourceKey; +- exact old-observation reads after head advancement and deletion; +- digest verification for captured and referenced values; +- referenced-value unavailability and mutation; +- Scope isolation and explicit owner preservation; +- Definition version compatibility and unavailable-definition behavior; and +- explicit registration conflict handling. + +A named projection can be advertised only after conformance verifies deterministic output for exact observations, +schema and version conflict handling, exact SourceRef lineage, and explicit failure when the capability is absent. + +A Connector capability can be advertised only after conformance verifies checkpoint replay, per-item outcome +visibility, durable checkpoint ordering, complete-versus-incomplete run behavior, and the claimed deletion evidence. +Provider-specific behavior is established by its implementation evidence rather than generalized into the standard +contract. + +A remote worker path additionally verifies manifest fingerprint and conflict handling, rejection of unregistered or +schema-invalid observations, exact projection-set validation, durable receipt ordering, and stale checkpoint CAS +rejection across a Server restart. + +# Drawbacks + +- Separating SourceKey, SourceRef, Source head, and Definition version introduces more concepts than one immutable + `(source_type, source_id)` pair. +- Exact SourceRefs retain owner Scope and observation identity, increasing lineage payload size. +- Definition authors must specify canonicalization, provenance, and compatibility instead of relying on arbitrary + metadata. +- Named projections and Connector lifecycle state add contracts that must evolve independently from Source values. +- Referenced Sources are unavailable for providers that expose only current values, so some integrations must retain + captured data. +- Explicit registration requires deployment coordination before a custom Source observation can be accepted. + +# Rationale and alternatives + +## Extend ContentSource into the general integration model + +Adding provider fields to ContentSource would preserve the `POST /v1/sources/content` capture API, but it would keep logical identity, +observation identity, and provenance inside caller conventions. Different integrations would encode incompatible +schemas in metadata, and non-text Source values would still need another model. ContentSource remains a useful +single-observation implementation instead. + +## Use one opaque Source envelope + +A universal JSON payload would make persistence and transport uniform, but would move schema validation and +compatibility into runtime conventions. Definition-owned typed values and provenance make the extension boundary +reviewable and allow consumers to reject unsupported Source types before interpretation. + +## Put an observation digest inside source_id + +An integration can preserve the two-part SourceRef shape by composing logical identity and digest into `source_id`. This +makes immutable capture possible but hides the continuing logical Source from the catalog. Updates, current-head +selection, deletion, and provider identity then become integration-private conventions. The standard model represents +both identities directly. + +## Make SourceRef logical and add a separate ObservationRef + +Two public reference types would make SourceRef logical, but Artifact evidence would need +to reject SourceRef and accept only ObservationRef. Defining SourceRef itself as exact follows the existing ArtifactRef +principle that durable lineage references immutable state. + +## Add hybrid materialization + +A third mode that sometimes reads externally and sometimes falls back to captured data obscures which value is +authoritative and which failures are visible. A captured observation can retain a complete external reference as +provenance. A referenced observation either resolves exactly or fails. + +## Let Parent or Connector identity own Sources + +Scope Parent is organization, and Connector identity is acquisition provenance. Neither is a durable ownership +boundary. Using either would conflict with the Scope organization contract and would make reorganization or +Connector replacement change Source identity. + +# Prior art + +- The [Scope organization and Agent integration design](https://github.com/oceanbase/powercontext/pull/1345) separates + Scope ownership, read sharing, organization, delivery, and observation. This RFC applies the same separation to + Source ownership, identity, exact evidence, and acquisition. +- [Apache OpenDAL OFS RFC-0016](https://github.com/apache/opendal-ofs/blob/main/rfcs/0016_filesystem_architecture.md) + separates namespace authority from access frontends and forbids a frontend from advertising guarantees that the + underlying layers cannot enforce. Source materialization follows the same authority rule. +- [opendalfs](https://github.com/fsspec/opendalfs) exposes OpenDAL services through the fsspec interface and + demonstrates backend-neutral filesystem acquisition. Its paths and file metadata do not define Source identity or + immutable revision semantics. A backend read can satisfy referenced materialization only when the complete stack + addresses and verifies an immutable revision. +- DataHub stateful ingestion separates connector checkpoints and stale-entity detection from emitted metadata + identity. Airbyte treats connector state as an opaque recovery boundary rather than record identity. +- OpenMetadata separates the Source that emits records from connection checks, workflow status, and the sink. +- Nowledge Mem's TiddlyWiki importer uses stable logical IDs, canonical payload digests, source revalidation, and + per-item outcomes. Those behaviors inform the separation between Source observations and Connector run state. + + +## TencentDB-Agent-Memory research (research input) + +This section is non-normative research input carried forward from #1388; it does not add a second Source contract. This RFC is informed by [TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory): @@ -156,69 +577,30 @@ References: - [Knowledge metadata schema](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/db/client.ts) - [CodeGraph service](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/store/code-graph-service.ts) -# Persistence and Runtime impact - -The implementation is expected to introduce a mutable Source/current-head representation alongside an immutable snapshot -table. Artifact lineage should reference the immutable snapshot key rather than only source_type/source_id. The exact -migration shape is left to implementation, but it MUST preserve old captured Sources and reject payload replacement under -an existing snapshot identity. - -A new typed Source must register with Source resolution and exact reads, persistence encoding and decoding, Runtime -composition where applicable, HTTP/client mapping if exposed, evidence projection, citation validation, and focused -persistence and end-to-end tests. Evidence projectors SHOULD use adapter capabilities or a registry instead of accumulating -ContentSource-specific branches in Runtime composition. - -Existing ContentSource captures remain readable. Existing POST /v1/sources/content remains idempotent for its current -identity/payload contract. Legacy two-part references require an explicit migration rule and MUST NOT silently mean the -latest snapshot when that could change historical Artifact lineage. OpenAPI changes follow agreement on this identity -and legacy behavior. - -# Alternatives considered - -## Keep the current stable Source key - -This keeps the smallest API, but provider changes remain conflicts and multiple snapshots cannot be cited. It is suitable -only for already-immutable captures. - -## Use only provider references - -This minimizes storage, but exact evidence depends on provider retention, permissions, availability, and historical-read -semantics. It cannot satisfy replay for arbitrary integrations. - -## Make every provider value a ContentSource - -This reduces public types but moves provider semantics into untyped metadata, weakens validation, and makes evidence -projection provider-blind. It is not a durable extension boundary. - -## Store only a local revision - -A local revision helps ordering and cursors but does not prove which provider state was observed. It complements, rather -than replaces, provider revision and content hash. - -# Rollout and validation +# Unresolved questions -The first bounded validation should cover one mutable provider object and one immutable provider revision. A GitHub -issue/commit or Git repository commit is a practical candidate. It should demonstrate: +- Must every durable SourceRef carry `scope_id` directly, or may a canonical scoped envelope contain a local exact + SourceRef while preserving the same fully qualified identity? +- Which Source Definition versions must a Runtime retain simultaneously before a Definition can be considered + supported? +- Should Source head deletion be one common catalog state, or should the standard contract expose only an + active exact head and leave deletion entirely to Connector state? +- Which projection names and schemas have enough implementation evidence to become shared standards rather than + namespaced capabilities? -1. repeated delivery of one provider revision is idempotent; -2. a later provider revision creates a new snapshot and advances the current head without replacing the old snapshot; -3. current search and ingestion use the new snapshot; -4. an Artifact still cites and reads the old snapshot after the provider changes; -5. a missing or unverifiable Ref is rejected rather than silently refreshed; -6. evidence projection and HTTP/client mapping preserve the exact snapshot reference. +# Future possibilities -# Open questions +Explicit plugin discovery and deployment policy may build on Definition and Connector registration without making +package installation equivalent to activation. -- Should snapshot_id be opaque and generated by PowerContext, content-addressed, or expose both forms? -- Should SourceRef grow a snapshot_id, or should a separate SnapshotRef be introduced? -- Which provider revision guarantees qualify for Ref-only materialization? -- What retention and garbage-collection policy applies to snapshots no longer referenced by an Artifact? -- Should large captures use an external blob store while retaining a canonical hash and durable locator? -- Should logical Source heads be public API, or remain connector/persistence metadata initially? +Retention policies may define when unreferenced observations are reclaimed and how unavailable exact evidence is +reported; they must preserve the minimum lineage protection above. Legal or user-requested deletion can be specified as +a separate audited workflow. Source head deletion alone does not authorize evidence removal. # Decision requested -Approve the mutable Source head plus immutable snapshot model, snapshots as the exact-evidence boundary, Capture as the -default materialization, Hybrid as the preferred traceable form, and ContentSource as one concrete capture type rather -than a universal provider model. After approval, implementation can define the concrete schema, migration, OpenAPI -fields, retention rules, and one bounded connector validation. +Approve this Source Definition and observation contract as the baseline for #1388, with the TencentDB-Agent-Memory +research retained as prior art. In particular, approve the SourceKey/SourceRef separation, immutable observations +and exact materialization, named projections, Connector lifecycle boundary, and the persistence, retention, +operation-freezing, and migration invariants above. Concrete schema and rollout work may proceed only if it preserves +these invariants. diff --git a/docs/zh/rfcs/1388_source_integration_shape.md b/docs/zh/rfcs/1388_source_integration_shape.md index 18bfa7acc..2ff39091b 100644 --- a/docs/zh/rfcs/1388_source_integration_shape.md +++ b/docs/zh/rfcs/1388_source_integration_shape.md @@ -1,123 +1,519 @@ -# Source 集成形态 - -- 提案名称：source_integration_shape -- 状态：Proposed -- 开始日期：2026-08-25 +- Proposal Name: `source_integration_shape` +- Status: Proposed +- Start Date: 2026-08-25 - RFC PR：[oceanbase/powercontext#1388](https://github.com/oceanbase/powercontext/pull/1388) - Tracking Issue：[oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240) -- 相关 RFC：[RFC 0019](0019_local_source_memory_runtime.md)、[RFC 0020](0020_runtime_backed_memory_remote_access.md)、[RFC 0051](0051_experience_skill_artifact_families.md) +- Related Discussion: [oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240), + [oceanbase/powercontext#1363](https://github.com/oceanbase/powercontext/issues/1363) +- Related Design: [oceanbase/powercontext#1345](https://github.com/oceanbase/powercontext/pull/1345) +- Related RFCs: [RFC 0002](0002_core_sdk_product_model.md)、[RFC 0014](0014_memory_layer_design.md)、 + [RFC 0019](0019_local_source_memory_runtime.md)、[RFC 0048](0048_handoff_artifact.md) + +# Summary + +本 RFC 定义标准 Source 模型，以及新增 Source 类型时必须遵守的契约。 + +每个 Source 只属于一个 Scope。在该 Scope 内，`SourceKey` 标识一个逻辑 Source，`SourceRef` 标识这个 +Source 的一次不可变观察。推进当前观察、观察到删除、修改外部 locator 或断开 Connector，都不会改变 +已经接受的观察，也不会将其移动到另一个 Scope。 + +Source Definition 为一个稳定的 Source 类型定义 value schema、provenance schema、身份规则、观察规则、 +materialization 契约、canonicalization 与兼容策略。Definition 显式注册，并在组合完成的 Runtime 生命周期内 +保持不变。持久化、传输与 Artifact consumer 按稳定的 Definition 名称和版本路由，而不是按具体 Python 类路由。 + +Definition 可以为无法理解 native value 的 consumer 声明 named projection capability。每个 projection 拥有独立 +版本的 schema，并对一个精确 observation 具有确定语义。Consumer 按 capability name 与 version 选择 projection， +而不是检查具体 Source class。 + +Connector lifecycle 将 provider acquisition 绑定到 Scope，在 worker 内解析 definition-native input，提交 +materialized observation，记录 per-item outcome，并且只在接受的 observation 已持久化后推进 opaque checkpoint。 +Connector run 区分 complete discovery 与 incomplete discovery，避免把缺失对象静默转换为删除。 + +Materialization 表达解析某个精确观察时所依赖的权威来源。Captured observation 从 PowerContext 保留的 +canonical value 解析；referenced observation 从外部不可变 revision 解析。仅有外部 locator、修改时间、 +ETag 或 provider 当前值读取，并不能满足 referenced 契约。 + +`ContentSource` 继续作为简单的 captured-text Source。调用方提供稳定身份，加上 immutable-payload 冲突规则， +适合一次性内容捕获，但它不是通用的外部集成模型。 + +本 RFC 定义 Source、projection、Connector lifecycle，以及 worker 与 PowerContext Server 之间的远程摄取边界。 +它不定义插件发现、scheduler、credential transport、具体 Source family 或 Connector 实现。 + +# Motivation + +`ContentSource` 与 `POST /v1/sources/content` 提供 captured-text ingestion。调用方选择一个 +`source_id`；使用完全相同的 payload 重放具有幂等性，而用不同 payload 复用该身份会产生冲突。只有调用方把 +这个身份当作不可变身份时，它才能表达精确证据。 + +外部系统通常具有不同的生命周期。Wiki 页面、issue、object、message 或 file 拥有一个逻辑身份，但会随时间 +产生多个值。外部对象可能被重命名、修订、删除、恢复或暂时无法读取。使用过旧值的 Artifact 必须继续引用当时的 +精确证据。二元 `(source_type, source_id)` Source reference 无法同时表达稳定的逻辑对象和不可变观察，只能迫使每个集成自行发明复合 +`source_id`。 + +扩展边界也不完整。Source adapter 将 native input class 绑定到具体 Source class 和读取结果，而内置 +Runtime 与关系型持久化会组装固定 adapter 集合。它没有说明独立定义的 Source 类型在身份、持久化、传输与 +Artifact evidence 上必须长期满足哪些规则。 + +标准模型必须回答六个问题，且不能把它们压进同一个 identifier： + +1. 哪个 Scope 拥有这份证据？ +2. 它描述哪个逻辑上的外部或内部 Source？ +3. Artifact 使用的是哪个精确观察值？ +4. PowerContext 从哪里读取该精确值？ +5. 哪个 Definition 赋予 value 与 provenance 语义？ +6. Consumer 可以使用哪个 declared view，而不必理解 native value？ + +Connector concerns 与此相邻但不同。Discovery、credentials、filtering、checkpoints、retries、provider +change handling 与 deletion detection 决定提交哪些观察；它们不定义 Source identity，不能削弱精确证据， +也不能改变 Scope ownership。 + +# Guide-level explanation + +## Domain model + +理解该模型时，依次确定 ownership、logical identity、exact observation、materialization authority 与 type +semantics： + +| Concept | Representation | Question answered | +| --- | --- | --- | +| Ownership | Scope | Source 属于哪里？ | +| Logical identity | `SourceKey` | 这是哪个持续存在的 Source？ | +| Exact evidence | `SourceRef` | 引用的是哪个不可变观察？ | +| Read authority | materialization | 从哪里解析该精确值？ | +| Type semantics | Source Definition | 如何解释 value、provenance 与 identity？ | +| Consumer view | named projection | Consumer 可以使用哪个 declared representation？ | +| Acquisition | Connector or direct caller | 如何发现并提交新观察？ | + +这些职责形成单向依赖： + +```text +Connector or direct caller + | + v +Source Definition + | + v +Scope-owned Source history + | + +---- mutable head selection + | + `---- exact SourceRef ----> Artifact evidence +``` + +一个 Connector 可以使用一个 Source Definition，多个 Connector 可以共用同一个 Definition，直接调用方也可以 +在没有 Connector 的情况下提交 Source。因此 Connector identity 不会成为 Source type identity。 + +## Scope ownership + +每个 SourceKey 与 observation 都只属于一个 Scope。Scope ownership 不从外部 workspace、path、repository、 +provider account、Connector instance 或 Source locator 推导。这些值可以参与 binding 或 provenance，但不能 +分配或替代 `scope_id`。 + +完整限定的逻辑身份为： + +```text +SourceKey = (scope_id, source_type, source_id) +``` + +完整限定的精确身份为： + +```text +SourceRef = (scope_id, source_type, source_id, observation_id) +``` + +Scope-bound operation 可以从固定 request binding 获得 `scope_id`，而不把它作为任意参数接收。持久化的解析结果 +仍然保留 owner Scope，使证据在 publication、reporting 或 export 后仍无歧义。 + +修改 Scope Parent、Context References、Agent binding 或 observation selection，都不会改变 SourceKey 或 +SourceRef。跨 Scope 发布 Artifact 时，provenance 保留原始 Scope 与精确 SourceRef；不会移动或隐式复制 +Source history。 + +## Logical Source and immutable observation + +`source_id` 在一个 `(scope_id, source_type)` namespace 内命名逻辑 Source，其含义由 Source Definition +规定。它可以对应 provider object ID、稳定 import identity 或其他 normalized key。观察到新值时，它不能静默改变。 + +`observation_id` 在一个 SourceKey 下命名一次不可变观察。对通用 PowerContext component 而言它是不透明的； +可以派生自 provider revision、canonical value digest 或 Definition 特有组合。它不隐含整数序列、时间顺序或祖先关系。 + +适用以下不变量： + +- 一个 `(SourceKey, observation_id)` 永远标识同一个 canonical observation； +- 再次观察相同 canonical observation 具有幂等性； +- 不同 canonical observation 不能复用 observation ID； +- 如果 identity-bearing provenance 不同，同一个 SourceKey 下可以有 value digest 相同的多个 observation； +- value digest 相同的 observation 不会自动成为同一个逻辑 Source； +- Artifact 只引用精确 SourceRef，绝不引用移动的 SourceKey 或 `latest` observation。 + +例如，更新一个逻辑 Source 会保留其 SourceKey，并产生新的 SourceRef： + +```text +SourceKey(scope-a, record, provider-object-42) +|-- SourceRef(..., observation-1) "Initial value" +`-- SourceRef(..., observation-2) "Revised value" +``` + +即使 `observation-2` 已成为 current，派生自 `observation-1` 的 Artifact 仍然引用后者。 + +## Source Definition + +Source Definition 是一个 `source_type` 的持久语义契约。它声明： + +- 稳定的 Definition name 与 version； +- Source value 与 typed provenance 的结构； +- Source ID normalization 与 equality； +- observation ID normalization 与 equality； +- identity-bearing fields 与 non-identifying annotations； +- canonical bytes 与 value digest algorithm； +- 支持的 materialization modes 与 exact-read requirements； +- limits 与 validation failures； +- older Definition versions 的 compatibility rules。 + +Definition 将 definition-native input 解析为 canonical observation，并从精确的 persisted observation 读取 +Definition 拥有的 value。解析不会选择 Scope、修改 catalog、推进 head 或发现外部 object；读取不会解析 +`latest`，也不会替换为另一个 observation。 + +Definition 必须显式且类型化。新的集成不能通过在 `ContentSource.metadata` 中放置未声明 schema 来模拟新 +Source 类型。Provider-specific provenance 可以扩展 Definition 声明的 schema，但影响 identity、exactness +或 compatibility 的字段必须由 Definition 命名。 + +## Named projection capabilities + +Named projection 是一个 exact observation 的可选 Definition-owned view。它让 Artifact family 或其他 consumer +无需理解 native Source value 或具体 Python class，就能使用声明过的 representation。 + +Projection 通过稳定的 name 与 version 选择。其 Definition 声明 output schema、canonicalization、digest rules +与 failures。Projection 针对精确 SourceRef 求值，不能解析 head、`latest` 或 provider current value。对于相同的 +Definition version、projection version 与 exact observation，它必须返回相同的 canonical result。 + +Projection capability 必须显式声明。需要某个 projection 的 consumer 会拒绝未声明兼容 capability 的 Source， +而不会从 metadata 推断 content，也不会回退到形态相似的 Source class。Projection 可以作为 derivative 被缓存或 +持久化，但其 authority 仍是 exact Source observation，lineage 保留对应 SourceRef。 + +本契约不规定标准 projection name 或 payload schema 的目录。只有当多个 Definition 与 consumer 的互操作证明其 +语义稳定后，projection 才成为 shared standard。在此之前，Definition 可以暴露 namespaced projection，但不会让 +它成为其他 Source type 的 mandatory capability。 + +## Materialization authority + +Materialization 回答精确 SourceRef 的返回值来自哪里： + +| Materialization | Authority | Required guarantee | +| --- | --- | --- | +| `captured` | PowerContext 保留的 canonical value | 保留值与 observation digest 一致 | +| `referenced` | Immutable external revision | 重读 reference 得到相同 canonical value 与 digest | + +Captured Source 可以把 external locator、provider revision 与 digest 保留为 provenance。因为读取权威仍是 +保留值，所以它依然是 captured。这覆盖了 hybrid design 中有价值的部分，而不引入 fallback 语义含糊的第三种模式。 + +只有当外部系统及其 reader 能够寻址不可变历史值时，Definition 才能使用 referenced materialization。读取 +path、page ID、issue ID 或 URL 的当前值并不足够。Modification time 与 ETag 可以参与 provenance 或 conflict +detection，但 Definition 必须说明 provider 是否保证它们指向不可变值。 + +Referenced value 不可用或 digest 不同时，精确解析失败。PowerContext 不返回 provider 当前值、stale cache +entry 或其他 observation。不能满足该规则的 provider 必须使用 captured materialization，或者拒绝该 observation。 + +## Current head and deletion + +Source history 不可变；current head 是可变的 catalog selection。Head 可以选择一个精确 SourceRef，或记录已 +明确观察到逻辑 Source 被删除。Head 可用于 current-state query 与后续 acquisition，但它不是 evidence，不能 +出现在 Artifact citation 中。 + +推进或删除 head 不改变任何 observation。Timeout、permission failure、incomplete listing、Connector +unavailable 或 disconnect 都不是明确的 deletion evidence，不能改变 head。只有当 deletion 本身是有意义的 +Source evidence 时，Source Definition 才可以定义 tombstone value；通用 head deletion 不会伪造这种值。 -# 摘要 +## ContentSource -PowerContext 需要为会随时间变化的外部系统定义 Source contract。GitHub issue、Notion page、Slack message 或 -Linear issue 具有稳定的 provider identity，但当前值允许变化；已经被 Artifact 使用的原文必须保持精确。本 RFC -因此将可变的 Source head 与某个时间点观察到的不可变 snapshot 分开。 +`ContentSource` 继续作为 RFC 0019 定义的 neutral captured-text path。调用方选择一个只能与一个 canonical +payload 一起提交的身份。Persistence conflict rule 使接受后的 ContentSource 可作为精确证据，但它不提供独立的 +logical Source lifecycle。 -建议的模型分为两层： +标准模型把它视为有效的 single-observation Source： - Source：标识外部对象，并指向最新 snapshot。 - Source Snapshot：保存某个 provider revision 和捕获时间下的确切值。 +- 现有 identity 保持不可变； +- 相同内容重放继续保持幂等； +- 同一 identity 下的不同内容继续发生冲突； +- 解析 ContentSource 的 reference 保持精确且不变； +- 不从 metadata 推导 mutable head 或 multi-observation behavior。 -Artifact、Candidate、Handoff 以及其他 exact-evidence 消费者必须引用不可变 Source Snapshot。只有 provider URL 或 -逻辑对象 ID 不是充分证据，除非 provider 保证该 revision 不可变且可重新读取。 +ContentSource 适合 prompt、显式文本捕获、import record，以及调用方已经拥有不可变身份的其他场景。持续观察同一 +逻辑对象的集成应定义或复用 multi-observation Source type。 -默认读取路径可以使用最新 snapshot 进行检索和 ingest；历史 Artifact lineage 必须继续使用 Artifact 生成时采用的 -snapshot。本 RFC 只定义 PowerContext 的边界；Connector 继续负责 discovery、同步、checkpoint、重试、凭证和 -provider-specific change handling。 +# Reference-level explanation -# 动机 +## Source identity contract -当前 Source 实现用同一个 identity 同时表示逻辑对象和捕获值。关系数据库主键实际上是 -(scope_id, source_type, source_id)，同一 key 写入第二个 payload 会冲突。这对不可变 capture 是正确的，但无法表示 -可变 Source head 以及保留的历史 snapshot。 +`scope_id` 是 Scope organization design 定义的 ownership boundary。`source_type` 是稳定的 Source Definition +name。`source_id` 是非空的 normalized identifier，其 equality 与 bounds 由 Definition 声明。 -当前 SourceRef 也只有 source_type 和 source_id。因此 Artifact lineage 只能标识逻辑 key，不能标识生成 Artifact 时 -实际使用的 provider revision 或 payload。provider 后续更新后，同一个引用可能解析到不同内容，或者无法再解析。 +Source identity 以 Scope 为本地边界。两个 Scope 可以包含等价的外部材料，但不共享 ownership 或 identity。 +需要避免碰撞时，Definition 可以在 `source_id` 规则中包含稳定的 external instance 或 connection discriminator， +但 discriminator 不替代 `scope_id`。 -设计必须保持：最新 Source 可以变化并重新 ingest 而不改写历史证据；被 Artifact 引用的 snapshot 在 provider 变化或 -不可用后仍可读取；同一个 snapshot 的重复投递幂等；connector 可以 -使用 Git commit SHA、Notion edit marker、Slack message timestamp 或文件 SHA-256；现有 captured text 仍然有用； -typed Source 可以接入 persistence、Runtime、transport 和 evidence projection；connector 同步逻辑不进入 Source model。 +Rename 行为由 Definition 决定。Provider object ID 可以在 locator 变化时保留 SourceKey；path-derived identity +通常把 rename 视为一次逻辑 deletion 与一次 creation。当 provider 与 acquisition path 无法证明 rename-stable +identity 时，Definition 不能宣称支持它。 -# 设计说明 +## Observation contract -## Source head 与 snapshot +Observation 包含以下标准字段： -公共模型应区分可变 Source head 与不可变 snapshot 引用。名称仅为示意： +```text +SourceObservation +|-- source_key +|-- observation_id +|-- definition_version +|-- materialization +|-- value_digest +|-- provenance +`-- definition-owned value or exact external reference +``` - { - "source_type": "github-issue", - "source_id": "oceanbase/powercontext#1240", - "snapshot_id": "snap_01J..." - } +`value_digest` 对 Definition 声明的 canonical bytes 使用 SHA-256，并编码为 `sha256:`。对结构化 +value，Definition 指定 deterministic canonicalization。Digest 用于验证 value equality，不替代 SourceKey +或 observation identity。 -可变 Source head 至少包含： +Canonical observation 包含所有被 Definition 认定会影响 identity 或 exact meaning 的字段。Retry count、 +last scan time 或 processing status 等 operational facts 不是 Source value，不改变 observation identity。 +如果 timestamp 或 provider attribute 会影响 provenance meaning，Definition 必须显式分类并 canonicalize。 -- source_type、source_id：稳定逻辑身份； -- latest_snapshot_id：当前读取和 ingest 使用的 snapshot； -- connector 所需的 locator 与 provider metadata。 +## Source reference contract -Snapshot 至少包含： +SourceRef 标识精确 observation，并包含 owner Scope。它不接受缺失 observation ID、`latest`、head version 或 +current provider locator。 -- snapshot_id：PowerContext 为不可变 snapshot 生成的身份； -- source_type、source_id：稳定逻辑对象身份； -- provider_revision：provider-native revision（如果有）； -- materialization：captured 或 referenced； -- content_hash：canonical observed value 的 hash； -- captured_at：PowerContext 捕获时间； -- payload：materialized 时的 canonical captured value； -- locator：需要时保存 provider URL 或 provider-specific locator。 +在 scope-bound operation 内，只有当 current Scope 固定且解析出的 durable value 会恢复 `scope_id` 时，紧凑的 +local representation 才可以省略重复的 `scope_id`。跨越 Scope boundary、离开 Runtime 或进入 durable +cross-Scope provenance 的 reference 必须显式携带 owner Scope。 -snapshot_id 必须只对应一份不可变 payload。相同 snapshot identity 写入不同 payload 必须冲突。provider revision -变化时生成新 snapshot 并推进 latest_snapshot_id；不得更新已经被 Artifact 引用的 snapshot payload。provider -provenance 等价时，相同 canonical payload 可以复用 snapshot。 +Reference resolution 会验证全部四个 identity components，以及 stored observation 的 Definition version 与 +digest。无法解析精确 observation，不等同于 logical Source 已删除、head 已推进或 Connector 不可用。 -SnapshotRef 是 exact-evidence 的 citation boundary。迁移期间，缺少 snapshot_id 的旧 SourceRef 只能解析到现有旧行 -代表的那份不可变 payload。新创建的 exact evidence 必须带 snapshot_id；如果会改变历史 Artifact lineage 的含义， -两段式引用不得静默解析为 latest snapshot。 +## Definition registration contract -## Capture、Ref 与 Hybrid +Executable Definition 属于 worker。Worker 用它解析 definition-native input、canonicalize Source value，并计算 +named projection。Server 不导入 Connector 或 Definition package，也不执行其中的 Python class。 -进入 Artifact lineage 的证据默认使用 Capture。它将 canonical value 和 hash 作为 snapshot 保存在本地，因此 -provider 可用性和未来编辑不会改变历史证据。Connector 可以捕获每次变化，也可以只把进入 PowerContext ingestion -boundary 的变化保存下来；一旦 snapshot 被 Artifact 引用，就必须按 retention policy 保留。 +提交 observation 前，worker 注册不可变的声明式 manifest。Manifest 包含稳定的 Definition name 与 version、 +canonical Source JSON Schema、每个 projection key 与 output JSON Schema，以及覆盖完整声明的 fingerprint。 +Fingerprint 是 RFC 8785 canonical JSON 的 SHA-256。相同 manifest 的注册是幂等的；同一个 +`(source_type, definition_version)` 对应不同声明时必须拒绝。 -只有在 provider 明确保证 revision 不可变且可重新读取时，才允许使用 Ref。可变 URL、对象 ID 或当前 updated_at 不够。 -如果 Ref 无法按记录的 revision 解析，snapshot 必须标记为 unavailable，不能静默替换为 provider 当前值。变得 -unavailable 的 Ref 不能满足 Artifact 的 exact-evidence 要求，除非先物化为保留的 snapshot。 +Server 验证 manifest schema，以及其识别为 shared standard 的 named projection。Manifest 不传输可执行的 identity +rule、canonicalization code、read behavior、credential 或 provider configuration；这些仍由 worker 持有。 +注册后的 manifest 足以让 Server 在不加载 plugin code 的情况下验证并保存 opaque canonical observation。 -Hybrid 在 snapshot 中同时保存 provider locator/revision 和 canonical captured value。当需要外部追溯、provider -读取成本较高，或 connector 需要协调后续变更时，优先使用 Hybrid。 +Definition discovery 与 registration 相互独立。Package entry point 或其他 discovery mechanism 可以报告 +可用 Definition，但安装不意味着激活。本 RFC 不选择 entry points、central settings format、pluggy 或 +Connector marketplace。 -流程为： +## Remote worker ingestion contract - connector 发现 source - -> 内容变化时生成带 provider revision 和 hash 的新 snapshot - -> 推进 Source.latest_snapshot_id - -> Artifact lineage 引用生成时采用的 snapshot +Connector 在独立 worker 进程中运行。Worker 拥有 provider access 与所有 executable Definition behavior；Server +拥有 durable Source history、Artifact consumption 与 checkpoint comparison。双方的数据面交互只有四个通用操作： -## ContentSource 与 content API +1. 注册不可变的 Source Definition manifest； +2. 读取一个 Connector binding 的 opaque checkpoint； +3. 提交 worker 已物化的 Source observation 及其全部声明 projection； +4. 从 run 开始时读到的值 compare-and-swap binding checkpoint。 -ContentSource 保持为一个具体的内置 captured-text Source。它当前实际上是每个 source ID 只有一份不可变 snapshot；未来 -同一逻辑 ID 收到新内容时应允许生成新 snapshot。POST /v1/sources/content 继续作为已有调用方的兼容入口和 -最小 ingestion path，不升级为 GitHub、Notion、Slack、Linear 等 provider 对象的统一表示。 +Observation envelope 携带 Definition name、version 与 fingerprint、canonical Source payload，以及 manifest 声明的 +每个 projection value。Server 在 durable acceptance 前验证 envelope identity、payload schema、projection key +集合相等、projection schema 与标准 projection invariant。Provider name、storage service、path、credential 或其他 +Connector-specific configuration 不出现在该 API 中；只有 Definition 刻意将其声明为 canonical Source schema 的 +一部分时才例外。 -新增集成应定义 typed Source 和 capture model。它们可以复用通用 snapshot persistence、hash、幂等和 citation 机制， -但 provider-specific 字段应放在 typed payload 中。 +Server 必须先返回 durable Source receipt，worker 才能提交 checkpoint。Checkpoint operation 使用 optimistic +comparison，防止同一 binding 的并发 run 静默覆盖。相同 Source identity 与 payload 的提交是幂等的；已接受 identity +对应不同内容时必须拒绝。 -## Connector 边界 +## Definition compatibility contract -Connector 负责 discovery、provider authentication、cursor、checkpoint、polling、webhook、retry、rate limit，以及将 -provider response 转换为 typed snapshot。 +Definition name 在兼容 schema 演进中保持稳定。每个 persisted observation 记录验证和 canonicalize 它时使用的 +Definition version。新的 Definition version 必须声明如何在不改变 canonical meaning 的前提下读取旧 observation， +或与旧版本 reader 共存。 -PowerContext 负责校验 Source 和 snapshot identity、durable snapshot storage 与幂等、按 snapshot reference -精确读取、scope isolation、Artifact foreign-key integrity、evidence projection 和 citation validation。 +如果 Definition change 会改变已接受 observation 的 SourceKey equality、observation equality、canonical value +bytes、provenance meaning 或 materialization guarantee，它就是不兼容变更。此类变更需要新的 Definition version， +且不能重写已有 SourceRef。 -这部分借鉴 TencentDB-Agent-Memory 的 ISourceFetcher 和 SourceFetcherRegistry：fetcher 路由 provider protocol 并返回 -provider version，core 保存 metadata 并提供 memory operation。PowerContext 还需要额外保留不可变 evidence，因为 -Artifact lineage 需要更强的 replay guarantee。 +如果 projection change 会改变已接受 observation 的 output schema、canonical bytes 或 meaning，它就是不兼容 +变更，需要新的 projection version。如果 Source value 与 observation semantics 保持不变，则不要求新的 Source +Definition version。 -# TencentDB-Agent-Memory 调研参考 +重命名 Definition 会产生新的 `source_type`。把已有 observation 重新分类到另一个 Definition 是带 provenance +的显式 derivation，不是 identity 的原地 migration。 + +## Connector lifecycle contract + +Connector 负责 provider interaction：discovery、credentials、filtering、checkpoints、retries、rate limits、 +provider change handling 与 positive deletion detection。它依据 Scope binding 提交 definition-native input， +并接收接受后的精确 SourceRef。 + +Source Definition 负责 semantic normalization：logical identity、observation identity、canonical value、 +provenance、materialization validity 与 exact read。Connector 不能覆盖这些规则。如果 provider capabilities、 +Connector behavior 与 Definition requirements 的交集无法满足选定 materialization，则拒绝 observation，或在 +合法模式下 captured。 + +```text +provider capabilities + intersect Connector behavior + intersect Source Definition requirements + = valid Source observation +``` + +Connector type 声明稳定的 name 与 version、configuration schema、可提交的 Source Definition，以及它提供的 +acquisition capability。Capability 是可选且显式的，通常包括 complete snapshot、change feed、checkpoint resume +和 authoritative deletion event。Connector 不能声明 provider 与 acquisition path 无法兑现的 capability。 + +Connector binding 为一个 Scope 激活一份 Connector configuration。Binding 拥有用于 checkpoint 与 provider +namespace continuity 的稳定 identity，但不拥有 Source，也不替代 `scope_id` 或 `source_type`。Credential 由 +hosting environment 解析，不会成为 Source value 或 provenance。 + +Connector run 从 opaque binding checkpoint 开始，在 worker 内解析零个或多个 definition-native input，再提交其 +materialized observation，并记录每个 item 的 outcome。Accepted 或 idempotently replayed observation 返回精确 +SourceRef。Rejected 或 failed item 会保留在 run outcome 中；如果尚不能安全重放，checkpoint 不能越过这些工作。 + +Run 以 complete 或 incomplete 结束。Complete snapshot 可以为之前已知但本次缺失的 provider object 产生 positive +deletion evidence。Incomplete listing、timeout、permission failure、cancellation 或 lost connection 不会产生 +absence-based deletion evidence。当 binding 与 object identity 均已验证时，authoritative provider deletion event +可以独立于 snapshot completeness 产生 positive deletion evidence。 + +Completed checkpoint 只有在 accepted observation 与 deletion evidence 均已持久化后才能推进。由于 Source +observation submission 具有幂等性，从更早 checkpoint 重试是合法行为。Connector checkpoint、health、retry 与 +run-status record 是 operational state，而不是 Source observation 或 Artifact evidence。 + +Installation、discovery、activation 与 execution 相互独立。安装 Connector package 不会激活 binding。Connector +package 在 PowerContext Server 之外执行，并使用 remote worker ingestion contract；调度与进程监管属于部署环境。 + +## Artifact evidence and cross-Scope delivery + +Artifact revision 记录其计算直接使用的精确 SourceRef。推进 Source head 不改变现有 Artifact lineage。针对较新 +observation 的重新计算会产生新的 Artifact revision，而不是重写旧 evidence。 + +Source 保留在 producing Scope。Context Reference 可以按照 Scope organization contract 扩展 read selection， +但不会改变 Source ownership。跨 Scope 的精确 Artifact publication 在 lineage 中保留 origin Scope 与精确 +SourceRef。发布 Artifact 不会发布其 origin Scope 中的所有 Source。 + +如果 application 刻意把同一个外部值 captured 到另一个 Scope，target 会得到由该 Scope 拥有的新 Source +observation。其 provenance 可以引用 origin scoped SourceRef，但原始 Source 不会移动，两个 SourceKey 也不会 +因此变成同一 identity。 + + +## Persistence、retention、迁移与 operation freezing + +持久化模型必须把 Scope-owned Source history 与可变的 current-head selection 分开表示。每个接受的 observation 都会 +获得一个不可变 SourceRef。推进 head、删除 head 或修改 locator 都不能删除或改写 observation。Artifact、Candidate +和 Handoff lineage 必须保留其计算实际使用的精确 SourceRef。 + +被 durable Artifact、Candidate 或 Handoff 引用的 observation 必须免于普通 garbage collection。删除 head 不得删除 +历史 observation。Legal 或 user-requested hard deletion 是单独的、可审计的操作；如果它移除了已被引用的 evidence， +产生的 lineage break 必须显式且可观察。 + +消费 `latest` 的 operation 必须在开始时解析并冻结精确 SourceRef（或等价的 Source high-watermark）。之后 head +推进不能改变该 operation 的输入，输出 lineage 必须记录实际使用的 observation。Exact read 必须区分历史缺失、 +referenced content 不可用、Definition 不兼容和 current head 已删除；任何一种情况都不能静默替换为更新的 +observation 或 provider 当前值。 + +从当前两段式 Source row 迁移时，迁移必须一对一且幂等：每个旧 row 获得稳定的 SourceKey/head 和确定性的 +observation identity，已有 Artifact、Candidate 和 Handoff reference 继续解析到同一份 payload。旧的两段式读取 +仍是显式的兼容路径，不能解析到更新的 head。只有在维持这条规则时，旧 worker 与新 worker 才能并存。中断后的 +迁移必须可以继续执行，且不能重复创建 observation 或改变 payload-conflict 语义；回滚不能改写已接受的 evidence。 + +## Conformance + +Source Definition 只有在以下 mandatory contract 的 conformance scenario 通过后才能被支持： + +- identity normalization 与 collision rejection； +- identical observation replay； +- 同一 observation ID 的 conflicting payload rejection； +- 一个 SourceKey 下的多个 immutable observation； +- head advancement 与 deletion 后仍能精确读取旧 observation； +- captured 与 referenced value 的 digest verification； +- referenced-value unavailability 与 mutation； +- Scope isolation 与显式 owner preservation； +- Definition version compatibility 与 unavailable-definition behavior； +- explicit registration conflict handling。 + +Named projection 只有在 conformance 验证 exact observation 的 deterministic output、schema 与 version conflict +handling、exact SourceRef lineage，以及 capability 缺失时显式失败之后才能被声明。 + +Connector capability 只有在 conformance 验证 checkpoint replay、per-item outcome visibility、durable checkpoint +ordering、complete-versus-incomplete run behavior，以及其声明的 deletion evidence 后才能被声明。Provider-specific +behavior 由对应实现证据确定，不会被直接推广为标准契约。 + +Remote worker path 还必须验证 manifest fingerprint 与 conflict handling、拒绝未注册或 schema-invalid observation、 +projection set 精确校验、durable receipt ordering，以及跨 Server restart 的 stale checkpoint CAS rejection。 + +# Drawbacks + +- 分离 SourceKey、SourceRef、Source head 与 Definition version，比一个不可变的 `(source_type, source_id)` pair + 引入更多概念。 +- 精确 SourceRef 保留 owner Scope 与 observation identity，会增加 lineage payload 大小。 +- Definition author 必须声明 canonicalization、provenance 与 compatibility，而不能依赖任意 metadata。 +- Named projection 与 Connector lifecycle state 增加了需要独立于 Source value 演进的契约。 +- 只暴露当前值的 provider 无法使用 Referenced Source，因此部分集成必须保留 captured data。 +- Custom Source observation 被接受之前，显式 registration 需要部署协调。 + +# Rationale and alternatives + +## Extend ContentSource into the general integration model + +向 ContentSource 添加 provider fields 可以保留 `POST /v1/sources/content` capture API，但仍会把 logical identity、observation identity +与 provenance 留在调用方约定中。不同集成会在 metadata 中编码不兼容 schema，non-text Source value 仍需要另一 +套模型。因此 ContentSource 继续作为有用的 single-observation implementation。 + +## Use one opaque Source envelope + +通用 JSON payload 可以统一 persistence 与 transport，但会把 schema validation 和 compatibility 推给 runtime +convention。Definition-owned typed value 与 provenance 让扩展边界可审查，并允许 consumer 在解释前拒绝不支持的 +Source type。 + +## Put an observation digest inside source_id + +集成可以把 logical identity 与 digest 组合进 `source_id`，从而维持二元 SourceRef 形态。这可以表达 immutable +capture，却会在 catalog 中隐藏持续存在的 logical Source。Update、current-head selection、deletion 与 provider +identity 都会变成 integration-private convention。标准模型直接表达两类 identity。 + +## Make SourceRef logical and add a separate ObservationRef + +两个 public reference type 可以让 SourceRef 表示逻辑身份，但 Artifact evidence 必须拒绝 SourceRef，只接受 +ObservationRef。让 SourceRef 本身保持精确，符合现有 ArtifactRef 原则：durable lineage 引用 immutable state。 + +## Add hybrid materialization + +增加一种有时从外部读取、有时回退到 captured data 的第三种模式，会掩盖哪个 value 才是 authoritative，以及哪些 +failure 应对外可见。Captured observation 可以把完整 external reference 保留为 provenance；referenced +observation 要么被精确解析，要么失败。 + +## Let Parent or Connector identity own Sources + +Scope Parent 用于 organization，Connector identity 是 acquisition provenance，二者都不是持久 ownership +boundary。使用其中任意一个都会与 Scope organization contract 冲突，并让 reorganization 或 Connector +replacement 改变 Source identity。 + +# Prior art + +- [Scope organization and Agent integration design](https://github.com/oceanbase/powercontext/pull/1345) 分离 + Scope ownership、read sharing、organization、delivery 与 observation。本 RFC 对 Source ownership、identity、 + exact evidence 与 acquisition 应用同样的分离原则。 +- [Apache OpenDAL OFS RFC-0016](https://github.com/apache/opendal-ofs/blob/main/rfcs/0016_filesystem_architecture.md) + 分离 namespace authority 与 access frontend，并禁止 frontend 宣称底层无法兑现的保证。Source materialization + 遵循同样的 authority rule。 +- [opendalfs](https://github.com/fsspec/opendalfs) 通过 fsspec interface 暴露 OpenDAL services，展示了 + backend-neutral filesystem acquisition。它的 path 与 file metadata 不定义 Source identity 或 immutable + revision semantics。只有完整调用链能够寻址并验证不可变 revision 时，backend read 才能满足 referenced + materialization。 +- DataHub stateful ingestion 把 connector checkpoint 与 stale-entity detection 同 emitted metadata identity + 分离。Airbyte 把 connector state 当作 opaque recovery boundary，而不是 record identity。 +- OpenMetadata 把负责生成 record 的 Source 与 connection check、workflow status、sink 分离。 +- Nowledge Mem 的 TiddlyWiki importer 使用 stable logical ID、canonical payload digest、source revalidation 与 + per-item outcome。这些行为为 Source observation 与 Connector run state 的分离提供依据。 + +## TencentDB-Agent-Memory 调研参考（research input） + +本节是从 #1388 合并而来的非规范性调研输入，不增加第二套 Source contract。 本设计参考了 [TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)： @@ -141,62 +537,27 @@ PowerContext 的要求不同：当 Artifact 已经引用这份原文时，当前 - [Knowledge metadata schema](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/db/client.ts) - [CodeGraph service](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/store/code-graph-service.ts) -# Persistence 与 Runtime 影响 - -实现预计需要在可变 Source/current-head 表示之外增加不可变 snapshot 表。Artifact lineage 应引用不可变 snapshot key， -而不是只保存 source_type/source_id。具体迁移结构留给实现阶段，但必须保留旧 captured Source，并拒绝在已有 -snapshot identity 下替换 payload。 - -新增 typed Source 必须接入 Source resolve 与 exact read、persistence encoding 与 decoding、Runtime composition、远程暴露时 -的 HTTP/client mapping、evidence projection、citation validation，以及聚焦的 persistence 和 end-to-end tests。Evidence -projector 应通过 adapter capability 或 registry 路由，避免 Runtime composition 不断累积 ContentSource 特判。 - -现有 ContentSource capture 必须继续可读；现有 POST /v1/sources/content 在当前 identity/payload contract 下继续幂等。旧的 -两段式引用需要显式迁移规则，不能在会改变历史 Artifact lineage 含义时静默解释为 latest snapshot。OpenAPI 变更应在 -identity 和 legacy behavior 达成共识后进行。 - -# 备选方案 - -## 保持当前 stable Source key - -API 最小，但 provider 对象变化仍然冲突，也无法引用多次 snapshot，只适合已经不可变的 capture。 - -## 只使用 provider reference - -存储成本低，但 exact evidence 依赖 provider retention、权限、可用性和历史读取语义，无法满足任意集成的 replay 要求。 - -## 所有 provider value 都使用 ContentSource - -类型更少，但 provider semantics 进入无类型 metadata，校验能力变弱，evidence projection 也无法感知 provider，不是持久的 -扩展边界。 - -## 只保存本地 revision - -本地 revision 有助于排序和 cursor，但不能证明观察到哪一份 provider state，只能补充而不能替代 provider revision 和 hash。 - -# 落地与验证 +# Unresolved questions -首个有界验证应覆盖一个会变化的 provider object 和一个不可变 provider revision。GitHub issue/commit 或 Git repository -commit 都是候选。验证至少应证明： +- 每个 durable SourceRef 是否必须直接携带 `scope_id`，还是可以由 canonical scoped envelope 包含 local exact + SourceRef，同时保留相同的 fully qualified identity？ +- Runtime 必须同时保留哪些 Source Definition version，才能宣称某个 Definition 受支持？ +- Source head deletion 应是通用 catalog state，还是标准契约只暴露 active exact head，并把 deletion 完全留给 + Connector state？ +- 哪些 projection name 与 schema 已有足够实现证据，可以成为 shared standard 而不是 namespaced capability？ -1. 同一个 provider revision 的重复投递幂等； -2. 后续 provider revision 产生新 snapshot 并推进当前 head，而不替换旧 snapshot； -3. 当前检索和 ingest 使用新 snapshot； -4. provider 变化后 Artifact 仍能引用并读取旧 snapshot； -5. 缺失或无法验证的 Ref 被拒绝，而不是静默刷新； -6. evidence projection 与 HTTP/client mapping 保留精确 snapshot reference。 +# Future possibilities -# 未决问题 +显式 plugin discovery 与 deployment policy 可以建立在 Definition 和 Connector registration 之上，但不会让 +package installation 等同于 activation。 -- snapshot_id 应由 PowerContext 生成 opaque ID、使用 content-addressed ID，还是同时提供两者？ -- SourceRef 是否增加 snapshot_id，还是引入独立 SnapshotRef？ -- 哪些 provider revision guarantee 可以允许 Ref-only materialization？ -- 不再被 Artifact 引用的 snapshot 应采用什么 retention 和 garbage-collection policy？ -- 大型 capture 是否使用外部 blob store，同时保留 canonical hash 和 durable locator？ -- logical Source head 是否应成为首版 public API，还是初期只作为 connector/persistence metadata？ +Retention policy 可以进一步定义何时回收未被引用的 observation，以及如何报告 unavailable exact evidence；但必须 +保留上文规定的最低 lineage protection。Legal 或 user-requested deletion 可以作为单独的 audited workflow 规定。 +删除 Source head 本身不授权移除 evidence。 # 请求决策 -请评审并批准：可变 Source head 加不可变 snapshot 模型；snapshot 作为 exact-evidence boundary；Capture 作为默认 -materialization；Hybrid 作为可追溯场景的优先形态；ContentSource 作为一个具体 capture 类型而不是 universal provider -model。批准后再定义具体 schema、migration、OpenAPI 字段、retention 规则和一个有界 connector validation。 +请将本 Source Definition 与 observation contract 作为 #1388 的基线，并将 TencentDB-Agent-Memory 调研保留为 +prior art。具体请批准 SourceKey/SourceRef 分离、不可变 observation 与 exact materialization、named projection、 +Connector lifecycle 边界，以及上文的 persistence、retention、operation freeze 和 migration 不变量。后续具体 +schema 与 rollout 只有在保持这些不变量的前提下才能推进。 From 270798da9d2088be33b1057af021834d99fa1cae Mon Sep 17 00:00:00 2001 From: Nene7ko_ <1604009816@qq.com> Date: Sat, 29 Aug 2026 02:54:29 +0800 Subject: [PATCH 4/5] docs(rfc): align materialization with psiace baseline --- docs/en/rfcs/1388_source_integration_shape.md | 9 +-------- docs/zh/rfcs/1388_source_integration_shape.md | 8 +------- 2 files changed, 2 insertions(+), 15 deletions(-) diff --git a/docs/en/rfcs/1388_source_integration_shape.md b/docs/en/rfcs/1388_source_integration_shape.md index 76f69e65f..b4d13e027 100644 --- a/docs/en/rfcs/1388_source_integration_shape.md +++ b/docs/en/rfcs/1388_source_integration_shape.md @@ -217,8 +217,7 @@ Materialization answers where the value returned for an exact SourceRef comes fr | `referenced` | Immutable external revision | Re-reading the reference returns the same canonical value and digest | A captured Source may retain an external locator, provider revision, and digest as provenance. It remains captured -because the retained value is the read authority. This covers the useful part of a hybrid design without creating a -third mode with ambiguous fallback semantics. +because the retained value is the read authority. A Definition can use referenced materialization only when the external system and its reader can address immutable historical values. Reading the current value at a path, page ID, issue ID, or URL is not sufficient. Modification @@ -519,12 +518,6 @@ Two public reference types would make SourceRef logical, but Artifact evidence w to reject SourceRef and accept only ObservationRef. Defining SourceRef itself as exact follows the existing ArtifactRef principle that durable lineage references immutable state. -## Add hybrid materialization - -A third mode that sometimes reads externally and sometimes falls back to captured data obscures which value is -authoritative and which failures are visible. A captured observation can retain a complete external reference as -provenance. A referenced observation either resolves exactly or fails. - ## Let Parent or Connector identity own Sources Scope Parent is organization, and Connector identity is acquisition provenance. Neither is a durable ownership diff --git a/docs/zh/rfcs/1388_source_integration_shape.md b/docs/zh/rfcs/1388_source_integration_shape.md index 2ff39091b..9931a3793 100644 --- a/docs/zh/rfcs/1388_source_integration_shape.md +++ b/docs/zh/rfcs/1388_source_integration_shape.md @@ -204,7 +204,7 @@ Materialization 回答精确 SourceRef 的返回值来自哪里： | `referenced` | Immutable external revision | 重读 reference 得到相同 canonical value 与 digest | Captured Source 可以把 external locator、provider revision 与 digest 保留为 provenance。因为读取权威仍是 -保留值，所以它依然是 captured。这覆盖了 hybrid design 中有价值的部分，而不引入 fallback 语义含糊的第三种模式。 +保留值，所以它依然是 captured。 只有当外部系统及其 reader 能够寻址不可变历史值时，Definition 才能使用 referenced materialization。读取 path、page ID、issue ID 或 URL 的当前值并不足够。Modification time 与 ETag 可以参与 provenance 或 conflict @@ -481,12 +481,6 @@ identity 都会变成 integration-private convention。标准模型直接表达 两个 public reference type 可以让 SourceRef 表示逻辑身份，但 Artifact evidence 必须拒绝 SourceRef，只接受 ObservationRef。让 SourceRef 本身保持精确，符合现有 ArtifactRef 原则：durable lineage 引用 immutable state。 -## Add hybrid materialization - -增加一种有时从外部读取、有时回退到 captured data 的第三种模式，会掩盖哪个 value 才是 authoritative，以及哪些 -failure 应对外可见。Captured observation 可以把完整 external reference 保留为 provenance；referenced -observation 要么被精确解析，要么失败。 - ## Let Parent or Connector identity own Sources Scope Parent 用于 organization，Connector identity 是 acquisition provenance，二者都不是持久 ownership From 04df0bd4768ecf71bbc7484e0741b86c6d7907b6 Mon Sep 17 00:00:00 2001 From: Nene7ko_ <1604009816@qq.com> Date: Sat, 29 Aug 2026 02:58:53 +0800 Subject: [PATCH 5/5] docs(rfc): preserve psiace source model baseline --- docs/en/rfcs/1388_source_integration_shape.md | 51 +++++-------------- docs/zh/rfcs/1388_source_integration_shape.md | 45 ++++------------ 2 files changed, 24 insertions(+), 72 deletions(-) diff --git a/docs/en/rfcs/1388_source_integration_shape.md b/docs/en/rfcs/1388_source_integration_shape.md index b4d13e027..996348e39 100644 --- a/docs/en/rfcs/1388_source_integration_shape.md +++ b/docs/en/rfcs/1388_source_integration_shape.md @@ -1,6 +1,6 @@ -- Proposal Name: `source_integration_shape` +- Proposal Name: `source_definition_and_observation_model` - Status: Proposed -- Start Date: 2026-08-25 +- Start Date: 2026-08-27 - RFC PR: [oceanbase/powercontext#1388](https://github.com/oceanbase/powercontext/pull/1388) - Tracking Issue: [oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240) - Related Discussion: [oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240), @@ -217,7 +217,8 @@ Materialization answers where the value returned for an exact SourceRef comes fr | `referenced` | Immutable external revision | Re-reading the reference returns the same canonical value and digest | A captured Source may retain an external locator, provider revision, and digest as provenance. It remains captured -because the retained value is the read authority. +because the retained value is the read authority. This covers the useful part of a hybrid design without creating a +third mode with ambiguous fallback semantics. A Definition can use referenced materialization only when the external system and its reader can address immutable historical values. Reading the current value at a path, page ID, issue ID, or URL is not sufficient. Modification @@ -428,29 +429,6 @@ If an application deliberately captures the same external value into another Sco Scope-owned Source observation. Its provenance may cite the origin Scoped SourceRef, but the original Source is not moved and the two SourceKeys are not made identical. - -## Persistence, retention, migration, and operation freezing - -The persistence model MUST represent Scope-owned Source history separately from the mutable current-head selection. -Each accepted observation receives one immutable SourceRef. Head advancement, head deletion, and locator changes MUST -not delete or rewrite observations. Artifact, Candidate, and Handoff lineage MUST retain the exact SourceRefs used by -their computation. - -An observation referenced by a durable Artifact, Candidate, or Handoff MUST be protected from ordinary garbage -collection. Head deletion MUST NOT delete historical observations. Legal or user-requested hard deletion is a separate -audited operation; if it removes cited evidence, the resulting lineage break MUST be explicit and observable. - -An operation that consumes `latest` MUST resolve and freeze exact SourceRefs (or an equivalent Source high-watermark) -at operation start. Later head advancement MUST NOT change the operation's inputs, and its output lineage MUST record -the observations actually used. Exact reads MUST distinguish missing history, unavailable referenced content, definition -incompatibility, and a deleted current head; none may be silently replaced by a newer observation or current provider value. - -Migration from the current two-part Source rows MUST be one-to-one and idempotent: each legacy row receives a stable -SourceKey/head and deterministic observation identity, and existing Artifact, Candidate, and Handoff references continue -to resolve to that same payload. Legacy two-part reads remain an explicit compatibility path and MUST NOT resolve to a -newer head. Old and new workers may coexist only while this rule is enforced. An interrupted migration MUST resume -without duplicating observations or changing payload-conflict semantics; rollback MUST NOT rewrite accepted evidence. - ## Conformance A Source Definition can be supported only after its mandatory contract passes conformance scenarios for: @@ -518,6 +496,12 @@ Two public reference types would make SourceRef logical, but Artifact evidence w to reject SourceRef and accept only ObservationRef. Defining SourceRef itself as exact follows the existing ArtifactRef principle that durable lineage references immutable state. +## Add hybrid materialization + +A third mode that sometimes reads externally and sometimes falls back to captured data obscures which value is +authoritative and which failures are visible. A captured observation can retain a complete external reference as +provenance. A referenced observation either resolves exactly or fails. + ## Let Parent or Connector identity own Sources Scope Parent is organization, and Connector identity is acquisition provenance. Neither is a durable ownership @@ -542,7 +526,6 @@ Connector replacement change Source identity. - Nowledge Mem's TiddlyWiki importer uses stable logical IDs, canonical payload digests, source revalidation, and per-item outcomes. Those behaviors inform the separation between Source observations and Connector run state. - ## TencentDB-Agent-Memory research (research input) This section is non-normative research input carried forward from #1388; it does not add a second Source contract. @@ -586,14 +569,6 @@ References: Explicit plugin discovery and deployment policy may build on Definition and Connector registration without making package installation equivalent to activation. -Retention policies may define when unreferenced observations are reclaimed and how unavailable exact evidence is -reported; they must preserve the minimum lineage protection above. Legal or user-requested deletion can be specified as -a separate audited workflow. Source head deletion alone does not authorize evidence removal. - -# Decision requested - -Approve this Source Definition and observation contract as the baseline for #1388, with the TencentDB-Agent-Memory -research retained as prior art. In particular, approve the SourceKey/SourceRef separation, immutable observations -and exact materialization, named projections, Connector lifecycle boundary, and the persistence, retention, -operation-freezing, and migration invariants above. Concrete schema and rollout work may proceed only if it preserves -these invariants. +Retention policies may reclaim captured values only after defining how exact Artifact evidence reports unavailable +content and how legal or user-requested deletion interacts with immutable lineage. A Source head deletion alone does +not authorize evidence removal. diff --git a/docs/zh/rfcs/1388_source_integration_shape.md b/docs/zh/rfcs/1388_source_integration_shape.md index 9931a3793..738c66731 100644 --- a/docs/zh/rfcs/1388_source_integration_shape.md +++ b/docs/zh/rfcs/1388_source_integration_shape.md @@ -1,6 +1,6 @@ -- Proposal Name: `source_integration_shape` +- Proposal Name: `source_definition_and_observation_model` - Status: Proposed -- Start Date: 2026-08-25 +- Start Date: 2026-08-27 - RFC PR：[oceanbase/powercontext#1388](https://github.com/oceanbase/powercontext/pull/1388) - Tracking Issue：[oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240) - Related Discussion: [oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240), @@ -204,7 +204,7 @@ Materialization 回答精确 SourceRef 的返回值来自哪里： | `referenced` | Immutable external revision | 重读 reference 得到相同 canonical value 与 digest | Captured Source 可以把 external locator、provider revision 与 digest 保留为 provenance。因为读取权威仍是 -保留值，所以它依然是 captured。 +保留值，所以它依然是 captured。这覆盖了 hybrid design 中有价值的部分，而不引入 fallback 语义含糊的第三种模式。 只有当外部系统及其 reader 能够寻址不可变历史值时，Definition 才能使用 referenced materialization。读取 path、page ID、issue ID 或 URL 的当前值并不足够。Modification time 与 ETag 可以参与 provenance 或 conflict @@ -400,27 +400,6 @@ SourceRef。发布 Artifact 不会发布其 origin Scope 中的所有 Source。 observation。其 provenance 可以引用 origin scoped SourceRef，但原始 Source 不会移动，两个 SourceKey 也不会 因此变成同一 identity。 - -## Persistence、retention、迁移与 operation freezing - -持久化模型必须把 Scope-owned Source history 与可变的 current-head selection 分开表示。每个接受的 observation 都会 -获得一个不可变 SourceRef。推进 head、删除 head 或修改 locator 都不能删除或改写 observation。Artifact、Candidate -和 Handoff lineage 必须保留其计算实际使用的精确 SourceRef。 - -被 durable Artifact、Candidate 或 Handoff 引用的 observation 必须免于普通 garbage collection。删除 head 不得删除 -历史 observation。Legal 或 user-requested hard deletion 是单独的、可审计的操作；如果它移除了已被引用的 evidence， -产生的 lineage break 必须显式且可观察。 - -消费 `latest` 的 operation 必须在开始时解析并冻结精确 SourceRef（或等价的 Source high-watermark）。之后 head -推进不能改变该 operation 的输入，输出 lineage 必须记录实际使用的 observation。Exact read 必须区分历史缺失、 -referenced content 不可用、Definition 不兼容和 current head 已删除；任何一种情况都不能静默替换为更新的 -observation 或 provider 当前值。 - -从当前两段式 Source row 迁移时，迁移必须一对一且幂等：每个旧 row 获得稳定的 SourceKey/head 和确定性的 -observation identity，已有 Artifact、Candidate 和 Handoff reference 继续解析到同一份 payload。旧的两段式读取 -仍是显式的兼容路径，不能解析到更新的 head。只有在维持这条规则时，旧 worker 与新 worker 才能并存。中断后的 -迁移必须可以继续执行，且不能重复创建 observation 或改变 payload-conflict 语义；回滚不能改写已接受的 evidence。 - ## Conformance Source Definition 只有在以下 mandatory contract 的 conformance scenario 通过后才能被支持： @@ -481,6 +460,12 @@ identity 都会变成 integration-private convention。标准模型直接表达 两个 public reference type 可以让 SourceRef 表示逻辑身份，但 Artifact evidence 必须拒绝 SourceRef，只接受 ObservationRef。让 SourceRef 本身保持精确，符合现有 ArtifactRef 原则：durable lineage 引用 immutable state。 +## Add hybrid materialization + +增加一种有时从外部读取、有时回退到 captured data 的第三种模式，会掩盖哪个 value 才是 authoritative，以及哪些 +failure 应对外可见。Captured observation 可以把完整 external reference 保留为 provenance；referenced +observation 要么被精确解析，要么失败。 + ## Let Parent or Connector identity own Sources Scope Parent 用于 organization，Connector identity 是 acquisition provenance，二者都不是持久 ownership @@ -545,13 +530,5 @@ PowerContext 的要求不同：当 Artifact 已经引用这份原文时，当前 显式 plugin discovery 与 deployment policy 可以建立在 Definition 和 Connector registration 之上，但不会让 package installation 等同于 activation。 -Retention policy 可以进一步定义何时回收未被引用的 observation，以及如何报告 unavailable exact evidence；但必须 -保留上文规定的最低 lineage protection。Legal 或 user-requested deletion 可以作为单独的 audited workflow 规定。 -删除 Source head 本身不授权移除 evidence。 - -# 请求决策 - -请将本 Source Definition 与 observation contract 作为 #1388 的基线，并将 TencentDB-Agent-Memory 调研保留为 -prior art。具体请批准 SourceKey/SourceRef 分离、不可变 observation 与 exact materialization、named projection、 -Connector lifecycle 边界，以及上文的 persistence、retention、operation freeze 和 migration 不变量。后续具体 -schema 与 rollout 只有在保持这些不变量的前提下才能推进。 +Retention policy 只有在定义精确 Artifact evidence 如何报告 unavailable content，以及 legal/user-requested +deletion 如何与 immutable lineage 交互之后，才能回收 captured value。Source head deletion 本身不授权删除证据。