diff --git a/docs/en/rfcs/1388_source_integration_shape.md b/docs/en/rfcs/1388_source_integration_shape.md new file mode 100644 index 000000000..996348e39 --- /dev/null +++ b/docs/en/rfcs/1388_source_integration_shape.md @@ -0,0 +1,574 @@ +- Proposal Name: `source_definition_and_observation_model` +- Status: Proposed +- Start Date: 2026-08-27 +- RFC PR: [oceanbase/powercontext#1388](https://github.com/oceanbase/powercontext/pull/1388) +- Tracking Issue: [oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240) +- Related Discussion: [oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240), + [oceanbase/powercontext#1363](https://github.com/oceanbase/powercontext/issues/1363) +- Related Design: [oceanbase/powercontext#1345](https://github.com/oceanbase/powercontext/pull/1345) +- Related RFCs: [RFC 0002](0002_core_sdk_product_model.md), [RFC 0014](0014_memory_layer_design.md), + [RFC 0019](0019_local_source_memory_runtime.md), [RFC 0048](0048_handoff_artifact.md) + +# Summary + +This RFC defines the standard Source model and the contract for defining additional Source types. + +A Source belongs to exactly one Scope. Within that Scope, a `SourceKey` identifies one logical source and a +`SourceRef` identifies one immutable observation of that source. Advancing the current observation, observing a +deletion, changing an external locator, or disconnecting a Connector does not alter an earlier observation or move +it to another Scope. + +A Source Definition gives one stable Source type its value schema, provenance schema, identity rules, observation +rules, materialization contract, canonicalization, and compatibility policy. Definitions are registered explicitly +and remain fixed for the lifetime of a composed Runtime. Persistence, transport, and Artifact consumers route by the +stable definition name and version rather than by a concrete Python class. + +A Definition may advertise named projection capabilities for consumers that do not understand its native value. +Each projection has an independently versioned schema and deterministic meaning over one exact observation. A +consumer selects a projection by capability name and version, never by inspecting a concrete Source class. + +A Connector lifecycle binds provider acquisition to a Scope, resolves definition-native inputs in its worker, +submits materialized observations, records per-item outcomes, and advances an opaque checkpoint only after accepted +observations are durable. Connector runs distinguish complete discovery from incomplete discovery so that absence +is not silently converted into deletion. + +Materialization identifies the authority used to resolve an exact observation. A captured observation is resolved +from the canonical value retained by PowerContext. A referenced observation is resolved from an immutable external +revision. An external locator, modification time, ETag, or current-provider read does not by itself satisfy the +referenced contract. + +`ContentSource` remains a simple captured-text Source. Its caller-stable identity and immutable-payload conflict rule +make it useful for one-shot content capture, but it is not the general external integration model. + +This RFC defines Source, projection, Connector lifecycle, and the remote ingestion boundary between a worker and the +PowerContext Server. It does not define plugin discovery, a scheduler, credential transport, a concrete Source +family, or a Connector implementation. + +# Motivation + +`ContentSource` and `POST /v1/sources/content` provide captured-text ingestion. The caller +chooses one `source_id`; replaying an identical payload is idempotent, while reusing that identity with a different +payload is a conflict. This gives exact evidence only when the caller treats the identity as immutable. + +External systems usually expose a different lifecycle. A wiki page, issue, object, message, or file has one logical +identity but may produce several values over time. The external object can be renamed, revised, deleted, restored, +or become temporarily unreadable. Artifacts that used an earlier value must continue to cite that exact evidence. +The two-part `(source_type, source_id)` Source reference cannot express both the stable logical object and its immutable observation +without making every integration invent a composite `source_id`. + +The extension boundary is also incomplete. A Source adapter binds a native input class to a concrete +Source class and a read result, while the built-in Runtime and relational persistence assemble a fixed adapter set. +This does not state the durable rules an independently defined Source type must follow across identity, persistence, +transport, and Artifact evidence. + +The standard model must answer six questions without assigning them to one identifier: + +1. Which Scope owns this evidence? +2. Which logical external or internal source does it describe? +3. Which exact observed value did an Artifact use? +4. Where does PowerContext read that exact value from? +5. Which definition gives the value and provenance their meaning? +6. Which declared view may a consumer use without understanding the native value? + +Connector concerns are adjacent but distinct. Discovery, credentials, filtering, checkpoints, retries, provider +change handling, and deletion detection decide which observations are submitted. They do not define Source identity, +weaken exact evidence, or change Scope ownership. + +# Guide-level explanation + +## Domain model + +Read the model by establishing ownership first, then logical identity, exact observation, materialization authority, +and type semantics: + +| Concept | Representation | Question answered | +| --- | --- | --- | +| Ownership | Scope | Where does the Source belong? | +| Logical identity | `SourceKey` | Which continuing source is this? | +| Exact evidence | `SourceRef` | Which immutable observation is cited? | +| Read authority | materialization | Where is that exact value resolved? | +| Type semantics | Source Definition | How are value, provenance, and identity interpreted? | +| Consumer view | named projection | Which declared representation may a consumer use? | +| Acquisition | Connector or direct caller | How are new observations found and submitted? | + +These responsibilities form one direction of dependency: + +```text +Connector or direct caller + | + v +Source Definition + | + v +Scope-owned Source history + | + +---- mutable head selection + | + `---- exact SourceRef ----> Artifact evidence +``` + +A Connector can use one Source Definition, several Connectors can use the same Definition, and a direct caller can +submit a Source without a Connector. Connector identity therefore does not become Source type identity. + +## Scope ownership + +Every SourceKey and observation belongs to exactly one Scope. Scope ownership is not inferred from an external +workspace, path, repository, provider account, Connector instance, or Source locator. Those values may contribute to +binding or provenance, but they do not allocate or replace `scope_id`. + +The fully qualified logical identity is: + +```text +SourceKey = (scope_id, source_type, source_id) +``` + +The fully qualified exact identity is: + +```text +SourceRef = (scope_id, source_type, source_id, observation_id) +``` + +A scope-bound operation may obtain `scope_id` from its fixed request binding instead of accepting it as an arbitrary +argument. The durable resolved reference still retains the owner Scope so that evidence remains unambiguous after +publication, reporting, or export. + +Changing a Scope Parent, Context References, an Agent binding, or an observation selection changes no SourceKey or +SourceRef. Publishing an Artifact across Scopes preserves the original Scope and exact SourceRef in provenance. It +does not move or implicitly copy the Source history. + +## Logical Source and immutable observation + +`source_id` names a logical source within one `(scope_id, source_type)` namespace. Its meaning is defined by the +Source Definition. It may correspond to a provider object ID, a stable import identity, or another normalized key. +It must not silently change when a new value is observed. + +`observation_id` names one immutable observation under a SourceKey. It is opaque to generic PowerContext components. +It may be derived from a provider revision, a canonical value digest, or a definition-specific combination. It does +not imply an integer sequence, timestamp order, or ancestry. + +The following invariants apply: + +- one `(SourceKey, observation_id)` identifies one canonical observation forever; +- re-observing the same canonical observation is idempotent; +- a different canonical observation cannot reuse an observation ID; +- one SourceKey may have several observations with the same value digest when their identity-bearing provenance is + different; +- observations with the same value digest are not automatically the same logical Source; and +- an Artifact cites an exact SourceRef, never a moving SourceKey or `latest` observation. + +For example, updating one logical Source retains its SourceKey and produces another SourceRef: + +```text +SourceKey(scope-a, record, provider-object-42) +|-- SourceRef(..., observation-1) "Initial value" +`-- SourceRef(..., observation-2) "Revised value" +``` + +An Artifact derived from `observation-1` continues to cite it after `observation-2` becomes current. + +## Source Definition + +A Source Definition is the durable semantic contract for one `source_type`. It declares: + +- a stable definition name and version; +- the Source value and typed provenance shapes; +- Source ID normalization and equality; +- observation ID normalization and equality; +- identity-bearing fields and non-identifying annotations; +- canonical bytes and the value digest algorithm; +- supported materialization modes and exact-read requirements; +- limits and validation failures; and +- compatibility rules for older definition versions. + +A Definition resolves definition-native input into a canonical observation and reads the definition-owned value from +an exact persisted observation. Resolution does not select a Scope, mutate a catalog, advance a head, or discover +external objects. Reading does not resolve `latest` or substitute another observation. + +Definitions are explicit and typed. A new integration must not simulate a new Source type by placing an undocumented +schema inside `ContentSource.metadata`. Provider-specific provenance may extend a Definition's declared schema, but +fields that affect identity, exactness, or compatibility must be named by the Definition. + +## Named projection capabilities + +A named projection is an optional, Definition-owned view of one exact observation. It allows an Artifact family or +another consumer to use a declared representation without knowing the native Source value or concrete Python class. + +A projection is selected by a stable name and version. Its Definition declares the output schema, canonicalization, +digest rules, and failures. The projection is evaluated against an exact SourceRef and cannot resolve a head, +`latest`, or a current provider value. For the same Definition version, projection version, and exact observation, it +returns the same canonical result. + +Projection capability is explicit. A consumer that requires a projection rejects a Source that does not advertise a +compatible capability; it does not infer content from metadata or fall back to a similarly shaped Source class. A +projection can be cached or persisted as a derivative, but its authority remains the exact Source observation and +its lineage retains that SourceRef. + +This contract does not prescribe a catalog of standard projection names or payload schemas. A projection becomes a +shared standard only after interoperating definitions and consumers demonstrate that its semantics are stable. Until +then, a Definition may expose namespaced projections without making them mandatory for other Source types. + +## Materialization authority + +Materialization answers where the value returned for an exact SourceRef comes from: + +| Materialization | Authority | Required guarantee | +| --- | --- | --- | +| `captured` | Canonical value retained by PowerContext | The retained value matches the observation digest | +| `referenced` | Immutable external revision | Re-reading the reference returns the same canonical value and digest | + +A captured Source may retain an external locator, provider revision, and digest as provenance. It remains captured +because the retained value is the read authority. This covers the useful part of a hybrid design without creating a +third mode with ambiguous fallback semantics. + +A Definition can use referenced materialization only when the external system and its reader can address immutable +historical values. Reading the current value at a path, page ID, issue ID, or URL is not sufficient. Modification +times and ETags may contribute to provenance or conflict detection, but a Definition must state whether the provider +guarantees that they address an immutable value. + +When the referenced value is unavailable or its digest differs, exact resolution fails. PowerContext does not return +the current provider value, a stale cache entry, or another observation. A provider that cannot satisfy this rule +must use captured materialization or reject the observation. + +## Current head and deletion + +A Source history is immutable; its current head is a mutable catalog selection. The head can select one exact +SourceRef or record that the logical Source was positively observed as deleted. The head is useful for current-state +queries and later acquisition, but it is not evidence and cannot appear in an Artifact citation. + +Advancing or deleting a head changes no observation. A timeout, permission failure, incomplete listing, unavailable +Connector, or disconnect is not positive deletion evidence and does not change the head. A Source Definition may +define a tombstone value only when deletion itself is meaningful Source evidence; a generic head deletion does not +fabricate one. + +## ContentSource + +`ContentSource` remains the neutral captured-text path defined by RFC 0019. Its caller chooses an identity that can be +committed once with one canonical payload. The persistence conflict rule makes an accepted ContentSource exact, but +it does not provide a separate logical Source lifecycle. + +The standard model treats this as a valid single-observation Source implementation: + +- the existing identity remains immutable; +- an identical replay remains idempotent; +- different content under the same identity remains a conflict; +- references that resolve ContentSource remain exact and unchanged; and +- no mutable head or multi-observation behavior is inferred from metadata. + +ContentSource is suitable for prompts, explicit text capture, import records, and other cases where the caller +already owns an immutable identity. Integrations that observe one logical object over time should define or reuse a +multi-observation Source type instead. + +# Reference-level explanation + +## Source identity contract + +`scope_id` is the ownership boundary defined by the Scope organization design. `source_type` is the stable Source +Definition name. `source_id` is a non-empty, normalized identifier whose equality and bounds are declared by that +Definition. + +Source identity is Scope-local. Two Scopes may contain equivalent external material without sharing ownership or +identity. A Definition may include a stable external instance or connection discriminator in its `source_id` rules +when required to prevent collisions, but the discriminator does not replace `scope_id`. + +Renames are definition-specific. A provider object ID may preserve SourceKey across locator changes. A path-derived +identity normally treats a rename as one logical deletion and one creation. A Definition must not claim rename-stable +identity when its provider and acquisition path cannot prove it. + +## Observation contract + +An observation contains these standard fields: + +```text +SourceObservation +|-- source_key +|-- observation_id +|-- definition_version +|-- materialization +|-- value_digest +|-- provenance +`-- definition-owned value or exact external reference +``` + +`value_digest` uses SHA-256 over the canonical bytes declared by the Definition and is encoded as +`sha256:`. For structured values, the Definition specifies a deterministic canonicalization. The +digest verifies value equality; it does not replace SourceKey or observation identity. + +The canonical observation contains every field that the Definition says affects identity or exact meaning. +Operational facts such as a retry count, last scan time, or processing status are not Source value and do not change +observation identity. If a timestamp or provider attribute affects provenance meaning, the Definition must classify +and canonicalize it explicitly. + +## Source reference contract + +A SourceRef identifies an exact observation and includes its owner Scope. It never accepts an absent observation ID, +`latest`, a head version, or a current provider locator. + +Within a scope-bound operation, a compact local representation may omit a repeated `scope_id` only while the current +Scope is fixed and the resolved durable value restores it. Any reference that crosses a Scope boundary, leaves the +Runtime, or enters durable cross-Scope provenance carries the owner Scope explicitly. + +Reference resolution verifies all four identity components and the stored observation's definition version and +digest. Failure to resolve the exact observation is distinct from the logical Source being deleted, the head having +advanced, or the Connector being unavailable. + +## Definition registration contract + +Executable Definitions belong to the worker that resolves definition-native inputs, canonicalizes Source values, +and computes named projections. The Server does not import Connector or Definition packages and does not execute +their Python classes. + +Before submitting an observation, the worker registers an immutable declarative manifest containing the stable +Definition name and version, the canonical Source JSON Schema, every projection key and output JSON Schema, and a +fingerprint over the complete declaration. The fingerprint is SHA-256 over RFC 8785 canonical JSON. Registration is +idempotent for an identical manifest and rejects a different declaration for an existing `(source_type, +definition_version)`. + +The Server validates the manifest's schemas and any named projection it recognizes as a shared standard. A manifest +does not transfer executable identity rules, canonicalization code, read behavior, credentials, or provider +configuration. Those remain worker-owned. The registered manifest is sufficient for the Server to validate and +retain an opaque canonical observation without loading plugin code. + +Definition discovery and registration are separate. A package entry point or another discovery mechanism may report +available Definitions, but installation does not imply activation. This RFC does not select entry points, a central +settings format, pluggy, or a Connector marketplace. + +## Remote worker ingestion contract + +A Connector runs in an independent worker process. The worker owns provider access and all executable Definition +behavior. The Server owns durable Source history, Artifact consumption, and checkpoint comparison. Their data-plane +interaction consists of four generic operations: + +1. register an immutable Source Definition manifest; +2. read the opaque checkpoint for one Connector binding; +3. submit a worker-materialized Source observation with all declared projections; and +4. compare-and-swap the binding checkpoint from the value read at run start. + +The observation envelope carries the Definition name, version and fingerprint, canonical Source payload, and one +value for every projection declared by the manifest. The Server validates envelope identity, payload schema, +projection-key equality, projection schemas, and standard projection invariants before durable acceptance. Provider +names, storage services, paths, credentials, or other Connector-specific configuration do not appear in this API +unless a Definition deliberately includes them in its canonical Source schema. + +The Server returns a durable Source receipt before the worker may commit a checkpoint. The checkpoint operation uses +optimistic comparison so concurrent runs of the same binding cannot silently overwrite each other. Submission is +idempotent for an identical Source identity and payload; conflicting content for an accepted identity is rejected. + +## Definition compatibility contract + +The Definition name remains stable across compatible schema evolution. Each persisted observation records the +Definition version used to validate and canonicalize it. A newer Definition version must either declare how it reads +an older observation without changing its canonical meaning or coexist with a reader for the older version. + +A Definition change is incompatible when it changes SourceKey equality, observation equality, canonical value bytes, +provenance meaning, or materialization guarantees for an accepted observation. Such a change requires a new +Definition version and cannot rewrite existing SourceRefs. + +A projection change is incompatible when it changes the output schema, canonical bytes, or meaning for an accepted +observation. Such a change requires a new projection version. It does not require a new Source Definition version +when the Source value and observation semantics remain unchanged. + +Renaming a Definition creates a new `source_type`. Reclassifying an existing observation under another Definition is +an explicit derivation with provenance, not an in-place migration of identity. + +## Connector lifecycle contract + +A Connector owns provider interaction: discovery, credentials, filtering, checkpoints, retries, rate limits, +provider change handling, and positive deletion detection. It submits definition-native inputs against a Scope +binding and receives exact accepted SourceRefs. + +A Source Definition owns semantic normalization: logical identity, observation identity, canonical value, +provenance, materialization validity, and exact read. A Connector cannot override those rules. If the intersection of +provider capabilities, Connector behavior, and Definition requirements cannot satisfy a selected materialization, +the observation is rejected or captured under a valid mode. + +```text +provider capabilities + intersect Connector behavior + intersect Source Definition requirements + = valid Source observation +``` + +A Connector type declares a stable name and version, its configuration schema, the Source Definitions it can submit, +and the acquisition capabilities it provides. Capabilities are optional and explicit. Typical capabilities include a +complete snapshot, a change feed, checkpoint resume, and authoritative deletion events. A Connector cannot advertise +a capability that its provider and acquisition path cannot enforce. + +A Connector binding activates one Connector configuration for exactly one Scope. The binding has a stable identity +for checkpoint and provider-namespace continuity, but it does not own Sources and does not replace `scope_id` or +`source_type`. Credentials are resolved by the hosting environment and do not become Source value or provenance. + +A Connector run begins from an opaque binding checkpoint, resolves zero or more definition-native inputs inside the +worker, and submits their materialized observations. It records an outcome for every item. An accepted or +idempotently replayed observation returns its exact SourceRef. A rejected or failed item remains visible in the run +outcome and cannot be hidden by advancing the checkpoint past work that is not safely replayable. + +A run finishes as complete or incomplete. A complete snapshot may produce positive deletion evidence for previously +known provider objects that are absent. An incomplete listing, timeout, permission failure, cancellation, or lost +connection produces no absence-based deletion evidence. An authoritative provider deletion event may produce +positive deletion evidence independently of snapshot completeness when its binding and object identity are verified. + +The completed checkpoint advances only after its accepted observations and deletion evidence are durable. Retrying +from an earlier checkpoint is valid because Source observation submission is idempotent. Connector checkpoint, +health, retry, and run-status records are operational state rather than Source observations or Artifact evidence. + +Installation, discovery, activation, and execution are separate concerns. Installing a Connector package does not +activate a binding. A Connector package executes outside the PowerContext Server and uses the remote worker +ingestion contract; scheduling and process supervision belong to the deployment environment. + +## Artifact evidence and cross-Scope delivery + +An Artifact revision records exact SourceRefs used directly by its computation. Advancing a Source head does not +change existing Artifact lineage. Recalculation against a newer observation produces a new Artifact revision rather +than rewriting prior evidence. + +Sources remain in their producing Scope. A Context Reference may expand a read selection according to the Scope +organization contract, but it does not change Source ownership. Exact Artifact publication across Scopes retains the +origin Scope and exact SourceRef in lineage. Publishing an Artifact does not publish every Source in its origin Scope. + +If an application deliberately captures the same external value into another Scope, the target receives a new +Scope-owned Source observation. Its provenance may cite the origin Scoped SourceRef, but the original Source is not +moved and the two SourceKeys are not made identical. + +## Conformance + +A Source Definition can be supported only after its mandatory contract passes conformance scenarios for: + +- identity normalization and collision rejection; +- identical observation replay; +- conflicting payload rejection for one observation ID; +- several immutable observations under one SourceKey; +- exact old-observation reads after head advancement and deletion; +- digest verification for captured and referenced values; +- referenced-value unavailability and mutation; +- Scope isolation and explicit owner preservation; +- Definition version compatibility and unavailable-definition behavior; and +- explicit registration conflict handling. + +A named projection can be advertised only after conformance verifies deterministic output for exact observations, +schema and version conflict handling, exact SourceRef lineage, and explicit failure when the capability is absent. + +A Connector capability can be advertised only after conformance verifies checkpoint replay, per-item outcome +visibility, durable checkpoint ordering, complete-versus-incomplete run behavior, and the claimed deletion evidence. +Provider-specific behavior is established by its implementation evidence rather than generalized into the standard +contract. + +A remote worker path additionally verifies manifest fingerprint and conflict handling, rejection of unregistered or +schema-invalid observations, exact projection-set validation, durable receipt ordering, and stale checkpoint CAS +rejection across a Server restart. + +# Drawbacks + +- Separating SourceKey, SourceRef, Source head, and Definition version introduces more concepts than one immutable + `(source_type, source_id)` pair. +- Exact SourceRefs retain owner Scope and observation identity, increasing lineage payload size. +- Definition authors must specify canonicalization, provenance, and compatibility instead of relying on arbitrary + metadata. +- Named projections and Connector lifecycle state add contracts that must evolve independently from Source values. +- Referenced Sources are unavailable for providers that expose only current values, so some integrations must retain + captured data. +- Explicit registration requires deployment coordination before a custom Source observation can be accepted. + +# Rationale and alternatives + +## Extend ContentSource into the general integration model + +Adding provider fields to ContentSource would preserve the `POST /v1/sources/content` capture API, but it would keep logical identity, +observation identity, and provenance inside caller conventions. Different integrations would encode incompatible +schemas in metadata, and non-text Source values would still need another model. ContentSource remains a useful +single-observation implementation instead. + +## Use one opaque Source envelope + +A universal JSON payload would make persistence and transport uniform, but would move schema validation and +compatibility into runtime conventions. Definition-owned typed values and provenance make the extension boundary +reviewable and allow consumers to reject unsupported Source types before interpretation. + +## Put an observation digest inside source_id + +An integration can preserve the two-part SourceRef shape by composing logical identity and digest into `source_id`. This +makes immutable capture possible but hides the continuing logical Source from the catalog. Updates, current-head +selection, deletion, and provider identity then become integration-private conventions. The standard model represents +both identities directly. + +## Make SourceRef logical and add a separate ObservationRef + +Two public reference types would make SourceRef logical, but Artifact evidence would need +to reject SourceRef and accept only ObservationRef. Defining SourceRef itself as exact follows the existing ArtifactRef +principle that durable lineage references immutable state. + +## Add hybrid materialization + +A third mode that sometimes reads externally and sometimes falls back to captured data obscures which value is +authoritative and which failures are visible. A captured observation can retain a complete external reference as +provenance. A referenced observation either resolves exactly or fails. + +## Let Parent or Connector identity own Sources + +Scope Parent is organization, and Connector identity is acquisition provenance. Neither is a durable ownership +boundary. Using either would conflict with the Scope organization contract and would make reorganization or +Connector replacement change Source identity. + +# Prior art + +- The [Scope organization and Agent integration design](https://github.com/oceanbase/powercontext/pull/1345) separates + Scope ownership, read sharing, organization, delivery, and observation. This RFC applies the same separation to + Source ownership, identity, exact evidence, and acquisition. +- [Apache OpenDAL OFS RFC-0016](https://github.com/apache/opendal-ofs/blob/main/rfcs/0016_filesystem_architecture.md) + separates namespace authority from access frontends and forbids a frontend from advertising guarantees that the + underlying layers cannot enforce. Source materialization follows the same authority rule. +- [opendalfs](https://github.com/fsspec/opendalfs) exposes OpenDAL services through the fsspec interface and + demonstrates backend-neutral filesystem acquisition. Its paths and file metadata do not define Source identity or + immutable revision semantics. A backend read can satisfy referenced materialization only when the complete stack + addresses and verifies an immutable revision. +- DataHub stateful ingestion separates connector checkpoints and stale-entity detection from emitted metadata + identity. Airbyte treats connector state as an opaque recovery boundary rather than record identity. +- OpenMetadata separates the Source that emits records from connection checks, workflow status, and the sink. +- Nowledge Mem's TiddlyWiki importer uses stable logical IDs, canonical payload digests, source revalidation, and + per-item outcomes. Those behaviors inform the separation between Source observations and Connector run state. + +## TencentDB-Agent-Memory research (research input) + +This section is non-normative research input carried forward from #1388; it does not add a second Source contract. + +This RFC is informed by [TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory): + +- MemoryKnowledge routes source protocols through ISourceFetcher and SourceFetcherRegistry. +- Git synchronization returns a commit hash and stores repo_url, branch, version, and last_sync_at. +- Wiki source files use filename plus sha256 for incremental change detection and track ingest status separately. +- Wiki and CodeGraph maintain asset-level version counters and audit rows for sync lifecycle events. +- MemoryCore separates knowledge metadata from content/indexing and uses versioned records for memory evolution. + +These patterns are useful for connector boundaries, provider revision metadata, content hashing, incremental sync, and +operational audit. TencentDB-Agent-Memory treats a changed hash as a reason to pull and ingest the new value; retaining +the prior source text is not its primary contract. PowerContext has a different requirement: when an Artifact already +cites that text, the prior payload must remain available after the current Source changes. PowerContext therefore adopts +the metadata and fetcher separation, but adds a retained immutable snapshot for Artifact citations. + +References: + +- [SourceFetcher types](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/source-fetcher/types.ts) +- [SourceFetcher registry](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/source-fetcher/registry.ts) +- [Git fetcher](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/source-fetcher/git-fetcher.ts) +- [Wiki source index](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/engines/wiki/index-db.ts) +- [Knowledge metadata schema](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/db/client.ts) +- [CodeGraph service](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/store/code-graph-service.ts) + +# Unresolved questions + +- Must every durable SourceRef carry `scope_id` directly, or may a canonical scoped envelope contain a local exact + SourceRef while preserving the same fully qualified identity? +- Which Source Definition versions must a Runtime retain simultaneously before a Definition can be considered + supported? +- Should Source head deletion be one common catalog state, or should the standard contract expose only an + active exact head and leave deletion entirely to Connector state? +- Which projection names and schemas have enough implementation evidence to become shared standards rather than + namespaced capabilities? + +# Future possibilities + +Explicit plugin discovery and deployment policy may build on Definition and Connector registration without making +package installation equivalent to activation. + +Retention policies may reclaim captured values only after defining how exact Artifact evidence reports unavailable +content and how legal or user-requested deletion interacts with immutable lineage. A Source head deletion alone does +not authorize evidence removal. diff --git a/docs/zh/rfcs/1388_source_integration_shape.md b/docs/zh/rfcs/1388_source_integration_shape.md new file mode 100644 index 000000000..738c66731 --- /dev/null +++ b/docs/zh/rfcs/1388_source_integration_shape.md @@ -0,0 +1,534 @@ +- Proposal Name: `source_definition_and_observation_model` +- Status: Proposed +- Start Date: 2026-08-27 +- RFC PR:[oceanbase/powercontext#1388](https://github.com/oceanbase/powercontext/pull/1388) +- Tracking Issue:[oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240) +- Related Discussion: [oceanbase/powercontext#1240](https://github.com/oceanbase/powercontext/issues/1240), + [oceanbase/powercontext#1363](https://github.com/oceanbase/powercontext/issues/1363) +- Related Design: [oceanbase/powercontext#1345](https://github.com/oceanbase/powercontext/pull/1345) +- Related RFCs: [RFC 0002](0002_core_sdk_product_model.md)、[RFC 0014](0014_memory_layer_design.md)、 + [RFC 0019](0019_local_source_memory_runtime.md)、[RFC 0048](0048_handoff_artifact.md) + +# Summary + +本 RFC 定义标准 Source 模型,以及新增 Source 类型时必须遵守的契约。 + +每个 Source 只属于一个 Scope。在该 Scope 内,`SourceKey` 标识一个逻辑 Source,`SourceRef` 标识这个 +Source 的一次不可变观察。推进当前观察、观察到删除、修改外部 locator 或断开 Connector,都不会改变 +已经接受的观察,也不会将其移动到另一个 Scope。 + +Source Definition 为一个稳定的 Source 类型定义 value schema、provenance schema、身份规则、观察规则、 +materialization 契约、canonicalization 与兼容策略。Definition 显式注册,并在组合完成的 Runtime 生命周期内 +保持不变。持久化、传输与 Artifact consumer 按稳定的 Definition 名称和版本路由,而不是按具体 Python 类路由。 + +Definition 可以为无法理解 native value 的 consumer 声明 named projection capability。每个 projection 拥有独立 +版本的 schema,并对一个精确 observation 具有确定语义。Consumer 按 capability name 与 version 选择 projection, +而不是检查具体 Source class。 + +Connector lifecycle 将 provider acquisition 绑定到 Scope,在 worker 内解析 definition-native input,提交 +materialized observation,记录 per-item outcome,并且只在接受的 observation 已持久化后推进 opaque checkpoint。 +Connector run 区分 complete discovery 与 incomplete discovery,避免把缺失对象静默转换为删除。 + +Materialization 表达解析某个精确观察时所依赖的权威来源。Captured observation 从 PowerContext 保留的 +canonical value 解析;referenced observation 从外部不可变 revision 解析。仅有外部 locator、修改时间、 +ETag 或 provider 当前值读取,并不能满足 referenced 契约。 + +`ContentSource` 继续作为简单的 captured-text Source。调用方提供稳定身份,加上 immutable-payload 冲突规则, +适合一次性内容捕获,但它不是通用的外部集成模型。 + +本 RFC 定义 Source、projection、Connector lifecycle,以及 worker 与 PowerContext Server 之间的远程摄取边界。 +它不定义插件发现、scheduler、credential transport、具体 Source family 或 Connector 实现。 + +# Motivation + +`ContentSource` 与 `POST /v1/sources/content` 提供 captured-text ingestion。调用方选择一个 +`source_id`;使用完全相同的 payload 重放具有幂等性,而用不同 payload 复用该身份会产生冲突。只有调用方把 +这个身份当作不可变身份时,它才能表达精确证据。 + +外部系统通常具有不同的生命周期。Wiki 页面、issue、object、message 或 file 拥有一个逻辑身份,但会随时间 +产生多个值。外部对象可能被重命名、修订、删除、恢复或暂时无法读取。使用过旧值的 Artifact 必须继续引用当时的 +精确证据。二元 `(source_type, source_id)` Source reference 无法同时表达稳定的逻辑对象和不可变观察,只能迫使每个集成自行发明复合 +`source_id`。 + +扩展边界也不完整。Source adapter 将 native input class 绑定到具体 Source class 和读取结果,而内置 +Runtime 与关系型持久化会组装固定 adapter 集合。它没有说明独立定义的 Source 类型在身份、持久化、传输与 +Artifact evidence 上必须长期满足哪些规则。 + +标准模型必须回答六个问题,且不能把它们压进同一个 identifier: + +1. 哪个 Scope 拥有这份证据? +2. 它描述哪个逻辑上的外部或内部 Source? +3. Artifact 使用的是哪个精确观察值? +4. PowerContext 从哪里读取该精确值? +5. 哪个 Definition 赋予 value 与 provenance 语义? +6. Consumer 可以使用哪个 declared view,而不必理解 native value? + +Connector concerns 与此相邻但不同。Discovery、credentials、filtering、checkpoints、retries、provider +change handling 与 deletion detection 决定提交哪些观察;它们不定义 Source identity,不能削弱精确证据, +也不能改变 Scope ownership。 + +# Guide-level explanation + +## Domain model + +理解该模型时,依次确定 ownership、logical identity、exact observation、materialization authority 与 type +semantics: + +| Concept | Representation | Question answered | +| --- | --- | --- | +| Ownership | Scope | Source 属于哪里? | +| Logical identity | `SourceKey` | 这是哪个持续存在的 Source? | +| Exact evidence | `SourceRef` | 引用的是哪个不可变观察? | +| Read authority | materialization | 从哪里解析该精确值? | +| Type semantics | Source Definition | 如何解释 value、provenance 与 identity? | +| Consumer view | named projection | Consumer 可以使用哪个 declared representation? | +| Acquisition | Connector or direct caller | 如何发现并提交新观察? | + +这些职责形成单向依赖: + +```text +Connector or direct caller + | + v +Source Definition + | + v +Scope-owned Source history + | + +---- mutable head selection + | + `---- exact SourceRef ----> Artifact evidence +``` + +一个 Connector 可以使用一个 Source Definition,多个 Connector 可以共用同一个 Definition,直接调用方也可以 +在没有 Connector 的情况下提交 Source。因此 Connector identity 不会成为 Source type identity。 + +## Scope ownership + +每个 SourceKey 与 observation 都只属于一个 Scope。Scope ownership 不从外部 workspace、path、repository、 +provider account、Connector instance 或 Source locator 推导。这些值可以参与 binding 或 provenance,但不能 +分配或替代 `scope_id`。 + +完整限定的逻辑身份为: + +```text +SourceKey = (scope_id, source_type, source_id) +``` + +完整限定的精确身份为: + +```text +SourceRef = (scope_id, source_type, source_id, observation_id) +``` + +Scope-bound operation 可以从固定 request binding 获得 `scope_id`,而不把它作为任意参数接收。持久化的解析结果 +仍然保留 owner Scope,使证据在 publication、reporting 或 export 后仍无歧义。 + +修改 Scope Parent、Context References、Agent binding 或 observation selection,都不会改变 SourceKey 或 +SourceRef。跨 Scope 发布 Artifact 时,provenance 保留原始 Scope 与精确 SourceRef;不会移动或隐式复制 +Source history。 + +## Logical Source and immutable observation + +`source_id` 在一个 `(scope_id, source_type)` namespace 内命名逻辑 Source,其含义由 Source Definition +规定。它可以对应 provider object ID、稳定 import identity 或其他 normalized key。观察到新值时,它不能静默改变。 + +`observation_id` 在一个 SourceKey 下命名一次不可变观察。对通用 PowerContext component 而言它是不透明的; +可以派生自 provider revision、canonical value digest 或 Definition 特有组合。它不隐含整数序列、时间顺序或祖先关系。 + +适用以下不变量: + +- 一个 `(SourceKey, observation_id)` 永远标识同一个 canonical observation; +- 再次观察相同 canonical observation 具有幂等性; +- 不同 canonical observation 不能复用 observation ID; +- 如果 identity-bearing provenance 不同,同一个 SourceKey 下可以有 value digest 相同的多个 observation; +- value digest 相同的 observation 不会自动成为同一个逻辑 Source; +- Artifact 只引用精确 SourceRef,绝不引用移动的 SourceKey 或 `latest` observation。 + +例如,更新一个逻辑 Source 会保留其 SourceKey,并产生新的 SourceRef: + +```text +SourceKey(scope-a, record, provider-object-42) +|-- SourceRef(..., observation-1) "Initial value" +`-- SourceRef(..., observation-2) "Revised value" +``` + +即使 `observation-2` 已成为 current,派生自 `observation-1` 的 Artifact 仍然引用后者。 + +## Source Definition + +Source Definition 是一个 `source_type` 的持久语义契约。它声明: + +- 稳定的 Definition name 与 version; +- Source value 与 typed provenance 的结构; +- Source ID normalization 与 equality; +- observation ID normalization 与 equality; +- identity-bearing fields 与 non-identifying annotations; +- canonical bytes 与 value digest algorithm; +- 支持的 materialization modes 与 exact-read requirements; +- limits 与 validation failures; +- older Definition versions 的 compatibility rules。 + +Definition 将 definition-native input 解析为 canonical observation,并从精确的 persisted observation 读取 +Definition 拥有的 value。解析不会选择 Scope、修改 catalog、推进 head 或发现外部 object;读取不会解析 +`latest`,也不会替换为另一个 observation。 + +Definition 必须显式且类型化。新的集成不能通过在 `ContentSource.metadata` 中放置未声明 schema 来模拟新 +Source 类型。Provider-specific provenance 可以扩展 Definition 声明的 schema,但影响 identity、exactness +或 compatibility 的字段必须由 Definition 命名。 + +## Named projection capabilities + +Named projection 是一个 exact observation 的可选 Definition-owned view。它让 Artifact family 或其他 consumer +无需理解 native Source value 或具体 Python class,就能使用声明过的 representation。 + +Projection 通过稳定的 name 与 version 选择。其 Definition 声明 output schema、canonicalization、digest rules +与 failures。Projection 针对精确 SourceRef 求值,不能解析 head、`latest` 或 provider current value。对于相同的 +Definition version、projection version 与 exact observation,它必须返回相同的 canonical result。 + +Projection capability 必须显式声明。需要某个 projection 的 consumer 会拒绝未声明兼容 capability 的 Source, +而不会从 metadata 推断 content,也不会回退到形态相似的 Source class。Projection 可以作为 derivative 被缓存或 +持久化,但其 authority 仍是 exact Source observation,lineage 保留对应 SourceRef。 + +本契约不规定标准 projection name 或 payload schema 的目录。只有当多个 Definition 与 consumer 的互操作证明其 +语义稳定后,projection 才成为 shared standard。在此之前,Definition 可以暴露 namespaced projection,但不会让 +它成为其他 Source type 的 mandatory capability。 + +## Materialization authority + +Materialization 回答精确 SourceRef 的返回值来自哪里: + +| Materialization | Authority | Required guarantee | +| --- | --- | --- | +| `captured` | PowerContext 保留的 canonical value | 保留值与 observation digest 一致 | +| `referenced` | Immutable external revision | 重读 reference 得到相同 canonical value 与 digest | + +Captured Source 可以把 external locator、provider revision 与 digest 保留为 provenance。因为读取权威仍是 +保留值,所以它依然是 captured。这覆盖了 hybrid design 中有价值的部分,而不引入 fallback 语义含糊的第三种模式。 + +只有当外部系统及其 reader 能够寻址不可变历史值时,Definition 才能使用 referenced materialization。读取 +path、page ID、issue ID 或 URL 的当前值并不足够。Modification time 与 ETag 可以参与 provenance 或 conflict +detection,但 Definition 必须说明 provider 是否保证它们指向不可变值。 + +Referenced value 不可用或 digest 不同时,精确解析失败。PowerContext 不返回 provider 当前值、stale cache +entry 或其他 observation。不能满足该规则的 provider 必须使用 captured materialization,或者拒绝该 observation。 + +## Current head and deletion + +Source history 不可变;current head 是可变的 catalog selection。Head 可以选择一个精确 SourceRef,或记录已 +明确观察到逻辑 Source 被删除。Head 可用于 current-state query 与后续 acquisition,但它不是 evidence,不能 +出现在 Artifact citation 中。 + +推进或删除 head 不改变任何 observation。Timeout、permission failure、incomplete listing、Connector +unavailable 或 disconnect 都不是明确的 deletion evidence,不能改变 head。只有当 deletion 本身是有意义的 +Source evidence 时,Source Definition 才可以定义 tombstone value;通用 head deletion 不会伪造这种值。 + +## ContentSource + +`ContentSource` 继续作为 RFC 0019 定义的 neutral captured-text path。调用方选择一个只能与一个 canonical +payload 一起提交的身份。Persistence conflict rule 使接受后的 ContentSource 可作为精确证据,但它不提供独立的 +logical Source lifecycle。 + +标准模型把它视为有效的 single-observation Source: + +- 现有 identity 保持不可变; +- 相同内容重放继续保持幂等; +- 同一 identity 下的不同内容继续发生冲突; +- 解析 ContentSource 的 reference 保持精确且不变; +- 不从 metadata 推导 mutable head 或 multi-observation behavior。 + +ContentSource 适合 prompt、显式文本捕获、import record,以及调用方已经拥有不可变身份的其他场景。持续观察同一 +逻辑对象的集成应定义或复用 multi-observation Source type。 + +# Reference-level explanation + +## Source identity contract + +`scope_id` 是 Scope organization design 定义的 ownership boundary。`source_type` 是稳定的 Source Definition +name。`source_id` 是非空的 normalized identifier,其 equality 与 bounds 由 Definition 声明。 + +Source identity 以 Scope 为本地边界。两个 Scope 可以包含等价的外部材料,但不共享 ownership 或 identity。 +需要避免碰撞时,Definition 可以在 `source_id` 规则中包含稳定的 external instance 或 connection discriminator, +但 discriminator 不替代 `scope_id`。 + +Rename 行为由 Definition 决定。Provider object ID 可以在 locator 变化时保留 SourceKey;path-derived identity +通常把 rename 视为一次逻辑 deletion 与一次 creation。当 provider 与 acquisition path 无法证明 rename-stable +identity 时,Definition 不能宣称支持它。 + +## Observation contract + +Observation 包含以下标准字段: + +```text +SourceObservation +|-- source_key +|-- observation_id +|-- definition_version +|-- materialization +|-- value_digest +|-- provenance +`-- definition-owned value or exact external reference +``` + +`value_digest` 对 Definition 声明的 canonical bytes 使用 SHA-256,并编码为 `sha256:`。对结构化 +value,Definition 指定 deterministic canonicalization。Digest 用于验证 value equality,不替代 SourceKey +或 observation identity。 + +Canonical observation 包含所有被 Definition 认定会影响 identity 或 exact meaning 的字段。Retry count、 +last scan time 或 processing status 等 operational facts 不是 Source value,不改变 observation identity。 +如果 timestamp 或 provider attribute 会影响 provenance meaning,Definition 必须显式分类并 canonicalize。 + +## Source reference contract + +SourceRef 标识精确 observation,并包含 owner Scope。它不接受缺失 observation ID、`latest`、head version 或 +current provider locator。 + +在 scope-bound operation 内,只有当 current Scope 固定且解析出的 durable value 会恢复 `scope_id` 时,紧凑的 +local representation 才可以省略重复的 `scope_id`。跨越 Scope boundary、离开 Runtime 或进入 durable +cross-Scope provenance 的 reference 必须显式携带 owner Scope。 + +Reference resolution 会验证全部四个 identity components,以及 stored observation 的 Definition version 与 +digest。无法解析精确 observation,不等同于 logical Source 已删除、head 已推进或 Connector 不可用。 + +## Definition registration contract + +Executable Definition 属于 worker。Worker 用它解析 definition-native input、canonicalize Source value,并计算 +named projection。Server 不导入 Connector 或 Definition package,也不执行其中的 Python class。 + +提交 observation 前,worker 注册不可变的声明式 manifest。Manifest 包含稳定的 Definition name 与 version、 +canonical Source JSON Schema、每个 projection key 与 output JSON Schema,以及覆盖完整声明的 fingerprint。 +Fingerprint 是 RFC 8785 canonical JSON 的 SHA-256。相同 manifest 的注册是幂等的;同一个 +`(source_type, definition_version)` 对应不同声明时必须拒绝。 + +Server 验证 manifest schema,以及其识别为 shared standard 的 named projection。Manifest 不传输可执行的 identity +rule、canonicalization code、read behavior、credential 或 provider configuration;这些仍由 worker 持有。 +注册后的 manifest 足以让 Server 在不加载 plugin code 的情况下验证并保存 opaque canonical observation。 + +Definition discovery 与 registration 相互独立。Package entry point 或其他 discovery mechanism 可以报告 +可用 Definition,但安装不意味着激活。本 RFC 不选择 entry points、central settings format、pluggy 或 +Connector marketplace。 + +## Remote worker ingestion contract + +Connector 在独立 worker 进程中运行。Worker 拥有 provider access 与所有 executable Definition behavior;Server +拥有 durable Source history、Artifact consumption 与 checkpoint comparison。双方的数据面交互只有四个通用操作: + +1. 注册不可变的 Source Definition manifest; +2. 读取一个 Connector binding 的 opaque checkpoint; +3. 提交 worker 已物化的 Source observation 及其全部声明 projection; +4. 从 run 开始时读到的值 compare-and-swap binding checkpoint。 + +Observation envelope 携带 Definition name、version 与 fingerprint、canonical Source payload,以及 manifest 声明的 +每个 projection value。Server 在 durable acceptance 前验证 envelope identity、payload schema、projection key +集合相等、projection schema 与标准 projection invariant。Provider name、storage service、path、credential 或其他 +Connector-specific configuration 不出现在该 API 中;只有 Definition 刻意将其声明为 canonical Source schema 的 +一部分时才例外。 + +Server 必须先返回 durable Source receipt,worker 才能提交 checkpoint。Checkpoint operation 使用 optimistic +comparison,防止同一 binding 的并发 run 静默覆盖。相同 Source identity 与 payload 的提交是幂等的;已接受 identity +对应不同内容时必须拒绝。 + +## Definition compatibility contract + +Definition name 在兼容 schema 演进中保持稳定。每个 persisted observation 记录验证和 canonicalize 它时使用的 +Definition version。新的 Definition version 必须声明如何在不改变 canonical meaning 的前提下读取旧 observation, +或与旧版本 reader 共存。 + +如果 Definition change 会改变已接受 observation 的 SourceKey equality、observation equality、canonical value +bytes、provenance meaning 或 materialization guarantee,它就是不兼容变更。此类变更需要新的 Definition version, +且不能重写已有 SourceRef。 + +如果 projection change 会改变已接受 observation 的 output schema、canonical bytes 或 meaning,它就是不兼容 +变更,需要新的 projection version。如果 Source value 与 observation semantics 保持不变,则不要求新的 Source +Definition version。 + +重命名 Definition 会产生新的 `source_type`。把已有 observation 重新分类到另一个 Definition 是带 provenance +的显式 derivation,不是 identity 的原地 migration。 + +## Connector lifecycle contract + +Connector 负责 provider interaction:discovery、credentials、filtering、checkpoints、retries、rate limits、 +provider change handling 与 positive deletion detection。它依据 Scope binding 提交 definition-native input, +并接收接受后的精确 SourceRef。 + +Source Definition 负责 semantic normalization:logical identity、observation identity、canonical value、 +provenance、materialization validity 与 exact read。Connector 不能覆盖这些规则。如果 provider capabilities、 +Connector behavior 与 Definition requirements 的交集无法满足选定 materialization,则拒绝 observation,或在 +合法模式下 captured。 + +```text +provider capabilities + intersect Connector behavior + intersect Source Definition requirements + = valid Source observation +``` + +Connector type 声明稳定的 name 与 version、configuration schema、可提交的 Source Definition,以及它提供的 +acquisition capability。Capability 是可选且显式的,通常包括 complete snapshot、change feed、checkpoint resume +和 authoritative deletion event。Connector 不能声明 provider 与 acquisition path 无法兑现的 capability。 + +Connector binding 为一个 Scope 激活一份 Connector configuration。Binding 拥有用于 checkpoint 与 provider +namespace continuity 的稳定 identity,但不拥有 Source,也不替代 `scope_id` 或 `source_type`。Credential 由 +hosting environment 解析,不会成为 Source value 或 provenance。 + +Connector run 从 opaque binding checkpoint 开始,在 worker 内解析零个或多个 definition-native input,再提交其 +materialized observation,并记录每个 item 的 outcome。Accepted 或 idempotently replayed observation 返回精确 +SourceRef。Rejected 或 failed item 会保留在 run outcome 中;如果尚不能安全重放,checkpoint 不能越过这些工作。 + +Run 以 complete 或 incomplete 结束。Complete snapshot 可以为之前已知但本次缺失的 provider object 产生 positive +deletion evidence。Incomplete listing、timeout、permission failure、cancellation 或 lost connection 不会产生 +absence-based deletion evidence。当 binding 与 object identity 均已验证时,authoritative provider deletion event +可以独立于 snapshot completeness 产生 positive deletion evidence。 + +Completed checkpoint 只有在 accepted observation 与 deletion evidence 均已持久化后才能推进。由于 Source +observation submission 具有幂等性,从更早 checkpoint 重试是合法行为。Connector checkpoint、health、retry 与 +run-status record 是 operational state,而不是 Source observation 或 Artifact evidence。 + +Installation、discovery、activation 与 execution 相互独立。安装 Connector package 不会激活 binding。Connector +package 在 PowerContext Server 之外执行,并使用 remote worker ingestion contract;调度与进程监管属于部署环境。 + +## Artifact evidence and cross-Scope delivery + +Artifact revision 记录其计算直接使用的精确 SourceRef。推进 Source head 不改变现有 Artifact lineage。针对较新 +observation 的重新计算会产生新的 Artifact revision,而不是重写旧 evidence。 + +Source 保留在 producing Scope。Context Reference 可以按照 Scope organization contract 扩展 read selection, +但不会改变 Source ownership。跨 Scope 的精确 Artifact publication 在 lineage 中保留 origin Scope 与精确 +SourceRef。发布 Artifact 不会发布其 origin Scope 中的所有 Source。 + +如果 application 刻意把同一个外部值 captured 到另一个 Scope,target 会得到由该 Scope 拥有的新 Source +observation。其 provenance 可以引用 origin scoped SourceRef,但原始 Source 不会移动,两个 SourceKey 也不会 +因此变成同一 identity。 + +## Conformance + +Source Definition 只有在以下 mandatory contract 的 conformance scenario 通过后才能被支持: + +- identity normalization 与 collision rejection; +- identical observation replay; +- 同一 observation ID 的 conflicting payload rejection; +- 一个 SourceKey 下的多个 immutable observation; +- head advancement 与 deletion 后仍能精确读取旧 observation; +- captured 与 referenced value 的 digest verification; +- referenced-value unavailability 与 mutation; +- Scope isolation 与显式 owner preservation; +- Definition version compatibility 与 unavailable-definition behavior; +- explicit registration conflict handling。 + +Named projection 只有在 conformance 验证 exact observation 的 deterministic output、schema 与 version conflict +handling、exact SourceRef lineage,以及 capability 缺失时显式失败之后才能被声明。 + +Connector capability 只有在 conformance 验证 checkpoint replay、per-item outcome visibility、durable checkpoint +ordering、complete-versus-incomplete run behavior,以及其声明的 deletion evidence 后才能被声明。Provider-specific +behavior 由对应实现证据确定,不会被直接推广为标准契约。 + +Remote worker path 还必须验证 manifest fingerprint 与 conflict handling、拒绝未注册或 schema-invalid observation、 +projection set 精确校验、durable receipt ordering,以及跨 Server restart 的 stale checkpoint CAS rejection。 + +# Drawbacks + +- 分离 SourceKey、SourceRef、Source head 与 Definition version,比一个不可变的 `(source_type, source_id)` pair + 引入更多概念。 +- 精确 SourceRef 保留 owner Scope 与 observation identity,会增加 lineage payload 大小。 +- Definition author 必须声明 canonicalization、provenance 与 compatibility,而不能依赖任意 metadata。 +- Named projection 与 Connector lifecycle state 增加了需要独立于 Source value 演进的契约。 +- 只暴露当前值的 provider 无法使用 Referenced Source,因此部分集成必须保留 captured data。 +- Custom Source observation 被接受之前,显式 registration 需要部署协调。 + +# Rationale and alternatives + +## Extend ContentSource into the general integration model + +向 ContentSource 添加 provider fields 可以保留 `POST /v1/sources/content` capture API,但仍会把 logical identity、observation identity +与 provenance 留在调用方约定中。不同集成会在 metadata 中编码不兼容 schema,non-text Source value 仍需要另一 +套模型。因此 ContentSource 继续作为有用的 single-observation implementation。 + +## Use one opaque Source envelope + +通用 JSON payload 可以统一 persistence 与 transport,但会把 schema validation 和 compatibility 推给 runtime +convention。Definition-owned typed value 与 provenance 让扩展边界可审查,并允许 consumer 在解释前拒绝不支持的 +Source type。 + +## Put an observation digest inside source_id + +集成可以把 logical identity 与 digest 组合进 `source_id`,从而维持二元 SourceRef 形态。这可以表达 immutable +capture,却会在 catalog 中隐藏持续存在的 logical Source。Update、current-head selection、deletion 与 provider +identity 都会变成 integration-private convention。标准模型直接表达两类 identity。 + +## Make SourceRef logical and add a separate ObservationRef + +两个 public reference type 可以让 SourceRef 表示逻辑身份,但 Artifact evidence 必须拒绝 SourceRef,只接受 +ObservationRef。让 SourceRef 本身保持精确,符合现有 ArtifactRef 原则:durable lineage 引用 immutable state。 + +## Add hybrid materialization + +增加一种有时从外部读取、有时回退到 captured data 的第三种模式,会掩盖哪个 value 才是 authoritative,以及哪些 +failure 应对外可见。Captured observation 可以把完整 external reference 保留为 provenance;referenced +observation 要么被精确解析,要么失败。 + +## Let Parent or Connector identity own Sources + +Scope Parent 用于 organization,Connector identity 是 acquisition provenance,二者都不是持久 ownership +boundary。使用其中任意一个都会与 Scope organization contract 冲突,并让 reorganization 或 Connector +replacement 改变 Source identity。 + +# Prior art + +- [Scope organization and Agent integration design](https://github.com/oceanbase/powercontext/pull/1345) 分离 + Scope ownership、read sharing、organization、delivery 与 observation。本 RFC 对 Source ownership、identity、 + exact evidence 与 acquisition 应用同样的分离原则。 +- [Apache OpenDAL OFS RFC-0016](https://github.com/apache/opendal-ofs/blob/main/rfcs/0016_filesystem_architecture.md) + 分离 namespace authority 与 access frontend,并禁止 frontend 宣称底层无法兑现的保证。Source materialization + 遵循同样的 authority rule。 +- [opendalfs](https://github.com/fsspec/opendalfs) 通过 fsspec interface 暴露 OpenDAL services,展示了 + backend-neutral filesystem acquisition。它的 path 与 file metadata 不定义 Source identity 或 immutable + revision semantics。只有完整调用链能够寻址并验证不可变 revision 时,backend read 才能满足 referenced + materialization。 +- DataHub stateful ingestion 把 connector checkpoint 与 stale-entity detection 同 emitted metadata identity + 分离。Airbyte 把 connector state 当作 opaque recovery boundary,而不是 record identity。 +- OpenMetadata 把负责生成 record 的 Source 与 connection check、workflow status、sink 分离。 +- Nowledge Mem 的 TiddlyWiki importer 使用 stable logical ID、canonical payload digest、source revalidation 与 + per-item outcome。这些行为为 Source observation 与 Connector run state 的分离提供依据。 + +## TencentDB-Agent-Memory 调研参考(research input) + +本节是从 #1388 合并而来的非规范性调研输入,不增加第二套 Source contract。 + +本设计参考了 [TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory): + +- MemoryKnowledge 通过 ISourceFetcher 和 SourceFetcherRegistry 路由 source protocol; +- Git 同步返回 commit hash,并保存 repo_url、branch、version、last_sync_at 等 metadata; +- Wiki source file 使用 filename 加 sha256 做增量变化检测,并独立跟踪 ingest status; +- Wiki 和 CodeGraph 使用 asset-level version counter 与 audit row 记录同步生命周期; +- MemoryCore 将 knowledge metadata 与 content/indexing service 分离,并用 versioned record 管理 memory 演进。 + +这些做法适合借鉴 connector 边界、provider revision metadata、content hashing、增量同步和运维审计。 +TencentDB-Agent-Memory 把 hash 变化视为重新拉取并 ingest 最新值的信号,保留旧 source 原文不是它的主要 contract。 +PowerContext 的要求不同:当 Artifact 已经引用这份原文时,当前 Source 变化后旧 payload 仍必须可用。因此 PowerContext +采用 metadata 与 fetcher 分离,同时为 Artifact citation 增加保留的 immutable snapshot。 + +上游关键参考文件: + +- [SourceFetcher types](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/source-fetcher/types.ts) +- [SourceFetcher registry](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/source-fetcher/registry.ts) +- [Git fetcher](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/source-fetcher/git-fetcher.ts) +- [Wiki source index](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/engines/wiki/index-db.ts) +- [Knowledge metadata schema](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/db/client.ts) +- [CodeGraph service](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/5299c00aaf65481703c180fd69df066d11254eb7/MemoryKnowledge/src/store/code-graph-service.ts) + +# Unresolved questions + +- 每个 durable SourceRef 是否必须直接携带 `scope_id`,还是可以由 canonical scoped envelope 包含 local exact + SourceRef,同时保留相同的 fully qualified identity? +- Runtime 必须同时保留哪些 Source Definition version,才能宣称某个 Definition 受支持? +- Source head deletion 应是通用 catalog state,还是标准契约只暴露 active exact head,并把 deletion 完全留给 + Connector state? +- 哪些 projection name 与 schema 已有足够实现证据,可以成为 shared standard 而不是 namespaced capability? + +# Future possibilities + +显式 plugin discovery 与 deployment policy 可以建立在 Definition 和 Connector registration 之上,但不会让 +package installation 等同于 activation。 + +Retention policy 只有在定义精确 Artifact evidence 如何报告 unavailable content,以及 legal/user-requested +deletion 如何与 immutable lineage 交互之后,才能回收 captured value。Source head deletion 本身不授权删除证据。 diff --git a/zensical.toml b/zensical.toml index 3365122da..520197b30 100644 --- a/zensical.toml +++ b/zensical.toml @@ -64,6 +64,7 @@ nav = [ { "RFCs" = [ { "Overview" = "en/rfcs/README.md" }, { "1229 Unified Workloads and Long-Horizon Memory Evaluation" = "en/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md" }, + { "1388 Source Integration Shape" = "en/rfcs/1388_source_integration_shape.md" }, { "1223 Human-Agent Work Continuity" = "en/rfcs/1223_human_agent_work_continuity.md" }, { "0082 Handoff Report" = "en/rfcs/0082_handoff_report.md" }, { "0081 End-to-end Evaluation Architecture" = "en/rfcs/0081_end_to_end_evaluation_architecture.md" }, @@ -145,6 +146,7 @@ nav = [ { "RFC" = [ { "概览" = "zh/rfcs/README.md" }, { "1229 统一工作负载与长程 Memory 评估" = "zh/rfcs/1229_unified_workloads_and_long_horizon_memory_evaluation.md" }, + { "1388 Source 集成形态" = "zh/rfcs/1388_source_integration_shape.md" }, { "1223 人与 Agent 工作连续性" = "zh/rfcs/1223_human_agent_work_continuity.md" }, { "0082 Handoff 报告" = "zh/rfcs/0082_handoff_report.md" }, { "0081 端到端评估架构" = "zh/rfcs/0081_end_to_end_evaluation_architecture.md" },