Skip to content

fix(memory): enforce authoritative revision integrity - #1303

Open
knqiufan wants to merge 2 commits into
oceanbase:masterfrom
knqiufan:fix/issue-1297-memory-integrity
Open

fix(memory): enforce authoritative revision integrity#1303
knqiufan wants to merge 2 commits into
oceanbase:masterfrom
knqiufan:fix/issue-1297-memory-integrity

Conversation

@knqiufan

@knqiufan knqiufan commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Which issue or RFC does this PR close?

Closes #1297.

Rationale for this change

MemoryUnitOfWork.commit() is the final persistence trust boundary, but it previously accepted incomplete or internally inconsistent revisions. A malformed direct SPI writer could persist illegal predecessor links or body/hash mismatches, and search could then return text that authoritative entry reads rejected.

What changes are included in this PR?

  • Validate the stored base, every history row reachable from its manifest heads, and the complete base-to-revision transition inside the commit transaction.
  • Require the manifest, changes, new entry versions, direct predecessors, and active projections to agree exactly.
  • Recompute entry hashes and validate declared/manifest hashes, canonical searchable text, and configured embedding bindings before any write.
  • Enforce scope-global entry_version_id identity with a database unique index, including startup migration and duplicate-data detection for existing SQLite and OceanBase schemas; translate insert races into MemoryBackendConfigurationError with full rollback.
  • Reject semantic revisions of inactive entries while retaining the narrow organize(normalize) path when canonical content bytes are unchanged.
  • Validate dense embeddings by dimension, finite values, and unit-norm tolerance without a second exact floating-point normalization comparison.
  • Revalidate entries(), expand(), projection reads, and every FTS/vector candidate against the exact authoritative revision, manifest, entry row, and current head projection.
  • Add regression coverage for malformed/cross-entry/cross-Memory/skipped predecessors, pre-existing corrupted history, missing/extra/duplicate rows, hash and projection corruption, inactive revisions, dense vectors, migrations, rollback behavior, and read/search/expand agreement.

Are there any user-facing changes?

Malformed commits made through the public Memory backend SPI now fail with MemoryBackendConfigurationError instead of becoming durable state. Existing SQLite and OceanBase schemas receive the scope-global entry-version unique index during startup; startup reports MemoryBackendConfigurationError if legacy duplicate identities must be repaired first. Valid service workflows and the documented no-vector degradation remain unchanged. There are no HTTP API changes.

How was this change tested?

  • uv lock --locked
  • uv run --no-sync prek run -a
  • uv run --no-sync ty check
  • uv run --no-sync python -m pytest tests/builtin/artifacts/memory tests/builtin/persistence/test_memory.py tests/builtin/persistence/test_memory_integrity.py tests/builtin/persistence/test_sqlite_profile.py tests/builtin/persistence/test_oceanbase_profile.py tests/e2e/test_memory_search_concurrency.py tests/e2e/test_builtin_runtime.py -q (60 passed, 7 skipped)
  • Full Windows run: 620 passed, 15 skipped; 8 unrelated pre-existing Windows failures remain in symlink privileges, the unavailable python3 command, POSIX mode bits, encoding, and path-separator assertions.
  • The live two-connection OceanBase race regression is opt-in through POWERCONTEXT_TEST_OCEANBASE_URL; it is skipped when no dedicated OceanBase test database is configured.

AI usage statement

OpenAI Codex (GPT-5) was used to inspect the issue and review feedback, implement the fixes, design regression tests, and review the final change.

@Teingi Teingi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tightening the Memory integrity boundary. The original issue is real and the main reproducer is addressed, but the three cases below still allow revisions outside the documented contract. Requesting changes until these integrity gaps are closed.

Comment thread src/powercontext/builtin/persistence/memory.py
Comment thread src/powercontext/builtin/persistence/memory.py
Comment thread src/powercontext/builtin/persistence/memory.py
except (TypeError, ValueError) as error:
raise _InvalidMemoryCommitError("vector") from error
if canonical != projection.embedding or expected_hash != projection.embedding_content_hash:
raise _InvalidMemoryCommitError("vector")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] 不要用精确浮点相等判断单位向量已经规范化

MemoryService._embed_texts() 已先执行一次 canonical_embedding();这里再次归一化后再做 tuple 精确比较。单位归一化不是位级幂等的:公开 MemoryService.remember() 探针让模型返回 (0.2407121489724894, -0.9705965492093231),服务层得到 (0.24071214897248938, -0.9705965492093231),这里二次归一化又把首项改成 0.24071214897248935,于是合法的有限非零向量被抛为 _InvalidMemoryCommitError("vector"),整个 Memory 写入失败。现有向量测试只使用 (1, 0, 0) 这类轴向量,无法覆盖该舍入路径。请改为校验维度、有限性和带容差的单位范数(或确保只规范化一次并复用结果),并补一个非轴向量的公开 MemoryService.remember() 回归测试。

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

感谢指正。我使用你给出的非轴向量复现并确认了这个问题:二次归一化后的精确浮点比较确实会误拒绝合法写入。现已修复为校验维度、有限性和带容差的单位范数,同时保留 profile/content hash 校验,并补充了公开 MemoryService.remember() 稠密向量回归测试及非单位向量拒绝测试。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Memory commit accepts malformed revisions and search can return text that entries() rejects

3 participants