Skip to content

bug: Memory append storage and latency grow superlinearly with entry history #1321

Description

@AlexStocks

Describe the bug

Repeated explicit Memory appends have superlinear persistent-storage growth and steadily increasing write latency, even when each entry contains only a few bytes of text.

Every Memory Revision stores the complete current manifest. Appending one entry therefore writes manifests of size 1, 2, 3, ... N across the immutable revision history. Entry bodies are not duplicated, but the cumulative manifest directory approaches quadratic growth.

There is currently no maximum entry count, manifest-byte budget, automatic Memory split, or tombstone compaction boundary. The unpaginated list_memory_entries path also materializes the complete current manifest and all referenced entry versions.

The Memory RFC identifies split and compaction thresholds as future work, but the current service can grow past practical limits without an explicit error or degradation signal.

Steps to reproduce

At commit a6e0dfe:

  1. Create a fresh SQLite-backed Server with Dashboard, MCP, metrics, and access logging disabled.
  2. Repeatedly call POST /v1/memory/remember in one scope with one short unique fact per request.
  3. Close the Server.
  4. Run PRAGMA wal_checkpoint(TRUNCATE) and VACUUM.
  5. Record the database size and average duration of the final 20 writes.
  6. Repeat with 50, 100, and 200 entries.

Observed:

Entries SQLite database bytes Average final 20 writes
50 671,744 40.57 ms
100 1,634,304 65.21 ms
200 5,222,400 111.81 ms

The test body was only bounded fact NNNN.

Relevant paths:

  • src/powercontext/builtin/artifacts/memory/models.py:62-65
  • src/powercontext/builtin/artifacts/memory/service.py:699-731
  • src/powercontext/builtin/persistence/artifacts.py:243-262
  • src/powercontext/builtin/persistence/memory.py:157-175
  • docs/en/rfcs/0014_memory_layer_design.md, especially the deferred split/compaction thresholds

Expected behavior

A long-lived Memory service needs an explicit, observable capacity contract.

At minimum:

  • Enforce a current-manifest entry or byte budget before unbounded growth.
  • Return a stable error or route new entries to another Memory when the budget is reached.
  • Paginate or otherwise bound list_memory_entries and complete-history operations.
  • Define and test split and inactive-tombstone compaction behavior.
  • Add a scale regression test that measures storage and write amplification at increasing entry counts.

The exact split/compaction design may require an RFC, but the current API should not accept growth indefinitely without a budget.

Actual behavior

Each append reconstructs and persists the complete manifest in a new immutable Artifact Revision. Storage and write work increase with the number of prior entries, with no configured ceiling or automatic split.

Environment

  • PowerContext: 0.0.3.dev11+ga6e0dfe8b.d20260823
  • Commit: a6e0dfe
  • Python: 3.14.3
  • Backend: SQLite with WAL, checkpointed and vacuumed before measurement
  • OS: Windows 11
  • External inference and embedding providers: disabled

Are you willing to submit a PR to fix this bug?

  • Yes, I would like to submit a PR.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status
Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions