fix(export): refuse dumps past the transport limit instead of dropping the worker

[dmazhukov]

Summary

Fixes #1142.

Past ~16 MiB of serialized response, GET /agentmemory/export kills the worker↔engine
WebSocket. Export complete is logged, then [iii] Reconnecting attempt 1, and every
endpoint 404s until the worker re-registers about a second later. One GET takes down the whole
REST surface.

Bisected on a local instance: 16 771 046 B returns 200 in 365 ms, one 27 KB step further
returns 500 in 103 ms. That is 6 170 B short of 16 MiB (16 777 216). The failing request is
faster than the succeeding one, so this is a size limit and not a timeout. The limit is not
in this package — iii-sdk opens its socket with no maxPayload (so the ws default of
100 MiB applies), while the engine binary carries tungstenite's WebSocketConfig field names
and tungstenite defaults max_frame_size to exactly 16 MiB. Details and the reproduction are
in the issue.

What this changes

1. mem::export measures the payload and refuses to send one that cannot arrive.
Buffer.byteLength(JSON.stringify(exportData)) against EXPORT_MAX_BYTES (default 15 MiB,
leaving room for framing). Over the ceiling it returns a structured ExportTooLarge, which
api::export maps to 413 with the byte counts and the parameters that get around it.

The guard has to sit inside mem::export, not in the trigger: api::export reaches it
through sdk.trigger, so the result crosses the boundary twice and dies on the first hop —
the HTTP layer never receives the oversized object at all.

2. ?collectionLimit= / ?collectionOffset= page the other collections.
Today ?maxSessions= slices only sessions and their observations; memories,
summaries, and the 16 top-level collections come in full regardless, so on my repro store
?maxSessions=1 still returned 5.50 MB of which ~4.9 MB could not be reduced. That floor
grows with the store, and once it alone passes 16 MiB the export is impossible at any
parameter — the dead end #890 describes. The response reports collectionPagination with
per-collection totals and a combined hasMore so a caller can walk to the end.

3. ?collections= narrows the payload to what the caller actually reads.
An allowlist over the same 18 collections, so an agent after a lesson is not also pulling
graphEdges. sessions, observations and profiles are deliberately outside the
vocabulary — they are windowed by maxSessions/offset and profiles are derived from the
session page. totals still report every collection even when deselected, since that is what
clients read for corpus size, while hasMore considers only the selected ones so a client
that asked for six of eighteen can stop on the flag instead of hand-rolling a comparison
against totals.

4. The refusal survives the trip to every consumer.
src/mcp/server.ts was returning whatever mem::export produced inside a 200, so a refusal
would have shipped as a successful export whose body happens to be an error object. It now
branches on the same shared guard. rest-proxy.ts discarded 4xx bodies and threw with status
text alone, which stranded the caller with 413 and no idea which parameters help; it now
carries a truncated body into the error.

Behaviour

Anything that returns 200 today returns exactly the same bytes. A test/export-import.test.ts
case pins the full-corpus response shape field by field so the change cannot quietly alter it.
The only altered path is the one that currently fails, and it now fails as one request instead
of an outage.

What this doesn't fix

• The 16 MiB frame limit itself. It is enforced by the engine, and I found no config key
  or flag for it. This PR keeps a daemon alive that would otherwise drop its worker; it does
  not make the transport carry more. If the limit is configurable, that is a better fix and
  this guard should follow it.
• Observations stay atomic per session. collectionLimit does not slice them; they follow
  the session window. A single session whose observations alone exceed the ceiling cannot be
  exported at any parameter combination.
• mesh/export (mesh/export has no pagination — backlogs past the worker WS message limit fail forever with "Invocation stopped", making mesh sync unrecoverable after an outage #890) has the same shape and is untouched here.
• The 180 s KV default (mem::graph-extract hangs to its own 180s invocation timeout on every batch, eventually stalling state::get and crashing the worker #1127/fix: give StateKV calls their own short timeout, separate from the 180s LLM-sized worker default #1128). A worker drop leaves an in-flight state::set
  hanging to that ceiling; I saw it on every reproduction. Fewer drops means fewer hangs, but
  the timeout itself is fix: give StateKV calls their own short timeout, separate from the 180s LLM-sized worker default #1128's subject.

Testing

• test/export-import.test.ts: full-corpus shape pinned; refusal returned over the ceiling
  and not under it; collectionLimit bounds every collection while the unparameterised call
  still returns everything; a three-page walk ends with hasMore: false; guard rejects
  look-alike values.
• test/mcp-standalone-proxy.test.ts: a 413 body reaches the reported failure.
• npx vitest run --exclude test/integration.test.ts → 1572 passed, 0 failed.
• npx tsc --noEmit → 25 errors, unchanged from main and none in the touched files.
• npm run build clean.

CI was red on main itself when this was opened — npm run skills:check reported drift in
the generated REST reference and test/consistency.test.ts disagreed with the README's
endpoint count. That refresh is folded in here as its own commit, and stands alone as #1144 if
you would rather take it separately; the two are the same commit, so whichever merges first
makes the other a no-op.

Summary by CodeRabbit

• New Features

  • Added pagination for large exports, including session and collection offsets, limits, totals, and “more available” indicators.
  • Added collection filtering, including support for selecting specific or empty collections.
  • The memory_export tool now accepts pagination and collection-selection options.

• Bug Fixes

  • Oversized exports are refused with a clear error and HTTP 413 status.
  • Proxy error messages now include relevant response details and actionable pagination guidance.

• Documentation

  • Updated export reference documentation with pagination, filtering, and size-limit behavior.

[vercel]

@dmazhukov is attempting to deploy a commit to the rohitg00's projects Team on Vercel.

A member of the Team first needs to authorize it.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The export flow now supports collection selection and pagination, reports collection totals, and enforces a configurable serialized-size limit. REST and MCP responses return HTTP 413 for oversized exports. Proxy errors include truncated response details.

Changes

Export protection

Layer / File(s)	Summary
Export contracts and assembly src/types.ts, src/functions/export-import.ts, test/export-import.test.ts	Exports support collection selection, offsets, and limits. Results include totals and hasMore metadata. Oversized serialized results return ExportTooLarge refusals.
HTTP and MCP export responses src/triggers/api.ts, src/mcp/tools-registry.ts, src/mcp/server.ts, test/mcp-export-tool.test.ts, test/memories-pagination.test.ts, plugin/skills/agentmemory-mcp-tools/REFERENCE.md	REST and MCP handlers validate and forward export parameters. Oversized results return HTTP 413. Tool documentation describes paging, collection selection, and refusal behavior.
Proxy error detail reporting src/mcp/rest-proxy.ts, test/mcp-standalone-proxy.test.ts	Proxy errors include up to 500 characters from the response body. The regression test checks the export-size hint.
Endpoint count updates AGENTS.md, README.md, plugin/skills/agentmemory-rest-api/REFERENCE.md, src/index.ts	Documentation and startup status text report the updated REST endpoint counts.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant api_export as api::export
  participant mem_export as mem::export
  participant collections as Collection reads
  Client->>api_export: Request export with collection selection and pagination
  api_export->>mem_export: Forward validated export parameters
  mem_export->>collections: Read selected collections and totals
  collections-->>mem_export: Paginated data and totals
  mem_export-->>api_export: ExportData or ExportTooLarge
  api_export-->>Client: HTTP 200 or HTTP 413

Loading

Possibly related PRs

• rohitg00/agentmemory#849: Both PRs modify export behavior in src/functions/export-import.ts, src/types.ts, and test/export-import.test.ts.
• rohitg00/agentmemory#1144: Both PRs update REST endpoint counts in project documentation and startup status text.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes address issue #1142 by refusing oversized exports, preserving worker availability, and adding pagination for large collections.
Out of Scope Changes check	✅ Passed	The implementation, tests, MCP updates, and documentation changes support export refusal and pagination; no unrelated code changes are present.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: oversized exports are refused to prevent the worker from failing.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/mcp/server.ts (1)

365-381: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Forward memory_export pagination args to mem::export.

memory_export always calls sdk.trigger({ function_id: "mem::export", payload: {} }), so mem::export ignores pagination and never returns collectionPagination. The REST export handler already reads and forwards maxSessions, offset, collectionLimit, and collectionOffset, while the MCP tool schema still declares an empty properties. Add argument validation and pass the same fields into the payload.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/mcp/server.ts` around lines 365 - 381, Update the memory_export case and
its MCP tool schema to validate and accept maxSessions, offset, collectionLimit,
and collectionOffset, then forward those values in the payload of sdk.trigger
for mem::export. Preserve the existing isExportTooLarge handling and response
formatting while ensuring pagination arguments reach the export function.

🧹 Nitpick comments (1)

src/functions/export-import.ts (1)

176-191: 🚀 Performance & Scalability | 🔵 Trivial

Collection pagination still requires a full kv.list() read per collection.

sliceCollection pages memories, graphNodes, and the other collections only after kv.list() has already fetched every row for that scope. For very large collections (the issue mentions 8K+ memories, 34K observations per session), collectionLimit/collectionOffset shrink the response size but do not reduce the KV read cost behind each collection. This matches the PR's stated goal (avoid oversized WebSocket responses), so it is not a blocking concern, but if a collection ever grows large enough that the full-list read itself becomes slow, pagination here will not help.

Consider tracking this as a follow-up if StateKV.list gains offset/limit support later.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/functions/export-import.ts` around lines 176 - 191, Track this as a
follow-up rather than changing the current export flow: collection pagination in
sliceCollection occurs after each kv.list call and cannot reduce the full KV
read. If StateKV.list later supports offset/limit parameters, update the
collection reads in the export function to pass collectionOffset and
collectionLimit directly while preserving the existing collection mappings and
response slicing behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/mcp/server.ts`:
- Around line 365-381: Update the memory_export case and its MCP tool schema to
validate and accept maxSessions, offset, collectionLimit, and collectionOffset,
then forward those values in the payload of sdk.trigger for mem::export.
Preserve the existing isExportTooLarge handling and response formatting while
ensuring pagination arguments reach the export function.

---

Nitpick comments:
In `@src/functions/export-import.ts`:
- Around line 176-191: Track this as a follow-up rather than changing the
current export flow: collection pagination in sliceCollection occurs after each
kv.list call and cannot reduce the full KV read. If StateKV.list later supports
offset/limit parameters, update the collection reads in the export function to
pass collectionOffset and collectionLimit directly while preserving the existing
collection mappings and response slicing behavior.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 562fdced-a35f-4cde-bb2d-b2080bd80994

📥 Commits

Reviewing files that changed from the base of the PR and between 5023cf3 and 58b2b42.

📒 Files selected for processing (7)

• src/functions/export-import.ts
• src/mcp/rest-proxy.ts
• src/mcp/server.ts
• src/triggers/api.ts
• src/types.ts
• test/export-import.test.ts
• test/mcp-standalone-proxy.test.ts

[dmazhukov]

Heads-up on the red CI here: both failures reproduce on a clean checkout of main and are not from this branch.

• npm run skills:check → DRIFT: plugin/skills/agentmemory-rest-api/REFERENCE.md (records 118, the generator produces 119)
• test/consistency.test.ts → README says 129 endpoints on port, 130 are registered

An endpoint landed in #1132 / #1136 without the generated docs moving with it. I checked that this branch adds no endpoints: npm run skills:gen produces the same 119 on main and here.

#1144 refreshes the four files that carry the count. Once it lands I'll rebase this branch so the matrix reflects the actual change.

[dmazhukov]

Good catch on memory_export — fixed in 712d9ba.

The empty payload: {} predates this PR, but the refusal I added tells the caller to page, and that advice was not actionable from the MCP tool because the schema declared no properties. src/mcp/tools-registry.ts now declares maxSessions, offset, collectionLimit and collectionOffset, and src/mcp/server.ts validates and forwards them with the same integer bounds the REST handler uses. plugin/skills/agentmemory-mcp-tools/REFERENCE.md is regenerated via npm run skills:gen.

test/mcp-export-tool.test.ts covers the four cases: arguments forwarded, unusable bounds dropped, a refusal reported as 413, and a normal export still 200.

🤖 Addressed by Claude Code

[dmazhukov]

Correction to my note above: I've folded the same 4-line refresh into this branch (3966cb8, identical to #1144's commit) rather than waiting.

npm run skills:check runs ahead of the test suite, so leaving it out meant the whole matrix went red at the first step and showed nothing about the change itself. With it in, this branch is green locally: skills:check passes and npx vitest run --exclude test/integration.test.ts gives 1560 passed, 0 failed.

#1144 still stands on its own if you'd rather take the housekeeping separately — it is the same commit, so whichever merges first makes the other a no-op.

🤖 Addressed by Claude Code