Draft: runtime-validated non-destructive reconstruction workflow for ChromaDB 1.x

[apajon]

Summary

This draft PR isolates the reconstruction workflow for rebuilding a ChromaDB 0.6.x palace into a separate ChromaDB 1.x target.

This is still not a claim of universal 1.x support.
This is not an in-place migration path.
This is a non-destructive, explicitly validated reconstruction path with bounded guarantees.

The key change from the earlier framing is this:

reconstructed 1.x targets are now validated not only structurally, but also at real MemPalace MCP runtime level within the tested scope.

In other words, this branch is no longer just an exploration lab.
It now demonstrates a controlled reconstruction workflow that can produce runtime-usable 1.x targets, while preserving the source palace and failing explicitly on invalid inputs.

---

Base / dependency

This branch is currently stacked on top of:

• split/mcp-json-migration

That keeps the reconstruction work aligned with the current stable configuration baseline while remaining isolated from the conservative stable-path contract.

---

What’s included

• palace format detection
• stable-path palace safety gate
• .mcp.json migration work
• ChromaDB 1.x exploration findings
• reconstruction prototype
• neutral reconstruction bundle format
• structural validation
• retrieval / usage comparison tooling
• runtime compatibility guardrails
• real MCP runtime validation tooling
• adversarial robustness harness
• structured reconstruction diagnostics
• experimental workflow docs

---

What this branch now demonstrates

1. Non-destructive reconstruction

• source palace is preserved
• reconstruction always targets a separate 1.x palace
• no destructive in-place migration is introduced

2. Runtime-valid reconstructed targets within tested scope

Reconstructed 1.x palaces have been validated against native 1.x palaces at multiple levels:

• structural parity
• document / metadata parity
• embedding parity
• retrieval parity
• MCP runtime parity

In the tested scope, reconstructed 1.x targets were indistinguishable from native 1.x targets at real MCP runtime level:

• same tools exposed
• same queries
• same results
• no semantic differences observed

3. Explicit runtime mismatch handling

This branch also formalizes a failure mode that previously produced misleading results:

• 0.6.x and 1.x runtime stacks can be mixed accidentally
• this can produce failures such as _type / masked “No palace found” behavior
• the branch now detects and surfaces those mismatches explicitly

4. Safe failure model

Adversarial and invalid inputs were exercised to ensure the reconstruction path does not fail ambiguously:

• no silent corruption observed
• invalid inputs rejected explicitly
• low-level SQLite failures wrapped into structured reconstruction errors

---

What this branch does not claim

This PR does not claim:

• universal compatibility with all historical MemPalace palaces
• support for all real-world datasets
• automatic repair of corrupted sources
• that all ChromaDB / MemPalace version combinations are supported
• that the reconstruction path should replace the stable conservative 0.6.x path today

This remains a bounded reconstruction workflow, not a blanket 1.x support declaration.

---

Stable vs experimental contract

Stable path

The existing stable path remains conservative and 0.6.x-oriented.

Experimental / reconstruction path

This branch adds a separate reconstruction workflow for producing validated 1.x targets, with:

• explicit runtime targeting
• explicit diagnostics
• validation-first workflow
• no source mutation

This keeps the stable support contract narrow while allowing the reconstruction path to mature independently.

---

Validation performed

Test suite

• pytest tests/

Reconstruction validation

Validated on controlled fixtures with:

• structure checks
• retrieval checks
• usage comparison
• runtime compatibility checks

MCP runtime validation

Validated through real MemPalace MCP server behavior:

• server startup
• tool exposure
• tool calls
• query execution
• same observable behavior vs native 1.x palaces in the tested scope

Adversarial robustness

Tested against invalid / hostile cases such as:

• corrupted SQLite files
• wrong schema inputs
• duplicated identifiers
• inconsistent metadata
• malformed / rejected inputs

Observed result:

• no silent corruption
• explicit structured failures
• no uncontrolled hard failures in the tested adversarial matrix

---

Reviewer focus

Please review:

1. whether the reconstruction workflow is correctly isolated from the stable path
2. whether the current guarantees are stated precisely enough
3. whether the runtime-validation claims are appropriately bounded
4. whether the safety / failure model is clear enough for future users
5. whether this should remain a long-lived draft branch or be split further before merge

---

Why this is still a draft

This branch proves much more than the original exploration framing, but it is still draft because the support boundary must remain explicit.

What is proven:

• safe non-destructive reconstruction
• runtime-usable reconstructed 1.x targets in tested scope
• explicit runtime mismatch detection
• no silent corruption in tested adversarial cases

What is not yet claimed:

• universal migration support
• broad real-world compatibility across all legacy palaces
• replacement of the conservative stable path

---

Expected outcome

This branch should be treated as:

• a serious reconstruction path with real validation
• a bounded migration workflow
• a safer foundation for future 1.x support work

It should not be read as:

• “everything about ChromaDB 1.x is now solved”
• “stable support contract has been widened”
• “all MemPalace data can now be migrated automatically”