fix: reduce checkpoint packing I/O before publication

[vincentkoc]

What Problem This Solves

Checkpoint packing rereads the entire database and every compressed part solely to compute hashes, consuming time needed for verified publication.

User Impact

Packing avoids those redundant reads while preserving the compressed format, authenticated restored-file hashes and full SQLite checks. The existing publication reserve and deadline remain unchanged. Sufficient production headroom still needs a measured end-to-end run.

Why This Change Was Made

The packing owner now carries the database hash from its existing input traversal and compressed hashes from successful output writes, including gzip headers and closing trailers. Incomplete writes, hash failures and close failures stop before upload. Fixed phase timings distinguish download, reconstruction, disk hashing and SQLite verification.

This follows the checkpoint transport introduced in #287 and its shared deadline in #291. Recovery, workflow triggers, credentials, retention and scheduling are unchanged.

Production LOC: +62/-32 (net +30) for bounded stream accounting and phase attribution. Tests: +131/-3.

Evidence

• All 95 Python checkpoint tests passed, including exact gzip byte parity across empty, exact-boundary and multipart inputs; short/invalid writes; flush/trailer accounting; hash and close failures; and unchanged authenticated readback checks.
• Both independent source reviews and fresh P0-P3 autoreview passed on 3c3f7feef8cd622b72c38a73d72a6377352186b7 (scoped-clean, no findings).
• git diff --check passed. All 18 source checks passed, including CI 39056366959, Docker and CodeQL; later conditional bot-dispatch checks were skipped.
• Exact-head ClawSweeper review reported no findings or before-merge requirements. Its optional Rank-up move—measure the shipped save path with representative state and authenticated readback—is deferred to the separately authorized postmerge manual qualification. Production publication headroom remains unproven; this PR does not activate workflows or schedules.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs maintainer review before merge.

What this changes

This PR computes checkpoint hashes during packing and adds separate transfer and verification timings.

Example: Packing a 1,073,741,824-byte SQLite checkpoint into two raw chunks.

• Before: Packing separately rereads the database and both compressed files to compute hashes.
• After: Hashes come from the existing input and output streams; downloaded files still receive independent verification.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	Focused and source-correct, with meaningful regression coverage and successful checks; behavior evidence remains limited to an in-process setup.
Proof confidence	🦐 gold shrimp (3/6)	This maintainer-authored optimization is outside the ordinary live-proof gate; supplied evidence exercises real packing and SQLite with a fake GitHub transport, and explicitly leaves production publication headroom unqualified.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Product

Kind: Performance · Worth it: Yes · Fix scope: Complete
User problem: Redundant disk reads consume the checkpoint save budget available for verified publication.
Reason: The maintainer-authored change has clear bounded value, preserves the existing product contract, and adds no configuration or public API.

Merge readiness

✅ Ready for maintainer review

Keep this maintainer-authored draft open: it provides a focused optimization absent from main, with no introduced correctness or security defect found.

Priority: P2
Reviewed head: 3c3f7feef8cd622b72c38a73d72a6377352186b7

Before merge

None.

Findings

None.

Tests

• Missing end-to-end proof: No shipped save run demonstrates measured packing savings and successful publication within the unchanged deadline; the supplied regressions were inspected, not rerun.

Agent review details

How this fits together

The private checkpoint helper snapshots native SQLite state, packs and uploads release assets, then authenticates a complete restore before publishing the checkpoint.

flowchart LR
A[Native SQLite state] --> B[Pinned snapshot]
B --> C[Compression and stream hashes]
C --> D[Private draft assets]
D --> E[Authenticated download]
E --> F[Disk and SQLite verification]
F --> G[Checkpoint promotion]

Loading

Technical review

Best possible solution:

Land the focused stream-hashing optimization while retaining the existing separate production qualification before workflow activation.

Do we have a high-confidence way to reproduce the issue?

Main visibly hashes the standalone database separately and rereads each compressed part; this establishes redundant I/O, but does not establish a production timeout.

Is this the best way to solve the issue?

Computing hashes during existing traversals addresses the redundant reads directly while preserving the independent authenticated restore and publication checks.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 95d1154f8972.

Provenance checked

• Checkpoint packing and manifest hashes keeps the original intent (1e88d45: fix(ci): preserve native state in private checkpoints #287 established complete-state preservation with authenticated byte-for-byte restore before publication.)
• Deadline-aware hashing and phase reporting keeps the original intent (206aeb3: fix: recover unpublished checkpoints without collecting again #291 bounded checkpoint saves and added safe phase timings while retaining authenticated readback.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

• Pinned change and current main: The introduced diff replaces separate packing hash reads with stream accounting; GitHub still reports main at the pinned base revision. (scripts/private_discord_checkpoint.py:208, 3c3f7feef8cd)
• Publication verification remains intact: Compressed descriptors are finalized after gzip and file closure; authenticated download, independent disk hashes, SQLite checks, parent revalidation, and the publication reserve remain before promotion. (scripts/private_discord_checkpoint.py:642, 3c3f7feef8cd)
• Compatibility boundary: The manifest schema, gzip settings, chunk boundaries, and restore validation remain unchanged; inspected regressions compare exact compressed bytes and digests across boundary cases, establishing a no-migration change. (scripts/test_private_discord_checkpoint_save.py:119, 3c3f7feef8cd)
• History and intended contract: The transport in fix(ci): preserve native state in private checkpoints #287 and deadline work in fix: recover unpublished checkpoints without collecting again #291 establish full authenticated restore as the publication gate; this optimization preserves that contract. (scripts/private_discord_checkpoint.py:183, 95d1154f8972)
• Discussion and exact-head checks: The full PR body explicitly limits its claims to source validation; discussion contains only bot status messages, and the reviewed head has 18 successful checks and two skipped checks. (3c3f7feef8cd)
• Repository guidance and clean checkout: The complete root AGENTS.md is the only ancestor policy; no maintainer-notes directory exists, the P0–P3 review range was retained, and no helper, second reviewer, or target tests were run. (AGENTS.md:3, 3c3f7feef8cd)

Review metrics

Metric	Value	Why it matters
Implementation and regression coverage	Production +62/-32 lines; tests +131/-3 lines	Growth covers bounded stream accounting, phase attribution, byte parity, and failure handling within the existing packing owner.

Labels

Label changes:

• add P2: Removing redundant checkpoint I/O is a useful, limited-scope performance improvement without evidence of an urgent regression.
• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Label justifications:

• P2: Removing redundant checkpoint I/O is a useful, limited-scope performance improvement without evidence of an urgent regression.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Measure the shipped checkpoint save path with representative state and authenticated readback before claiming production publication headroom.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed October 17, 2026, 2:15 PM ET / 18:15 UTC.