Skip to content

fix(emlx): skip re-ingest when attachment restoration changes nothing - #960

Open
salmonumbrella wants to merge 1 commit into
kenn-io:mainfrom
salmonumbrella:fix/emlx-restore-rerun
Open

salmonumbrella wants to merge 1 commit into
kenn-io:mainfrom
salmonumbrella:fix/emlx-restore-rerun

Conversation

@salmonumbrella

Copy link
Copy Markdown
Contributor

What changed

  • import-emlx now restores cached attachments into the stored raw of an archived partial message first, and skips the message when that changes nothing. Only messages that gain a newly cached attachment are re-ingested.
  • Skipped messages still get the current mailbox label and the Message-ID / In-Reply-To repair, same as the ordinary skip path. If the label add fails, the message still goes through ingestion so the label gets retried, as it did before.
  • A shrinking Apple Mail cache still never removes attachment bytes that are already archived.
  • The summary's restored count now means attachments this run added to the archive, counted once per message after ingestion succeeds. An idle rerun reports 0 instead of every cached attachment it read again.

Why

Since 0.20.0, every rerun rewrote each archived .partial.emlx message whose attachments are still in Apple Mail's Attachments/ cache, whether or not the archive already had them. With a large Mail library, a nightly incremental import went from about an hour to 6–10 hours, logged slow-transaction warnings, and rebuilt the analytics cache from all the rewritten rows while adding only a few hundred new messages.

Usage

No usage change. A rerun over an unchanged Mail tree now reports Updated: 0 and counts those messages under Skipped (dup).

Closes #957

🤖 Generated with Claude Code

@roborev-ci

roborev-ci Bot commented Sep 26, 2026

Copy link
Copy Markdown

roborev: Combined Review (1b70533)

Verdict: Changes require fixes for 1 finding.

Medium

  • internal/importer/emlx_import.go:414: When stored raw MIME equals the restored MIME, ingestion is skipped entirely. A message first imported with --no-attachments may retain attachment bytes in message_raw without attachment files or rows, and a later normal import silently skips materializing them.

    Fix: Check attachment-storage completeness before skipping, or materialize attachments independently of the message upsert while retaining the no-rewrite optimization. Add a regression test for no-attachments import followed by normal import.


Reviewers: codex, codex (security) | Synthesis: codex, 5s | Total: 4m59s

@wesm wesm self-assigned this Sep 26, 2026
Since 0.20.0, import-emlx re-ingested every archived .partial.emlx message
whose Apple Mail attachment cache was still present, on every run, because
any parse-time restoration sent the message down the re-ingest path.

Restore cached attachments into the stored raw first and skip the message
when the result is byte-identical. Skipped messages still get the mailbox
label and header repair; a failed label add still falls through to
ingestion so the label is retried. AttachmentsRestored now counts
attachments a run added to the archive, once per message, after ingestion
succeeds.

Closes kenn-io#957

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wesm
wesm force-pushed the fix/emlx-restore-rerun branch from 1b70533 to 57aeb5f Compare September 27, 2026 00:45
@roborev-ci

roborev-ci Bot commented Sep 27, 2026

Copy link
Copy Markdown

roborev: Combined Review (57aeb5f)

Verdict: No findings at or above medium severity.


Reviewers: codex, codex (security) | Synthesis: codex | Total: 3m37s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

0.20.0: import-emlx rewrites the same partial messages on every run (attachment restoration from #883 has no already-restored check)

2 participants