perf(imap): write only the memberships a full enumeration changes - #750
Closed
exactmike wants to merge 1 commit into
Closed
perf(imap): write only the memberships a full enumeration changes#750exactmike wants to merge 1 commit into
exactmike wants to merge 1 commit into
Conversation
A Reset delta republishes an entire mailbox, and applyIMAPMailboxDeltas deleted and reinserted every saved membership row, then rebuilt labels and tombstone state for every message it touched. On a 96,000-message INBOX that is roughly 700,000 statements in one write transaction -- `sql tx slow duration_ms=230461` -- for a set of rows that is almost entirely unchanged. Read the mailbox's saved rows once, remove the UIDs the reset does not republish, and write only the memberships whose message or flags moved. Only those messages enter the label reconciliation pass. Flags are compared after decoding, so SQLite text and PostgreSQL JSONB compare alike, and a row whose saved flags do not decode is always rewritten. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DhzbNgimfaBhPUSMzB8v2A
roborev: Combined Review (
|
Member
|
rebasing |
Member
|
superseded by #765 |
wesm
added a commit
that referenced
this pull request
Sep 5, 2026
Supersedes #750. Closes #749. A full IMAP enumeration previously rewrote every saved membership row in a mailbox and rebuilt every message's labels, even when almost nothing changed. The store now diffs the enumeration against saved rows, writes only memberships whose message or flags changed, and rebuilds labels only for affected messages. A reset that empties a mailbox, such as a new UIDVALIDITY epoch, still clears it in one statement. Tombstone behavior is unchanged. A membership that never changes after an earlier partial label merge is no longer repaired as a side effect of a later full enumeration; #748 tracks an explicit repair path for that case. On a 118,000-message Microsoft 365 account, a forced full enumeration's store transaction dropped from 282 seconds to 2.9 seconds while preserving final memberships, folder states, labels, and tombstones. No configuration or usage changes. Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #749
A full enumeration used to rewrite every saved membership row of a mailbox
and rebuild every message's labels, even when almost nothing had changed.
The store now diffs against the saved rows and writes only the memberships
whose message or flags moved, rebuilding labels only for those messages.
A reset that empties a mailbox, such as a new UIDVALIDITY epoch, still
clears it in one statement. Tombstones are unaffected. A message whose
membership never changes again after an earlier partial label merge no
longer gets its labels rebuilt by a later full enumeration. #748 proposes an
explicit repair command for that case.
On a 118,000-message Microsoft 365 account, a forced full enumeration's
store transaction dropped from 282s to 2.9s. Final memberships, folder
states, labels and tombstones matched.
No configuration or usage changes.