fix: recover Date headers mangled by mbox continuation artifacts - #506
Merged
wesm merged 1 commit intoJul 25, 2026
Merged
Conversation
enmime folds a stray, unindented continuation line (a lone ".", left
over from legacy mbox/Thunderbird sources) onto the preceding Date
header instead of erroring, appending a trailing " ." that no
parseDate format tolerated. Also add a date layout for weekday + US
month-day ordering ("Tue, Oct 17 2000 02:15:24 -0700"), which is
likewise silently dropped today.
Both patterns occur in real mailboxes and are silently swallowed
during ingestion today: parseDate returns the zero time and the
message's sent date is lost rather than being read from a perfectly
good Date header.
Authored-By: Jesse Robbins (@jesserobbins) <https://jesserobbins.com>
roborev: Combined Review (
|
jesserobbins
marked this pull request as ready for review
July 25, 2026 17:58
This was referenced Jul 25, 2026
Closed
Contributor
Author
wesm
pushed a commit
that referenced
this pull request
Jul 26, 2026
## Motivation I imported an older mail archive and found messages appearing decades away from where they belonged because syntactically valid but implausible `Date:` headers were treated as authoritative. Those timestamps affect timeline ordering, date-range searches, statistics, and analytics partitions, and reimporting would accept the same bad values again. ## Summary - add one plausibility resolver for email ingestion, bounded from 1990 through 30 days in the future, using a valid `Date:` header, then the oldest plausible `Received:` timestamp, then plausible source metadata - add `repair-dates`, with a read-only default and explicit `--apply` mode - apply repairs atomically with changed-row protection and a privacy-safe JSON audit ledger - rebuild the Parquet analytics cache for SQLite archives while correctly skipping that SQLite-only cache on PostgreSQL - route the command through the daemon and document recovery for partial failures - warn when ingestion rejects a nonzero implausible header so genuinely old archives do not change silently - fix a `mime.Parse` bug where a mangled Date header (mbox continuation artifact, or US month-day ordering) silently dropped the sent date instead of resolving it, recovering 62 of the messages this PR's own audit found unresolvable - report *why* each remaining `repair-dates` candidate is unresolved (date header present but unusable, Received headers present but unusable, or no date signal at all) instead of a bare count ## Note This branch currently includes the `mime.Parse` fix (commit `6bf31dc9`) as its own commit, duplicated from #506, which splits that fix out for isolated review since it's a general ingestion bug, not specific to `repair-dates`. Kept here too so this branch builds and tests standalone; will rebase to drop the duplicate once #506 merges. ## Test plan - [x] `go test -tags "fts5 sqlite_vec" ./cmd/... ./internal/... ./pkg/... ./tools/...` - [x] `go vet ./...` - [x] `make lint-ci` - [x] verify dry-run, apply, idempotency, audit-ledger failure states, daemon admission, SQLite cache rebuilding, and PostgreSQL cache selection with synthetic fixtures Closes #501 Co-authored-by: Jesse Robbins <jesserobbins@users.noreply.github.com>
jesserobbins
added a commit
to jesserobbins/msgvault
that referenced
this pull request
Aug 6, 2026
…n-io#506) ## Motivation I ran into this while tracing why some archived messages had no sent date. It's not a historical-data problem — it happens at ingestion time, so the sent date is lost the moment a message with either pattern below is parsed, not just when re-scanning old data. I am submitting this as separate PR from kenn-io#501 kenn-io#504 because this is a clear bug with the importer flow. Fixes kenn-io#505 ## Summary - `parseDate` now trims a trailing `" ."` artifact before trying any layout. Some mbox/Thunderbird-derived sources have a lone, unindented `.` line directly after the `Date:` header; enmime folds it onto the header value as a continuation instead of erroring, leaving a trailing `" ."` that no layout tolerated. - Added a `dateFormats` layout for weekday + US month-day ordering (`"Tue, Oct 17 2000 02:15:24 -0700"`), which wasn't covered by any existing layout. ## Test plan - [x] `go test ./internal/mime/...` - [x] New `TestParse_MalformedContinuationLineAfterDate` reproduces the real mbox artifact end-to-end through `Parse` - [x] New `TestParseDate` cases for both patterns - [x] `go vet ./...`, `gofmt -l` clean --- *Authored by [Jesse Robbins](https://jesserobbins.com) (@jesserobbins)* Co-authored-by: Jesse Robbins <jesserobbins@users.noreply.github.com>
jesserobbins
added a commit
to jesserobbins/msgvault
that referenced
this pull request
Aug 6, 2026
## Motivation I imported an older mail archive and found messages appearing decades away from where they belonged because syntactically valid but implausible `Date:` headers were treated as authoritative. Those timestamps affect timeline ordering, date-range searches, statistics, and analytics partitions, and reimporting would accept the same bad values again. ## Summary - add one plausibility resolver for email ingestion, bounded from 1990 through 30 days in the future, using a valid `Date:` header, then the oldest plausible `Received:` timestamp, then plausible source metadata - add `repair-dates`, with a read-only default and explicit `--apply` mode - apply repairs atomically with changed-row protection and a privacy-safe JSON audit ledger - rebuild the Parquet analytics cache for SQLite archives while correctly skipping that SQLite-only cache on PostgreSQL - route the command through the daemon and document recovery for partial failures - warn when ingestion rejects a nonzero implausible header so genuinely old archives do not change silently - fix a `mime.Parse` bug where a mangled Date header (mbox continuation artifact, or US month-day ordering) silently dropped the sent date instead of resolving it, recovering 62 of the messages this PR's own audit found unresolvable - report *why* each remaining `repair-dates` candidate is unresolved (date header present but unusable, Received headers present but unusable, or no date signal at all) instead of a bare count ## Note This branch currently includes the `mime.Parse` fix (commit `6bf31dc9`) as its own commit, duplicated from kenn-io#506, which splits that fix out for isolated review since it's a general ingestion bug, not specific to `repair-dates`. Kept here too so this branch builds and tests standalone; will rebase to drop the duplicate once kenn-io#506 merges. ## Test plan - [x] `go test -tags "fts5 sqlite_vec" ./cmd/... ./internal/... ./pkg/... ./tools/...` - [x] `go vet ./...` - [x] `make lint-ci` - [x] verify dry-run, apply, idempotency, audit-ledger failure states, daemon admission, SQLite cache rebuilding, and PostgreSQL cache selection with synthetic fixtures Closes kenn-io#501 Co-authored-by: Jesse Robbins <jesserobbins@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
I ran into this while tracing why some archived messages had no sent date. It's not a historical-data problem — it happens at ingestion time, so the sent date is lost the moment a message with either pattern below is parsed, not just when re-scanning old data. I am submitting this as separate PR from #501 #504 because this is a clear bug with the importer flow.
Fixes #505
Summary
parseDatenow trims a trailing" ."artifact before trying any layout. Some mbox/Thunderbird-derived sources have a lone, unindented.line directly after theDate:header; enmime folds it onto the header value as a continuation instead of erroring, leaving a trailing" ."that no layout tolerated.dateFormatslayout for weekday + US month-day ordering ("Tue, Oct 17 2000 02:15:24 -0700"), which wasn't covered by any existing layout.Test plan
go test ./internal/mime/...TestParse_MalformedContinuationLineAfterDatereproduces the real mbox artifact end-to-end throughParseTestParseDatecases for both patternsgo vet ./...,gofmt -lcleanAuthored by Jesse Robbins (@jesserobbins)