fix: Components V2 messages are archived with empty text and missing from search

[obviyus]

What Problem This Solves

Fixes: Discord messages sent with Components V2 (message flag IS_COMPONENTS_V2, 1 << 15) are archived with empty content and normalized_content, and without their components, so search, FTS, SQL and published Git snapshots lose them. One example is the Hermit app's /github PR cards in the OpenClaw Discord ([OPEN] PR #160926 fix(secrets): … size: XL • P0 • needs proof). The publishing archive has 50 such rows since 2026-09-22, each with flags = 32768, content = "", embeds = [] and no components key.

User Impact

User impact: newly synced Components V2 messages keep their component tree in raw_json, and their visible text is searchable via search, normalized_content, FTS, embeddings input and published snapshots. content is left as Discord sent it (empty), which matches how embeds are handled.

Action required on the publishing side: rows archived before this fix never stored their components, so they can't be repaired locally. They have to be fetched from Discord again. None of the existing commands do this on their own (see Backfill below).

Why This Change Was Made

Root cause

• Decoding works. discordgo v0.29.0 decodes every Components V2 type (components.go:41-77: 1 action row, 2 button, 9 section, 10 text display, 11 thumbnail, 12 media gallery, 14 separator, 17 container) into Message.Components.
• Marshaling drops the tree. discordgo@v0.29.0/message.go:89 tags Components []MessageComponent as json:"-". Message has a custom UnmarshalJSON (message.go:167-183) but no MarshalJSON, so internal/syncer/records.go:45 (raw := marshalJSONString(message, "{}")) writes raw_json without components.
• Normalization ignores components. internal/syncer/records.go:94-135 (normalizeMessageParts) builds normalized_content from content, attachments, embed title and description, reply and poll. A Components V2 message has none of these, so the result is "". FTS (internal/store/write.go:432-435) and snapshots both read normalized_content.
• Desktop wiretap (internal/discorddesktop/payload.go:175) had the same gap in normalizeText. Its raw_json is intentionally a sanitized summary, so that stays unchanged.

Fix:

• toMessageRecord marshals the message with its components restored (omitempty, so raw_json for messages without components is byte-identical).
• normalizeMessageParts reuses the embed path: when the Components V2 flag is set, it appends the tree's text to the same parts list. That covers text display content (including inside containers and sections), button labels and link URLs, and thumbnail and media-gallery descriptions. Classic messages whose components are only buttons keep their current normalized_content, so they don't churn and no embeddings are re-queued.
• The Desktop importer adds a matching componentText next to embedText.
• No schema change. Snapshot privacy filtering is untouched: @me rows are still excluded, and the change adds no new secret-bearing fields.

Backfill

Neither sync, sync --all-channels, sync --full nor sync --channels re-fetches messages that are already archived. Each resumes after the channel's latest_message_id cursor, and --full skips channels with history_complete. scripts/reconcile_cloud_source.go skips rows that already exist. I checked this against a pre-fix archive using the fixed binary: all three runs wrote messages=0, and --channels requested after=<the affected id>. Desktop imports have the same gap: a repeat wiretap skips unchanged cache files. Their sanitized raw_json also stores no flags, so the bot-row query can't find them.

The recovery procedure is now in the docs, under Re-fetching Components V2 messages archived with empty text in docs/commands/sync.md; the CHANGELOG entry links to it. The steps are:

0. Back up the archive with sqlite3 "$DB" ".backup '$DB.before-components-v2'", where DB comes from discrawl status. The online backup is consistent. To roll back, stop writers, then cp "$DB.before-components-v2" "$DB" && rm -f "$DB-wal" "$DB-shm".
1. Stop every sync, tail and scheduled sync that writes to the archive.
2. Run the exact SQL that lists affected channels, their broken-row counts and oldest broken ids. A broken row matches deleted_at is null and json_extract(raw_json,'$.flags') & 32768 and json_type(raw_json,'$.components') is null. Save the channel ids.
3. Rewind each affected channel's channel:<id>:latest_message_id cursor to (oldest broken id − 1).
4. Run discrawl sync --source discord --channels "$CHANNELS". It forward-crawls from the rewound cursor and upserts every message it receives, which rewrites raw_json, normalized_content, FTS and lexical indexes in place. AdvanceChannelLatestMessageID only moves forward, so the cursor ends at the channel head again.
5. Verify: the broken-row count should be 0, and discrawl search "<text from a known card>" should find the card. Any rows left are messages Discord no longer returns.
6. Desktop-imported rows: count candidates. Desktop rows with no text and no attachments are the closest check, because broken Desktop rows can't be counted exactly and genuinely empty messages also match.
7. Clear wiretap:file_index:% from sync_state and rerun discrawl wiretap, which rescans every cache file.
8. Verify the candidate count drops and a known card is found by search. This only recovers messages Discord Desktop still has cached. For remaining candidates in guild channels the bot can read, repeat steps 2–5 with BROKEN="$DESKTOP and guild_id <> '@me'", passing those channels with --channels.
9. Run discrawl publish --push.

For the current 50 rows this re-crawls about eight days of the affected channels. The live tail also re-fetches a message on MESSAGE_UPDATE, so cards edited later (for example OPEN → MERGED) repair themselves.

Evidence

• Regression test TestComponentsV2MessageKeepsCardTextSearchable (internal/syncer/records_test.go) runs a sanitized Hermit-style payload through json.Unmarshal → the syncer message crawl → the store. The payload is a container holding text displays, a separator, a section with a thumbnail accessory, a media gallery, and an action row with a link button. The test asserts that raw_json keeps components and re-decodes to the same text, that normalized_content holds the card text, and that FTS search finds the message by 160926, redact gateway tokens and needs proof. It fails on origin/main (normalized_content is "") and passes with this fix. TestParseMessageIndexesComponentsV2Text covers the Desktop importer and its flag gate.
• End-to-end smoke with the real CLI against a temp DB. discrawl sync --source discord ran in-process with every discord.com request routed to a local fake API; any other host was refused and nothing contacted Discord. search and sql then ran from the built binary:

=== before (origin/main) ===
$ discrawl search "160926"
(exit 0)                                  # no results
$ discrawl sql "select content, normalized_content, json_extract(raw_json,'$.flags') flags, json_type(raw_json,'$.components') components from messages"
content  normalized_content  flags  components
                             32768
=== after ===
$ discrawl search "160926"
[900000000000000001/github] Hermit 2026-09-28T10:15:00Z
### [OPEN] PR #160926 fix(secrets): redact gateway tokens in doctor output
size: XL • P0 • needs proof
View on GitHub
https://github.com/openclaw/openclaw/pull/160926
$ discrawl sql "…same query…"
         ### [OPEN] PR #160926 fix(secrets): … https://github.com/openclaw/openclaw/pull/160926  32768  array

• discrawl publish --no-commit to a local repo: before the fix, the messages shard row had normalized_content: '' and no raw_json.components. After the fix it has the card text and components present.
• discrawl wiretap against a fixture Desktop cache file: search 160926 returned nothing before and the card after.
• Recovery docs: every bash block in the section was run verbatim, in doc order, on copies of pre-fix archives. The only substitutions were 160926 for the search placeholder and a fake-Discord sync.
  • Bot rows (steps 0–5, then publish): the listing step showed 900000000000000002|1|1554528840938819725, the cursor was rewound to …724, and the sync ran GET …/messages?after=1554528840938819724 and wrote messages=1. The broken count then read 0 and search 160926 found the card. The shard pushed to a local bare remote carries the card text and components. The documented rollback restored the backup and the broken count went back to 1.
  • Desktop rows, still cached (steps 0, 6–8, then publish), on a DB imported by the pre-fix binary: candidate count 1, then wiretap wrote messages=1, then candidate count 0. search 160926 found the card and the pushed shard carries its text.
  • Desktop rows, evicted from the cache, in a bot-readable channel: wiretap wrote messages=0 and the candidate count stayed at 1. Steps 2–5 were then repeated with the documented BROKEN="$DESKTOP and guild_id <> '@me'", which listed the channel, rewound the cursor, and ran a sync that wrote messages=1 (after=…724). The count then read 0, search 160926 found the card, and it was published.
• make tidy-check fmt lint test-coverage smoke all pass locally. snapshot (goreleaser) was not run locally.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: blocked before merge. Reviewed September 30, 2026, 12:30 AM ET / 04:30 UTC (Revision 3).

ClawSweeper review

What this changes

The branch preserves Discord Components V2 trees during bot sync, indexes their visible text from bot and Desktop imports, and documents search behavior and recovery of older archives.

Merge readiness

⛔ Blocked before merge - 3 items remain

Current main and the latest release still omit Components V2 text from archived search content. This PR addresses both bot sync and Desktop imports, and its latest commit addresses the earlier Desktop recovery finding.

Priority: P2
Reviewed head: abf2f93390de8072586af99cf0aa8efe62562b2e
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	Focused production-path proof and regression coverage support the patch; the remaining gate is approval of the archive recovery procedure.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (terminal): The PR’s captured terminal evidence exercises the changed bot-sync writer and Desktop importer through the real CLI against old archive copies and a local Discord API fixture, showing searchable text and repaired snapshots after the fix. The documented replay was also checked on pre-fix bot and Desktop rows, supporting stored-state compatibility; deleted or evicted source messages remain an explicit recovery limit.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (terminal): The PR’s captured terminal evidence exercises the changed bot-sync writer and Desktop importer through the real CLI against old archive copies and a local Discord API fixture, showing searchable text and repaired snapshots after the fix. The documented replay was also checked on pre-fix bot and Desktop rows, supporting stored-state compatibility; deleted or evicted source messages remain an explicit recovery limit.
Evidence reviewed	9 items	Current main lacks the behavior: The bot record writer marshals discordgo.Message directly, and normalization passes from embeds to replies without reading Components V2 text. Dependency serialization contract: The directly imported discordgo v0.29.0 Message type excludes Components from JSON marshaling; its custom unmarshal path decodes them. Introduced bot repair: The PR restores decoded components to stored raw JSON and adds text extraction only when the Components V2 flag is set.
Findings	None	None.
Security	None	None.

How this fits together

Discrawl imports messages from the Discord bot API and Desktop cache into a local SQLite archive. The archive’s normalized text feeds search indexes and published Git snapshots.

flowchart LR
  A[Discord bot API] --> B[Bot sync]
  C[Desktop cache] --> D[Desktop importer]
  B --> E[SQLite archive]
  D --> E
  E --> F[Search indexes]
  E --> G[Git snapshots]

Loading

Decision needed

Question	Recommendation
Should the documented cursor rewind and Desktop file-index reset be the supported recovery procedure for existing Discrawl archives?	Approve scoped recovery: The archive owner confirms the backup, affected-channel scope, rollback limits, and unrecoverable-row limits before adopting the runbook.

Why: The procedure changes persisted archive state and can trigger a broad replay; technical proof does not establish the archive operator’s acceptance of that operation.

Before merge

• Resolve merge risk (P1) - Existing archives do not repair themselves. The runbook directly rewinds SQLite sync cursors and clears all Desktop file-index entries, potentially causing a broad re-fetch; archive-operator approval of its scope, backup, and rollback limits is still outstanding.
• Complete next step (P2) - Obtain archive-operator approval of the cursor-rewind, Desktop rescan, and rollback procedure before merge.
• Resolve maintainer decision - Resolve the maintainer decision shown above before merge.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Production and test code	production +83 lines, tests +116 lines	Growth covers two distinct import paths and focused regression coverage.

Merge-risk options

Maintainer options:

1. Approve the managed replay (recommended)
   Have the archive owner confirm the documented backup, channel selection, Desktop rescan, and rollback limits before merge.
2. Guard the recovery path
   Add a previewable, channel-scoped recovery command before recommending replay on existing archives.

Technical review

Best possible solution:

Ship the two import-path fixes with an archive-owner-approved, scoped recovery procedure that verifies repaired rows before publication and states which old rows cannot be recovered.

Do we have a high-confidence way to reproduce the issue?

Yes. The supplied Components V2 payload and before/after CLI transcript give a repeatable current-main path from bot sync to an empty search result; source inspection supports the same failure mechanism.

Is this the best way to solve the issue?

Yes. Extracting text in the two existing import paths while preserving Discord’s original content field is a focused repair; historical replay needs archive-operator approval.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against d55485748d2a.

Labels

Label changes:

• add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
• add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR’s captured terminal evidence exercises the changed bot-sync writer and Desktop importer through the real CLI against old archive copies and a local Discord API fixture, showing searchable text and repaired snapshots after the fix. The documented replay was also checked on pre-fix bot and Desktop rows, supporting stored-state compatibility; deleted or evicted source messages remain an explicit recovery limit.
• remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.
• remove rating: 🦐 gold shrimp: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.

Label justifications:

• P2: The patch addresses a concrete search and archive gap for a limited class of Discord messages.
• merge-risk: 🚨 compatibility: Previously archived rows need a manual replay that changes persisted cursors and Desktop scan state.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR’s captured terminal evidence exercises the changed bot-sync writer and Desktop importer through the real CLI against old archive copies and a local Discord API fixture, showing searchable text and repaired snapshots after the fix. The documented replay was also checked on pre-fix bot and Desktop rows, supporting stored-state compatibility; deleted or evicted source messages remain an explicit recovery limit.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR’s captured terminal evidence exercises the changed bot-sync writer and Desktop importer through the real CLI against old archive copies and a local Discord API fixture, showing searchable text and repaired snapshots after the fix. The documented replay was also checked on pre-fix bot and Desktop rows, supporting stored-state compatibility; deleted or evicted source messages remain an explicit recovery limit.

Evidence

What I checked:

• Current main lacks the behavior: The bot record writer marshals discordgo.Message directly, and normalization passes from embeds to replies without reading Components V2 text. (internal/syncer/records.go:45, d55485748d2a)
• Dependency serialization contract: The directly imported discordgo v0.29.0 Message type excludes Components from JSON marshaling; its custom unmarshal path decodes them. (message.go:89, 6e8fa27c7917)
• Introduced bot repair: The PR restores decoded components to stored raw JSON and adds text extraction only when the Components V2 flag is set. (internal/syncer/records.go:125, abf2f93390de)
• Introduced Desktop repair: Desktop parsing indexes flagged component text while continuing to store its sanitized raw payload. (internal/discorddesktop/payload.go:284, abf2f93390de)
• Search-index path: Message upserts write normalized content into the FTS table, connecting the changed import behavior to search. (internal/store/write.go:432, abf2f93390de)
• Desktop recovery follow-up: The earlier review identified Desktop rows missing from the bot-row predicate. The latest documentation adds a separate Desktop candidate check, cache rescan, and bot-readable fallback; genuinely empty candidates and evicted cache entries remain stated limits. Earlier review: fix: Components V2 messages are archived with empty text and missing from search #265 (comment). (docs/commands/sync.md:147, abf2f93390de)

Likely related people:

• Peter Steinberger: Raw commit 4ae71d1 adds internal/discorddesktop/payload.go:407 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 4ae71d1f63dc; files: internal/discorddesktop/payload.go)
• rnavarro: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• vincentkoc: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Obtain archive-operator approval for the documented replay and rollback procedure.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (2 earlier review cycles)

• reviewed 2026-09-30T04:05:41.694Z sha 3c3082f :: blocked before merge. :: none
• reviewed 2026-09-30T04:21:56.152Z sha a72cdb8 :: blocked before merge. :: [P2] Cover pre-fix Desktop rows in the recovery path

[obviyus]

Maintainer approval on exact head abf2f93390de8072586af99cf0aa8efe62562b2e.

As archive owner, I approve the documented recovery runbook in docs/commands/sync.md for Components V2 rows archived before this fix. It covers the backup first, the cursor rewind and targeted re-sync for bot-synced rows, the wiretap file-index rescan for Desktop-imported rows (limited to messages Discord Desktop still caches), verification and rollback. Every step was run verbatim on copies of pre-fix archives, including the rollback.

Revision 3 of the review has no findings, and all CI checks pass on this head. The fix is confirmed: the regression test fails on main and passes here, and discrawl search 160926 goes from no result to the full card on the before and after databases.