diff --git a/CHANGELOG.md b/CHANGELOG.md index b8463f8f..a436f887 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## Unreleased +- Keep Discord Components V2 messages, such as app-posted GitHub cards, searchable: bot sync keeps their component tree in `raw_json` and indexes its text, link buttons and media descriptions for search and snapshots, and Desktop imports index the same text. Messages archived before this fix keep empty text until they are re-read; the `sync` command docs describe backup-first recovery for bot-synced and Desktop-imported rows. - Refresh SQLite to v1.60.1 with its required libc v1.77.1, go-strftime to v1.1.0, and golangci-lint to v2.14.0, retaining the Go 1.27.0 minimum. - Update the TruffleHog secret-scanning action to v3.97.8. Thanks @dependabot. - Speed up exact semantic search 3.7× (10,000 × 1,536-dimensional embeddings: 506 ms → 137 ms, allocations 15.5M → 140k) by updating Crawlkit to v0.16.6 and skipping a redundant norm pass per candidate. diff --git a/docs/commands/search.md b/docs/commands/search.md index 2670533a..78eb56c8 100644 --- a/docs/commands/search.md +++ b/docs/commands/search.md @@ -32,7 +32,7 @@ Run [`discrawl lexical rebuild`](lexical.html) after enabling languages or repla - `--channel ` - resolve and restrict to one channel (exact id, exact name, or unique partial name) - `--author ` - restrict to one author - `--limit ` - cap result count -- `--include-empty` - include rows with no searchable content (attachment text/filenames, embeds, and replies still count as content) +- `--include-empty` - include rows with no searchable content (attachment text/filenames, embeds, Components V2 text, and replies still count as content) ## FTS behavior diff --git a/docs/commands/sync.md b/docs/commands/sync.md index 9399d962..6a3f69a9 100644 --- a/docs/commands/sync.md +++ b/docs/commands/sync.md @@ -90,6 +90,93 @@ discrawl sync --with-media - Interrupted recovery resumes from its own saved page checkpoint on the next unwindowed sync, including after cancellation or process termination. Messages ingested in the meantime do not mark the older history complete. - Failed recovery leaves partial history resumable. After cancellation, restoring a still-empty channel's completion marker uses the existing five-second failure-cleanup budget. +## Re-fetching Components V2 messages archived with empty text + +Discrawl versions before Components V2 support stored messages sent with the `IS_COMPONENTS_V2` flag (`32768`), such as app-posted GitHub cards, without their component tree or text. This affects both bot sync and Desktop cache imports. Those rows cannot be rebuilt locally, and routine runs don't re-read them: `sync`, `--all-channels`, `--full` and `--channels` only request messages after each channel's stored cursor, and `wiretap` skips cache files that haven't changed. Repair them with the fixed binary: + +### Prepare + +0. Back up the archive. `sqlite3 .backup` takes a consistent copy even while a writer is running: + + ```bash + DB="$(discrawl status | sed -n 's/^db=//p')" + sqlite3 "$DB" ".backup '$DB.before-components-v2'" + ``` + + To roll back later, stop every `sync`/`tail`, then run `cp "$DB.before-components-v2" "$DB" && rm -f "$DB-wal" "$DB-shm"`. + +1. Stop every `sync`, `tail` and scheduled sync that writes to this archive. + +### Bot-synced rows + +2. List the affected channels, their broken-row counts and oldest broken message ids, and save the channel ids: + + ```bash + BROKEN="deleted_at is null and json_extract(raw_json, '\$.flags') & 32768 and json_type(raw_json, '\$.components') is null" + sqlite3 -header "$DB" "select channel_id, count(*) as broken, min(cast(id as integer)) as oldest_id from messages where $BROKEN group by channel_id" + CHANNELS="$(sqlite3 "$DB" "select group_concat(distinct channel_id) from messages where $BROKEN")" + ``` + +3. Rewind each affected channel's `latest_message_id` cursor to just before its oldest broken message: + + ```bash + sqlite3 "$DB" " + update sync_state + set cursor = (select cast(min(cast(id as integer)) - 1 as text) from messages + where $BROKEN and sync_state.scope = 'channel:' || channel_id || ':latest_message_id') + where scope in (select distinct 'channel:' || channel_id || ':latest_message_id' from messages where $BROKEN)" + ``` + +4. Re-fetch those channels. The crawl moves forward from the rewound cursor and rewrites every message it receives, including `raw_json`, `normalized_content` and the search indexes. When it finishes, the cursor is back at the channel head: + + ```bash + discrawl sync --source discord --channels "$CHANNELS" + ``` + +5. Verify. The broken-row count should be `0`, and a search for the text of a known card should find it: + + ```bash + sqlite3 "$DB" "select count(*) from messages where $BROKEN" + discrawl search "" + ``` + + Any rows that remain are messages Discord no longer returns, for example because they were deleted. List them with the step 2 query. + +### Desktop-imported rows + +Desktop imports keep a sanitized `raw_json` without flags or components, so broken Desktop rows can't be counted exactly. The closest check is Desktop rows that have no text and no attachments. It also matches messages that are genuinely empty, such as sticker-only messages. + +6. Count the candidates: + + ```bash + DESKTOP="deleted_at is null and json_extract(raw_json, '\$.source') = 'discord_desktop' and trim(coalesce(normalized_content, '')) = '' and has_attachments = 0" + sqlite3 "$DB" "select count(*) from messages where $DESKTOP" + ``` + +7. Clear the Desktop file index so the next import rescans every cache file instead of skipping unchanged ones, then re-import: + + ```bash + sqlite3 "$DB" "delete from sync_state where scope like 'wiretap:file_index:%'" + discrawl wiretap + ``` + +8. Verify. The candidate count should drop by the number of recovered messages, and a search for the text of a known card should find it: + + ```bash + sqlite3 "$DB" "select count(*) from messages where $DESKTOP" + discrawl search "" + ``` + +This only recovers messages that Discord Desktop still has cached. If a remaining candidate is in a guild channel the bot can read, repeat steps 2-5 with `BROKEN="$DESKTOP and guild_id <> '@me'"`. Keep only those channels in `CHANNELS`, because a targeted sync fails on channels the bot cannot read. The rewind and `--channels` sync then fetch those messages through the bot. + +### Publish + +9. Publish the repaired archive: + + ```bash + discrawl publish --push + ``` + ## See also - [Sync sources](../guides/sync-sources.html) diff --git a/docs/guides/search-modes.md b/docs/guides/search-modes.md index 6bf0ea38..853d9b69 100644 --- a/docs/guides/search-modes.md +++ b/docs/guides/search-modes.md @@ -15,7 +15,7 @@ - supported presets are Korean with native Kiwi through `kiwigo`, Japanese with Kagome Search, Chinese with GSE search mode, and Arabic with in-process light stemming - user query terms are parameterized and quoted before `MATCH`, so tokens like `AND`, `OR`, `NOT`, `NEAR`, and `*` are searched as input terms instead of FTS operators - punctuation still follows FTS5 tokenization rules -- by default, `search` skips rows with no searchable content (attachment text, attachment filenames, embeds, and replies still count as content); use `--include-empty` to opt back in +- by default, `search` skips rows with no searchable content (attachment text, attachment filenames, embeds, Components V2 text, and replies still count as content); use `--include-empty` to opt back in ### Optional multilingual lexical fields diff --git a/internal/discorddesktop/import_value_helpers_test.go b/internal/discorddesktop/import_value_helpers_test.go index c76abbe6..ecafd489 100644 --- a/internal/discorddesktop/import_value_helpers_test.go +++ b/internal/discorddesktop/import_value_helpers_test.go @@ -163,3 +163,35 @@ func TestDiscordMessagePayloadHelpers(t *testing.T) { require.False(t, looksSnowflake("123")) require.False(t, looksSnowflake("12345678901x")) } + +func TestParseMessageIndexesComponentsV2Text(t *testing.T) { + var raw map[string]any + require.NoError(t, json.Unmarshal([]byte(`{ + "id": "333333333333333333", + "channel_id": "111111111111111111", + "guild_id": "999999999999999999", + "flags": 32768, + "content": "", + "timestamp": "2026-09-28T10:15:00Z", + "author": {"id": "444444444444444444", "username": "Hermit", "bot": true}, + "components": [{"type": 17, "components": [ + {"type": 10, "content": "### [OPEN] PR #160926 fix(secrets): redact gateway tokens"}, + {"type": 14, "divider": true}, + {"type": 9, "components": [{"type": 10, "content": "size: XL • P0"}], + "accessory": {"type": 11, "media": {"url": "https://avatars.example/1.png"}, "description": "avatar"}}, + {"type": 12, "items": [{"media": {"url": "https://cdn.example/chart.png"}, "description": "chart"}]}, + {"type": 1, "components": [{"type": 2, "style": 5, "label": "View on GitHub", "url": "https://github.com/openclaw/openclaw/pull/160926"}]} + ]}] + }`), &raw)) + + message, ok := parseMessage(raw, time.Time{}, nil) + require.True(t, ok) + require.Empty(t, message.Record.Content) + require.Equal(t, "### [OPEN] PR #160926 fix(secrets): redact gateway tokens\nsize: XL • P0\navatar\nchart\nView on GitHub\nhttps://github.com/openclaw/openclaw/pull/160926", message.Record.NormalizedContent) + require.NotContains(t, message.Record.RawJSON, "components", "desktop raw payloads stay sanitized") + + delete(raw, "flags") + message, ok = parseMessage(raw, time.Time{}, nil) + require.True(t, ok) + require.Empty(t, message.Record.NormalizedContent, "classic message components are not message bodies") +} diff --git a/internal/discorddesktop/payload.go b/internal/discorddesktop/payload.go index ad9eef5a..dffb8d68 100644 --- a/internal/discorddesktop/payload.go +++ b/internal/discorddesktop/payload.go @@ -9,6 +9,8 @@ import ( "time" "unicode" + "github.com/bwmarrin/discordgo" + "github.com/openclaw/discrawl/internal/store" ) @@ -172,7 +174,7 @@ func parseMessage(raw map[string]any, fallbackTime time.Time, channels map[strin editedAt := parseDiscordTime(stringField(raw, "edited_timestamp")) attachments := parseAttachments(raw, id, guildID, channelID, authorID) mentions := parseMentions(raw, id, guildID, channelID, authorID, createdAt) - normalized := normalizeText(content, attachmentText(attachments), embedText(raw)) + normalized := normalizeText(content, attachmentText(attachments), embedText(raw), componentText(raw)) return store.MessageMutation{ Record: store.MessageRecord{ ID: id, @@ -277,6 +279,44 @@ func embedText(raw map[string]any) []string { return out } +// componentText collects the visible text of a Components V2 message, whose +// whole body is its component tree. +func componentText(raw map[string]any) []string { + flags, _ := intField(raw, "flags") + if discordgo.MessageFlags(flags)&discordgo.MessageFlagsIsComponentsV2 == 0 { + return nil + } + return appendComponentText(nil, raw["components"]) +} + +func appendComponentText(out []string, value any) []string { + items, _ := value.([]any) + for _, item := range items { + component, _ := item.(map[string]any) + componentType, _ := intField(component, "type") + switch discordgo.ComponentType(componentType) { + case discordgo.TextDisplayComponent: + out = append(out, stringField(component, "content")) + case discordgo.ButtonComponent: + out = append(out, stringField(component, "label"), stringField(component, "url")) + case discordgo.ThumbnailComponent: + out = append(out, stringField(component, "description")) + case discordgo.MediaGalleryComponent: + galleryItems, _ := component["items"].([]any) + for _, galleryItem := range galleryItems { + media, _ := galleryItem.(map[string]any) + out = append(out, stringField(media, "description")) + } + case discordgo.SectionComponent: + out = appendComponentText(out, component["components"]) + out = appendComponentText(out, []any{component["accessory"]}) + case discordgo.ContainerComponent, discordgo.ActionsRowComponent: + out = appendComponentText(out, component["components"]) + } + } + return out +} + func normalizeText(parts ...any) string { flat := []string{} for _, part := range parts { diff --git a/internal/syncer/records.go b/internal/syncer/records.go index e17ab47c..134609d6 100644 --- a/internal/syncer/records.go +++ b/internal/syncer/records.go @@ -42,7 +42,7 @@ func effectiveMessageGuildID(message *discordgo.Message, fallbackGuildID string) } func toMessageRecord(message *discordgo.Message, channelName, guildID, normalizedContent string) store.MessageRecord { - raw := marshalJSONString(message, "{}") + raw := marshalMessageJSON(message) authorID := "" authorName := "" if message.Author != nil { @@ -87,6 +87,15 @@ func marshalJSONString(value any, fallback string) string { return string(raw) } +// marshalMessageJSON restores the components discordgo decodes but tags +// `json:"-"` on Message, so raw_json keeps Components V2 bodies. +func marshalMessageJSON(message *discordgo.Message) string { + return marshalJSONString(struct { + *discordgo.Message + Components []discordgo.MessageComponent `json:"components,omitempty"` + }{message, message.Components}, "{}") +} + func normalizeMessage(message *discordgo.Message) string { return normalizeMessageParts(message, nil) } @@ -113,6 +122,9 @@ func normalizeMessageParts(message *discordgo.Message, attachmentParts []string) parts = append(parts, embed.Description) } } + if message.Flags&discordgo.MessageFlagsIsComponentsV2 != 0 { + parts = appendComponentText(parts, message.Components) + } if message.ReferencedMessage != nil && message.ReferencedMessage.Content != "" { parts = append(parts, "reply:"+message.ReferencedMessage.Content) } @@ -134,6 +146,37 @@ func normalizeMessageParts(message *discordgo.Message, attachmentParts []string) return strings.Join(filtered, "\n") } +// appendComponentText collects the visible text of a component tree, which is +// the whole body of a Components V2 message. +func appendComponentText(parts []string, components []discordgo.MessageComponent) []string { + for _, component := range components { + switch c := component.(type) { + case *discordgo.TextDisplay: + parts = append(parts, c.Content) + case *discordgo.Button: + parts = append(parts, c.Label, c.URL) + case *discordgo.Thumbnail: + if c.Description != nil { + parts = append(parts, *c.Description) + } + case *discordgo.MediaGallery: + for _, item := range c.Items { + if item.Description != nil { + parts = append(parts, *item.Description) + } + } + case *discordgo.Section: + parts = appendComponentText(parts, c.Components) + parts = appendComponentText(parts, []discordgo.MessageComponent{c.Accessory}) + case *discordgo.Container: + parts = appendComponentText(parts, c.Components) + case *discordgo.ActionsRow: + parts = appendComponentText(parts, c.Components) + } + } + return parts +} + func sanitizeNormalizedPart(raw string) string { raw = strings.ToValidUTF8(raw, "") raw = norm.NFKC.String(raw) diff --git a/internal/syncer/records_test.go b/internal/syncer/records_test.go index 66d144bb..40aad34c 100644 --- a/internal/syncer/records_test.go +++ b/internal/syncer/records_test.go @@ -2,6 +2,7 @@ package syncer import ( "context" + "encoding/json" "path/filepath" "testing" @@ -11,6 +12,89 @@ import ( "github.com/openclaw/discrawl/internal/store" ) +// componentsV2CardPayload is a sanitized GitHub PR card sent with the +// IS_COMPONENTS_V2 flag: no content or embeds, only a component tree. +const componentsV2CardPayload = `[{ + "id": "1554528840938819725", + "channel_id": "c1", + "guild_id": "g1", + "type": 20, + "flags": 32768, + "content": "", + "embeds": [], + "attachments": [], + "mentions": [], + "timestamp": "2026-09-28T10:15:00.000000+00:00", + "author": {"id": "u-app", "username": "Hermit", "bot": true}, + "components": [{ + "type": 17, + "id": 1, + "accent_color": 3066993, + "components": [ + {"type": 10, "id": 2, "content": "### [OPEN] PR #160926 fix(secrets): redact gateway tokens in doctor output"}, + {"type": 10, "id": 3, "content": "size: XL • P0 • needs proof"}, + {"type": 14, "id": 4, "divider": true, "spacing": 1}, + {"type": 9, "id": 5, + "components": [{"type": 10, "id": 6, "content": "opened by contributor • +412 −37"}], + "accessory": {"type": 11, "id": 7, "media": {"url": "https://avatars.example/u/1.png"}, "description": "contributor avatar"}}, + {"type": 12, "id": 8, "items": [{"media": {"url": "https://cdn.example/diffstat.png"}, "description": "diffstat chart"}]}, + {"type": 1, "id": 9, "components": [ + {"type": 2, "id": 10, "style": 5, "label": "View on GitHub", "url": "https://github.com/openclaw/openclaw/pull/160926"} + ]} + ] + }] +}]` + +func TestComponentsV2MessageKeepsCardTextSearchable(t *testing.T) { + t.Parallel() + + ctx := context.Background() + s, err := store.Open(ctx, filepath.Join(t.TempDir(), "discrawl.db")) + require.NoError(t, err) + t.Cleanup(func() { _ = s.Close() }) + + var page []*discordgo.Message + require.NoError(t, json.Unmarshal([]byte(componentsV2CardPayload), &page)) + channel := &discordgo.Channel{ID: "c1", GuildID: "g1", Name: "github", Type: discordgo.ChannelTypeGuildText} + svc := New(&fakeClient{messages: map[string][]*discordgo.Message{"c1": page}}, s, nil) + count, err := svc.syncMessageChannelsSerial(ctx, "g1", []*discordgo.Channel{channel}, SyncOptions{Full: true}, nil) + require.NoError(t, err) + require.Equal(t, 1, count) + + var content, normalized, raw string + require.NoError(t, s.DB().QueryRowContext(ctx, + `select content, normalized_content, raw_json from messages where id = '1554528840938819725'`, + ).Scan(&content, &normalized, &raw)) + require.Empty(t, content, "content stays as Discord sent it") + require.Equal(t, "### [OPEN] PR #160926 fix(secrets): redact gateway tokens in doctor output\n"+ + "size: XL • P0 • needs proof\n"+ + "opened by contributor • +412 −37\n"+ + "contributor avatar\n"+ + "diffstat chart\n"+ + "View on GitHub\n"+ + "https://github.com/openclaw/openclaw/pull/160926", normalized) + + var stored struct { + Components []json.RawMessage `json:"components"` + } + require.NoError(t, json.Unmarshal([]byte(raw), &stored)) + require.Len(t, stored.Components, 1) + var reparsed discordgo.Message + require.NoError(t, json.Unmarshal([]byte(raw), &reparsed), "raw_json components must decode as Discord components") + require.Equal(t, normalized, normalizeMessage(&reparsed), "raw_json keeps every text-bearing component") + + for _, query := range []string{"160926", "redact gateway tokens", "needs proof"} { + results, err := s.SearchMessages(ctx, store.SearchOptions{Query: query, Limit: 10}) + require.NoError(t, err) + require.Len(t, results, 1, query) + require.Equal(t, "1554528840938819725", results[0].MessageID) + } + + classic := *page[0] + classic.Flags = 0 + require.Empty(t, normalizeMessage(&classic), "classic message components are not message bodies") +} + func TestToMemberRecordSkipsNilUser(t *testing.T) { t.Parallel()