feat(search): expand CJK full-text search to Japanese and Korean - #1734
Open
mariusvniekerk wants to merge 2 commits into
Open
feat(search): expand CJK full-text search to Japanese and Korean#1734mariusvniekerk wants to merge 2 commits into
mariusvniekerk wants to merge 2 commits into
Conversation
Japanese and Korean searches previously matched characters in the wrong order or scattered across a message. Preserve each term as a character phrase when the query contains kana or Hangul, while keeping Chinese word segmentation for Han-only queries. Use the CJK name in documentation, install commands, and runtime messages to reflect the broader support. Keep stored index names and fingerprints stable so the rename does not force an index rebuild.
Use CJK names for the optional search tables, triggers, and fingerprint identifiers so the storage schema matches the feature's language scope. The feature has not shipped, so update its definitions directly without migration code or aliases for the earlier Chinese-specific names.
roborev: Combined Review (
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Japanese and Korean message searches now preserve character order and adjacency within each search term. Previously, the Chinese query segmenter could make
かなmatchなか, or make Korean검색match separate occurrences of검and색.Rename the optional SQLite feature to CJK full-text search, including
make install-cjk-fts, its workflow, tables, triggers, and fingerprint identifiers. The feature has not shipped, so the schema definitions use the new names directly without migration code for the earlier names.Queries containing kana or Hangul use character phrases. Han-only queries retain Chinese word segmentation; Japanese kanji-only searches can use explicit quoted phrases. This remains SQLite message search, without Japanese or Korean grammatical analysis. The query behavior lives in
internal/db/cjk_fts_search.go.