feat(desktop): 搜索对中文短词按字召回，不再需要搜整句话才能命中 - #3569
Conversation
messages_fts 写入侧用 cjk_seg 在汉字间及中英边界插空格，查询侧收成相邻 phrase；侧栏 / search_chat_history 多词 OR，session_search 多词 AND。 snippet 从原文重建并转义标签，高亮按码点坐标切。存量重建 migration 为 0099。 Signed-off-by: yan <yan233@xd.com>
|
| Filename | Overview |
|---|---|
| apps/desktop/src/main/localDb/cjkSeg.ts | 新增 CJK 分词、坐标映射及原文 snippet 重建；后者无法定位 porter 词干命中和跨标点 phrase 命中。 |
| apps/desktop/src/main/localDb/chatHistorySearch.pure.ts | 新增有限长、转义后的 messages_fts MATCH 构造，并按入口支持 OR/AND 语义。 |
| apps/desktop/src/main/localDb/chatHistorySearch.ts | 侧栏和历史搜索改用分字 MATCH 及原文预览，但继承了 tokenizer 命中与预览定位不一致的问题。 |
| apps/desktop/src/main/maker-host/session-search.ts | session_search 改为多词 AND 并重建原文 snippet，同样可能返回不含实际命中的预览。 |
| apps/desktop/drizzle/scripts/0099_segment_messages_fts_cjk.ts | companion migration 原子地校验或重建 FTS、rowid 映射及触发器，未发现当前数据一致性缺陷。 |
| apps/desktop/src/main/localDb/registerCjkSeg.ts | 集中注册并探测连接级 cjk_seg 函数，主要生产连接均已覆盖。 |
| apps/desktop/src/renderer/features/cc-agent/sidebar/snippetMarkup.ts | 安全解析受控 <mark> 哨兵并按正确顺序还原实体，未发现注入路径。 |
| apps/desktop/src/renderer/features/cc-agent/sidebar/keywordRanges.ts | 新增折叠偏移到原文 UTF-16 偏移的映射，避免 Unicode 大小写扩展和代理对导致错切。 |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
Q[用户或工具查询] --> M[构造 FTS5 MATCH]
M --> F[messages_fts<br/>porter unicode61]
F --> R[召回消息]
R --> C[读取 messages.content 原文]
Q --> T[抽取原始 query tokens]
C --> S[buildSnippetFromContent]
T --> S
S --> O[侧栏或工具结果预览]
Prompt To Fix All With AI
### Issue 1
apps/desktop/src/main/localDb/cjkSeg.ts:164-180
**词干命中丢失预览定位**
当查询经 `porter unicode61` 词干化后命中不同表面形式，或中文 phrase 跨标点命中时，这里仍按原始查询 token 精确定位正文；定位失败便返回正文开头且不添加 `<mark>`，导致侧栏、`search_chat_history` 和 `session_search` 的合法结果看不到实际命中上下文。
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: "feat(desktop): 搜索对中文短词按字召回，不再需要搜整句话才能命中" | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6929773534
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
ConversationSearchBox 收到的是 visibleSnippet 原文，不是 buildSnippetFromContent 的 <mark> 协议。没有哨兵时原样展示，避免字面量 < 被解成 <。 Signed-off-by: yan <yan233@xd.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1b8611ddaa
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
长度帽只加在会按字展开的汉字 token 上。Latin / 数字整段是一个 FTS token， 截成 64 字后 quoted MATCH 不再精确命中。 Signed-off-by: yan <yan233@xd.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7aef4ab861
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@yan-xdt 👋 这个 PR 还有 2 条 review conversation 没 resolve(apps/desktop/src/main/localDb/conversationSearch.ts / apps/desktop/src/main/localDb/chatHistorySearch.ts),auto-review 因此暂时跳过、没法继续审查 / 合并。 如果你已经按评论改完或回应了,请到对应 thread 上点 Resolve conversation;全部 resolve 后,下一轮 auto-review 会自动重新审查这个 PR。 |
MagicLizi
added
touches:core
|
这条 PR 进入维护者确认：产品 UI（会话搜索高亮）、安全敏感写法、规则文档和较大核心改动都命中了。请维护者直接在 PR 上 Approve；若要改请 Request Changes，作者改完后再 Approve。讨论 issue：#3572 本轮 CI 未绿、还有未 resolve 的 conversation，确认通过后也还要先处理这些卡点。 |
|
命中 UI 路径(apps/desktop/src/renderer/features/cc-agent/lib/highlightSegments.tsx / apps/desktop/src/renderer/features/cc-agent/sidebar/ConversationSearchBox.tsx / apps/desktop/src/renderer/features/cc-agent/sidebar/keywordRanges.ts 等)但 description 未附界面效果证据——建议补充改动后效果:截图/录屏,或改动后界面的 HTML 页面(```html 代码块、.html 附件或在线预览链接),便于确认界面符合 DESIGN.md 设计规范 |
可见命中改为字面匹配或 cjk_seg token 序列，不再用 preview 非空当证据。 FTS 查询用 substr 限制传回字符数，snippet 扫描同一上限。 Signed-off-by: yan <yan233@xd.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ed9492d883
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@yan-xdt 👋 这个 PR 还有 1 条 review conversation 没 resolve(apps/desktop/src/renderer/features/cc-agent/sidebar/snippetMarkup.ts),auto-review 因此暂时跳过、没法继续审查 / 合并。 如果你已经按评论改完或回应了,请到对应 thread 上点 Resolve conversation;全部 resolve 后,下一轮 auto-review 会自动重新审查这个 PR。 |
searchConversations 给 renderer 的是原文切片。原文里同时出现 <mark> 和实体时， 旧启发式会进协议模式并把 < 解成 <。默认当原文，只有显式 protocol:true 才解码。 Signed-off-by: yan <yan233@xd.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 12e4c4d800
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@yan-xdt 👋 这个 PR 还有 1 条 review conversation 没 resolve(apps/desktop/drizzle/scripts/0099_segment_messages_fts_cjk.ts),auto-review 因此暂时跳过、没法继续审查 / 合并。 如果你已经按评论改完或回应了,请到对应 thread 上点 Resolve conversation;全部 resolve 后,下一轮 auto-review 会自动重新审查这个 PR。 |
SQLite 自定义函数是连接级的，持久触发器引用它会让回退后的旧客户端 插入消息直接报 no such function。持久 insert/update 触发器改回只写 原文，并用 pragma_function_list 守卫在已注册函数的连接上跳过；新 连接由 registerCjkSeg 挂 TEMP 触发器按字写入。应用 0099 后回退旧 客户端插入仍成功，只是新行不再按字切。 Signed-off-by: yan <yan233@xd.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a064bd7151
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
清理 powershell 的 2s 预算覆盖不了满载机器上自身冷启动加 Win32_Process BFS，超时后走 catch 静默放弃，孤儿后代留到存活检查超时——即 makecindy#3574 的 偶发失败形态。清理只在 probe 失败的收尾路径执行，放宽不占交互预算。 测试侧同步把存活检查的进程启动余量放宽，并让 probe 输出带诊断（操作 被预算截断）时与整体 ETIMEDOUT 同口径 skip，不改任何断言语义。 Signed-off-by: yan <yan233@xd.com>
0e7d237 to
7d209fe
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d209fec7e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. Swish! Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
7d209fe to
9adea2c
Compare
每个用例都经 resetModules 加动态 import 重新加载整条 SUT 模块链，首 用例承担全部依赖的首次编译加载，满载 CI 上超出 vitest 默认 5s 报超时。 只放宽本文件 testTimeout，断言与流程不变。 Signed-off-by: yan <yan233@xd.com>
helper powershell 的冷启动（Add-Type 编译、.NET 加载）与 mutex 竞争共用 一个超时预算，满载 CI 上冷启动吃光预算直接报 timed out acquiring—— Windows unit 已连续多轮挂在 mutex 用例上。helper 现在先报 started、锁住 后报 locked，超时按启动 / 等锁 / 探测三段独立计时；waitForExit 等真实 exit 事件再收口，消除 release 后重试与仍持锁进程的竞态。测试补 started/locked 解析、脚本顺序锁定与强杀后等 exit 的用例。 Signed-off-by: yan <yan233@xd.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2eefb2a62f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
token 码点帽从 64 提到 256，与 MCP schema 的 query 上限对齐。此前 65~256 字连续汉字被截成 64 字前缀 phrase，只含前缀的消息会假阳性命中 ——messages_fts 按字 phrase 与群历史整段 token 两条路径都是。整段保留 后两条路径都精确；超 256 截断只防御绕过 schema 的超长侧栏输入， MATCH_CHAR_CAP 按 token 粒度截断不会切出半个 phrase。 Signed-off-by: yan <yan233@xd.com>
2 participants
这次改了什么
摘要
侧栏任务搜索、
search_chat_history和session_search现在能对 2～4 字中文短词做按字召回，不再依赖语义索引。messages_fts一直用porter unicode61，连续汉字会被收成一个 token，搜「登录」打不中「登录报错了」。本 PR 按 #3331 的方案 A：写入时在汉字之间、以及汉字与英文/数字边界插入空格；查询时把中文收成相邻 phrase（"登 录"）。英文 porter 不变。预览从原文重建，不展示带空格的索引文本。变更类型
feat新功能fix缺陷修复refactor/perf重构或性能优化docs/test/chore文档、测试或工程维护范围
cjk_segSQL 函数：只处理\p{Script=Han}；combining mark 附着在前一个汉字上，不阻断分词；中英相邻也切开messages的建连点都注册该函数（factory、DB worker、inline 回滚 worker、0099 companion）；漏注册会让插入消息失败，而不是搜索静默降级buildMessagesFtsMatch：CJK 收成相邻 quoted phrase；侧栏 /search_chat_history多词 OR，session_search多词 ANDDROP messages_fts后按cjk_seg(m.content)回填；健康库再跑会复用现有 FTS 行，不整表重建messages.content重建并转义原文</>/&，避免用户内容里的<mark>被当成高亮标记；高亮按码点坐标切，emoji /İ不错位Win32_ProcessBFS，超时走 catch 静默放弃、孤儿后代留到存活检查超时，正是 Windows 全量单测中的 PowerShell 后代进程清理用例偶发失败，影响开发效率及CI稳定性 #3574 的失败形态；probe 输出带诊断（操作被预算截断）时与整体ETIMEDOUT同口径 skip；codexProxyHost.test.ts文件级超时放宽到 20s——首用例承担整条 SUT 模块链的冷编译加载，满载 CI 上超出 vitest 默认 5s。三处都只放宽预算 / 对齐既有 skip 口径，断言语义不变windowsPackagedInstanceBarriermutex 用例（2eefb2a62）：不修这个 PR 的 CI 过不了——该用例已连续多轮挂在timed out acquiring，且 fix(desktop): relink Codex thread across providers #3543 把同类修复摘除后没有其它修复在路上，只能在本 PR 帮修。helper 先报started、锁住后报locked，超时按启动 / 等锁 / 探测三段独立计时（此前 powershell 冷启动 + Add-Type 编译与 mutex 竞争共用一个预算，满载 runner 上必超）；waitForExit等真实 exit 事件再收口，消除 release 后重试与仍持锁进程的竞态。生产行为改动集中在 barrier acquire 的内部计时与脚本协议，对外契约不变UI 变化
侧栏搜索结果的 snippet / 标题高亮仍走既有
<mark>和bg-search-match-bg/text-search-match-fg，没有新控件、新文案或新颜色。改动是高亮切点：按原文码点切，emoji 不再被切成半个代理对。Light / Dark 都未做实机目检。docs/design-rules/DESIGN.md§10「Theme System & Token Reference」的 Light / Dark Dual-Mode Delivery Gate，以及 §15.4 将search-match-bg列为语义豁免色。本次复用既有搜索高亮 token，无新增硬编码颜色。怎么验证的
自动验证
手工验证
--isolated沙箱，用正式库副本做存量升级，未写共享 Cindy userData未执行的验证
better-sqlite3的定向套件最初因 NODE_MODULE_VERSION 137 vs 127 无法加载 native addon；本机按.nvmrc重建依赖后已复跑通过（cjkSeg10、segmentMessagesFtsCjkMigration4、cjkFtsMatch.integration、betterSqliteFactory、runtime）。风险
风险分类
影响与回滚
messages_fts。0099 升级时对白名单 role 做一次 DROP + 按字回填；之后每条消息写入经触发器调用cjk_seg。Memory / 通讯录 / 群历史表不改。漏注册cjk_seg会让插入消息失败。顺手修复里唯一的生产行为改动是 maker-core Windows probe 失败收尾路径的后代清理超时 2s→8s，不影响 probe 主流程预算。apps/mobile原生配置或 runtime fingerprint，不触发冷更。提交前检查
git commit -s，见 DCO）docs/dev-rules/database-and-migrations.md写明写messages必须注册cjk_seg）