Skip to content

Add Tokenizer Mode Support for Message Search #31540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
shinaoka wants to merge 5 commits into
base: develop
Choose a base branch
from
Open

Add Tokenizer Mode Support for Message Search #31540

shinaoka wants to merge 5 commits into from

Conversation

@shinaoka
Copy link
Contributor

@shinaoka shinaoka commented Dec 14, 2025 •
edited
Loading

Summary

This PR adds a user-facing setting to choose the tokenizer mode for local message search. N-gram tokenization is essential for languages without clear word boundaries such as Japanese, Chinese, and Korean (CJK languages).

Changes

  • Add tokenizer mode setting (Standard/Japanese) in Message Search preferences
  • Pass tokenizer mode to Seshat via element-desktop IPC
  • Add confirmation dialog when changing tokenizer mode (requires reindex)

Dependencies

⚠️ This PR depends on:

Both upstream PRs must be merged before this PR can be merged.

Merge Order

1. Seshat PR #150  →  npm release (e.g., [email protected])
2. element-desktop PR  →  update matrix-seshat dependency
3. element-web (this PR)  →  UI for tokenizer mode selection

Checklist

@shinaoka shinaoka requested a review from a team as a code owner December 14, 2025 23:13
@github-actions github-actions bot added the Z-Community-PR Issue is solved by a community member's PR label Dec 14, 2025
@shinaoka
feat: Add tokenizer mode support for message search
3ea2396 
- Add tokenizer mode setting (Standard/Japanese) in preferences
- Pass tokenizer mode to Seshat when indexing events
- Add confirmation dialog when changing tokenizer mode (requires reindex)
@t3chguy
Copy link
Member

t3chguy commented Dec 15, 2025

@shinaoka seems like you have lint & insufficient test coverage issues

@shinaoka
Copy link
Contributor Author

shinaoka commented Dec 22, 2025

I have managed to fix the CI errors!

@t3chguy
Copy link
Member

t3chguy commented Jan 2, 2026

Blocked on matrix-org/seshat#150

@dbkr dbkr added the X-Needs-Product More input needed from the Product team label Jan 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@t3chguy t3chguy Awaiting requested review from t3chguy t3chguy is a code owner automatically assigned from element-hq/element-web-reviewers

@MidhunSureshR MidhunSureshR Awaiting requested review from MidhunSureshR MidhunSureshR is a code owner automatically assigned from element-hq/element-web-reviewers

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

T-Enhancement X-Blocked X-Needs-Product More input needed from the Product team Z-Community-PR Issue is solved by a community member's PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@shinaoka @t3chguy @dbkr