feat(search): expand CJK full-text search to Japanese and Korean

.github/workflows/chinese-fts.yml → .github/workflows/cjk-fts.yml

@@ -1,9 +1,9 @@
-name: Chinese FTS
+name: CJK FTS
 
 on:
   pull_request:
     paths:
-      - ".github/workflows/chinese-fts.yml"
+      - ".github/workflows/cjk-fts.yml"
       - "Makefile"
       - "internal/db/**"
       - "internal/service/**"
@@ -12,7 +12,7 @@ on:
   push:
     branches: [main]
     paths:
-      - ".github/workflows/chinese-fts.yml"
+      - ".github/workflows/cjk-fts.yml"
       - "Makefile"
       - "internal/db/**"
       - "internal/service/**"
@@ -27,8 +27,8 @@ permissions:
   contents: read
 
 jobs:
-  test-chinese-fts:
-    name: Go Test (Chinese FTS sidecar)
+  test-cjk-fts:
+    name: Go Test (CJK FTS sidecar)
     runs-on: ubuntu-latest
     timeout-minutes: 25
     steps:
@@ -47,7 +47,7 @@ jobs:
       - name: Build pinned simple and cppjieba sidecar
         run: bash scripts/build-simple-fts.sh dist/agentsview-simple
 
-      - name: Run database, service, and MCP tests with Chinese FTS enabled
+      - name: Run database, service, and MCP tests with CJK FTS enabled
         env:
           AGENTSVIEW_SIMPLE_DIR: ${{ github.workspace }}/dist/agentsview-simple
           CGO_ENABLED: "1"

Makefile

@@ -35,7 +35,7 @@ AIR_BIN := $(shell if command -v air >/dev/null 2>&1; then command -v air; \
 	elif [ -x "$(GOPATH_FIRST)/bin/air" ]; then printf "%s" "$(GOPATH_FIRST)/bin/air"; \
 	fi)
 
-.PHONY: build build-release install install-chinese-fts simple-fts frontend frontend-dev dev check-air air-install desktop-dev desktop-build desktop-macos-app desktop-macos-dmg desktop-windows-installer desktop-linux-appimage desktop-app docs-install docs-build docs-serve docs-check docs-screenshots docs-assets-branch docs-generated-assets-branch docs-deploy-staging docs-deploy test test-short test-evalingest bench-backends bench-gate bench-gate-config bench-pg-usage test-postgres test-postgres-ci test-s3 postgres-up postgres-down test-ssh test-ssh-ci ssh-up ssh-down e2e e2e-duckdb vet lint lint-ci lint-golangci lint-golangci-ci nilaway nilaway-golangci-build lint-tools tidy clean release release-darwin-arm64 release-darwin-amd64 release-linux-amd64 install-hooks ensure-embed-dir pricing-snapshot sqlite-vec-header dev-snapshot help check-timing-budgets
+.PHONY: build build-release install install-cjk-fts simple-fts frontend frontend-dev dev check-air air-install desktop-dev desktop-build desktop-macos-app desktop-macos-dmg desktop-windows-installer desktop-linux-appimage desktop-app docs-install docs-build docs-serve docs-check docs-screenshots docs-assets-branch docs-generated-assets-branch docs-deploy-staging docs-deploy test test-short test-evalingest bench-backends bench-gate bench-gate-config bench-pg-usage test-postgres test-postgres-ci test-s3 postgres-up postgres-down test-ssh test-ssh-ci ssh-up ssh-down e2e e2e-duckdb vet lint lint-ci lint-golangci lint-golangci-ci nilaway nilaway-golangci-build lint-tools tidy clean release release-darwin-arm64 release-darwin-amd64 release-linux-amd64 install-hooks ensure-embed-dir pricing-snapshot sqlite-vec-header dev-snapshot help check-timing-budgets
 
 # Ensure go:embed has at least one file (no-op if frontend is built)
 ensure-embed-dir:
@@ -102,8 +102,8 @@ install: build-release
 simple-fts:
 	bash scripts/build-simple-fts.sh dist/agentsview-simple
 
-# Install the binary and its Chinese-search sidecar in sibling bin/lib trees.
-install-chinese-fts: install simple-fts
+# Install the binary and its CJK-search sidecar in sibling bin/lib trees.
+install-cjk-fts: install simple-fts
 	@if [ -d "$(HOME)/.local/bin" ]; then \
 		INSTALL_DIR="$(HOME)/.local/bin"; \
 	else \
@@ -127,7 +127,7 @@ install-chinese-fts: install simple-fts
 	for name in simple-LICENSE cppjieba-LICENSE; do \
 		install -m 0644 "dist/agentsview-simple/licenses/$$name" "$$SIMPLE_DIR/licenses/$$name"; \
 	done; \
-	echo "Installed Chinese FTS sidecar to $$SIMPLE_DIR"
+	echo "Installed CJK FTS sidecar to $$SIMPLE_DIR"
 
 # Build frontend SPA and copy into embed directory
 frontend:
@@ -653,7 +653,7 @@ help:
 	@echo "  build-release  - Release build (optimized, stripped)"
 	@echo "  pricing-snapshot - Restore LiteLLM snapshot from artifact branch"
 	@echo "  install        - Build and install to ~/.local/bin or GOPATH"
-	@echo "  install-chinese-fts - Install agentsview with the optional Chinese FTS sidecar"
+	@echo "  install-cjk-fts - Install agentsview with the optional CJK FTS sidecar"
 	@echo "  simple-fts     - Build the pinned simple/cppjieba SQLite extension"
 	@echo ""
 	@echo "  dev            - Run Go server with live reload via air (use with frontend-dev)"

README.md

@@ -311,8 +311,9 @@ agentsview stats --include-git-outcomes
 | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------- |
 | ![Search](https://agentsview.io/assets/generated/screenshots/search-results.png) | ![Heatmap](https://agentsview.io/assets/generated/screenshots/heatmap.png) |
 
-- **Full-text search** across all message content (FTS5), with optional
-  `simple`/cppjieba tokenization for Chinese queries
+- **Full-text search** across all message content (FTS5), with optional CJK
+  character and phrase search for Chinese, Japanese, and Korean text in
+  SQLite, including Chinese word segmentation through `simple`/cppjieba
 - **Semantic search** (opt-in) -- index session content with any
   OpenAI-compatible embeddings endpoint and search by meaning with
   `agentsview session search --semantic` or `--hybrid`; every content-search

docs/changelog.md

@@ -13,10 +13,12 @@ description: Release history for AgentsView
   files are gone remain in the archive. (#1677)
 - Browse and search Open Code Review sessions, including review comments,
   tools, thinking, recorded token usage, and resumed reviews. (#1660)
-- Search Chinese words and individual characters in SQLite message content
-  with an optional tokenizer sidecar. HTTP, CLI, and MCP search use word
-  segmentation; ASCII-only searches keep English stemming. Install with
-  `make install-chinese-fts`. (#1491)
+- Search Chinese, Japanese, and Korean text in SQLite message content with
+  the optional CJK full-text search sidecar. Japanese kana and Korean Hangul
+  queries preserve character order within each search term; Han-only queries
+  use Chinese word segmentation. Quoted phrases preserve order in all three
+  languages. ASCII-only searches keep English stemming. Install with
+  `make install-cjk-fts`. (#1491)
 - Browse and search Tau sessions, including the active conversation branch,
   thinking, tools, session names, and recorded token usage. (#1661)
 - Configure session directories and alternate homes in `[agents.<id>]` tables

docs/configuration.md

@@ -1309,25 +1309,48 @@ pulled in from PostgreSQL sync or copied from other archives.
 ## Database
 
 The SQLite database uses WAL mode for concurrent reads and includes FTS5
-full-text search indexes on message content. To add Chinese word, phrase, and
-single-character matching, build and install the pinned `simple`/cppjieba
-sidecar with `make install-chinese-fts`. Building it requires Git, CMake
-3.19 or newer, and a C++14 compiler. AgentsView discovers it next to the binary
+full-text search indexes on message content.
+
+### CJK full-text search
+
+To add character and phrase matching for Chinese, Japanese, and Korean text,
+including Chinese word segmentation, build and install the pinned
+`simple`/cppjieba sidecar with `make install-cjk-fts`. Building it requires Git,
+CMake 3.19 or newer, and a C++14 compiler. AgentsView discovers it next to the binary
 or under the sibling `lib/agentsview/simple` directory. A custom path can be
 selected with `AGENTSVIEW_SIMPLE_DIR`.
 
-The sidecar adds a parallel `messages_chinese_fts` index and routes only CJK
-queries through it. ASCII-only searches continue to use the existing Porter
-index, so searches such as `run` retain English stemming. The Chinese index is
-derived data: if the sidecar is removed, AgentsView drops that optional index
+The sidecar indexes individual CJK characters and routes queries containing Han,
+Hiragana, Katakana, or Hangul through that index. Query preparation depends on
+the scripts in the query:
+
+- Queries containing Japanese kana or Korean Hangul preserve character order
+  and adjacency within each whitespace-separated search term. For example,
+  `かな` does not match `なか`, and `검색` does not match separate occurrences
+  of `검` and `색`. Separate terms can match anywhere in the same message.
+- Queries containing Han without kana or Hangul use Chinese word segmentation
+  through cppjieba. Japanese queries written entirely in kanji take this same
+  path because the scripts alone do not distinguish the languages. Quote a
+  kanji phrase, such as `"検索方法"`, to require its characters in order.
+- A leading double quote opts into explicit FTS5 expressions, including phrases
+  and operators, in any language. For example, `"검색 기능"` requires the
+  two terms together, while `검색 기능` allows intervening text.
+
+Japanese and Korean matching is character-based; it does not analyze grammatical
+word forms or expand readings, romanizations, or spelling variants. Chinese word
+segmentation also stays off when a query mixes Han with kana or Hangul.
+
+ASCII-only searches continue to use the existing Porter index, so searches such
+as `run` retain English stemming. The `messages_cjk_fts` index is derived data:
+if the sidecar is removed, AgentsView drops that optional index
 and continues with the standard FTS5 path; reinstalling the sidecar backfills
 it on the next writable open. AgentsView fingerprints the native library and
 all cppjieba dictionaries, atomically rebuilding the index when that fingerprint
 changes. Writers running with another fingerprint leave a freshness marker
 instead of mixing incompatible token streams. Pinyin expansion is disabled in
 the derived index because ASCII-only queries continue to use the Porter index.
 
-Chinese word segmentation is specific to SQLite message search, including the
+CJK full-text search is specific to SQLite message search, including the
 HTTP, CLI, and MCP search paths. PostgreSQL/CockroachDB and DuckDB do not load
 this SQLite extension and keep their existing search behavior. Substring and
 regular-expression searches are unchanged. Session search result snippets
@@ -1338,18 +1361,18 @@ The first backfill, a changed fingerprint, or any pending session requires a
 full index rebuild before startup completes. AgentsView logs this wait. The
 freshness ledger stores session IDs rather than old message IDs and token
 content, so it cannot remove stale entries for individual replaced or deleted
-messages. Removing the sidecar drops the Chinese index but retains the
-`messages_chinese_fts_pending_sessions` ledger and three persistent session
+messages. Removing the sidecar drops the CJK index but retains the
+`messages_cjk_fts_pending_sessions` ledger and three persistent session
 triggers. The ledger holds at most one row per touched session ID until the
-next successful Chinese index rebuild clears it.
+next successful CJK index rebuild clears it.
 
 Index maintenance uses TEMP triggers on the writer connection. Writes made
 without these triggers or with another sidecar fingerprint leave the index
-stale. Chinese search then falls back to standard FTS5 and logs a warning once
+stale. CJK search then falls back to standard FTS5 and logs a warning once
 per database handle. Reopening the archive with the sidecar restores the index
 and its triggers.
 
-**Schema tables:**
+### Schema tables
 
 | Table                | Purpose                                                                      |
 | -------------------- | ---------------------------------------------------------------------------- |
@@ -1363,7 +1386,7 @@ and its triggers.
 | `stats`              | Aggregate counts (session_count, message_count)                              |
 | `skipped_files`      | Cache of non-interactive session files                                       |
 | `messages_fts`       | FTS5 virtual table for full-text search                                      |
-| `messages_chinese_fts` | Optional FTS5 index using the `simple` Chinese tokenizer                   |
+| `messages_cjk_fts`   | Optional CJK FTS5 index using the `simple` character tokenizer               |
 
 The database is automatically migrated on startup when the schema changes. When
 the stored data version is stale, AgentsView preserves the existing database and