Summary
On Windows (system locale zh-TW, ACP=CP950), search_code never matches CJK patterns even when the pattern verifiably exists in UTF-8 files, and raw_matches[].content returns mojibake with ? replacement characters. The tree-sitter/snippet pipeline (get_code_snippet) handles the same files' CJK content perfectly — only the grep/content pipeline is affected.
Environment
Repro
- A UTF-8 Python file contains the string
未來函數 (verified: present in 28 files; file reports "UTF-8 text" for all of them).
- Call
search_code with {"pattern": "未來函數", "project": "..."}.
- Result:
total_grep_matches: 0, total_results: 0.
Control A (ASCII pattern, same index): search_code for DualTrackRouter → 55 grep matches, works fine.
Control B (same file, snippet pipeline): get_code_snippet on a function whose docstring contains CJK → CJK returned byte-perfect.
Evidence of ANSI-codepage transcoding
For an ASCII pattern that lands on a line that also contains CJK, raw_matches[].content comes back like:
"content": " # ?w?w STEP 2d?G???J???T???J?w?y?{?]DualTrackRouter?^?A"
The original UTF-8 line is ── STEP 2d：載入資訊…（DualTrackRouter）…. This ?-folding pattern is the signature of routing UTF-8 bytes through WideCharToMultiByte/MultiByteToWideChar with CP_ACP (CP950 here): characters unmappable in the ANSI codepage become ?. Since the grep index/content path stores (or compares) the transcoded text, a UTF-8 CJK pattern can never match.
Expected behavior (industry baseline)
Treat file content as UTF-8 by default end-to-end (ripgrep's model: UTF-8 default, BOM sniffing for UTF-16, explicit flag for legacy encodings, U+FFFD for invalid sequences — never the OS ANSI codepage). v0.9.0 already does wide-char correctness for paths (_wfopen, wide-argv); the same treatment is needed for the content pipeline used by search_code.
Related
Adjacent Windows/CJK family: #903 (CJK paths, snippets), #973, #996, #1165 (env vars lossily decoded), #1171 (persistence crash under CJK paths). This one is distinct: it's the content/grep pipeline, not paths — the repo here is indexed under an ASCII junction precisely to avoid the path family, and snippet extraction proves the files themselves are readable.
Impact
Any non-English codebase on Windows (Chinese/Japanese/Korean comments, docstrings, identifiers in strings) silently loses search_code for native-language queries — it returns a clean "0 matches" with no error, so users assume the text simply isn't there.
Summary
On Windows (system locale zh-TW, ACP=CP950),
search_codenever matches CJK patterns even when the pattern verifiably exists in UTF-8 files, andraw_matches[].contentreturns mojibake with?replacement characters. The tree-sitter/snippet pipeline (get_code_snippet) handles the same files' CJK content perfectly — only the grep/content pipeline is affected.Environment
Repro
未來函數(verified: present in 28 files;filereports "UTF-8 text" for all of them).search_codewith{"pattern": "未來函數", "project": "..."}.total_grep_matches: 0, total_results: 0.Control A (ASCII pattern, same index):
search_codeforDualTrackRouter→ 55 grep matches, works fine.Control B (same file, snippet pipeline):
get_code_snippeton a function whose docstring contains CJK → CJK returned byte-perfect.Evidence of ANSI-codepage transcoding
For an ASCII pattern that lands on a line that also contains CJK,
raw_matches[].contentcomes back like:The original UTF-8 line is
── STEP 2d：載入資訊…（DualTrackRouter）…. This?-folding pattern is the signature of routing UTF-8 bytes throughWideCharToMultiByte/MultiByteToWideCharwith CP_ACP (CP950 here): characters unmappable in the ANSI codepage become?. Since the grep index/content path stores (or compares) the transcoded text, a UTF-8 CJK pattern can never match.Expected behavior (industry baseline)
Treat file content as UTF-8 by default end-to-end (ripgrep's model: UTF-8 default, BOM sniffing for UTF-16, explicit flag for legacy encodings, U+FFFD for invalid sequences — never the OS ANSI codepage). v0.9.0 already does wide-char correctness for paths (
_wfopen, wide-argv); the same treatment is needed for the content pipeline used bysearch_code.Related
Adjacent Windows/CJK family: #903 (CJK paths, snippets), #973, #996, #1165 (env vars lossily decoded), #1171 (persistence crash under CJK paths). This one is distinct: it's the content/grep pipeline, not paths — the repo here is indexed under an ASCII junction precisely to avoid the path family, and snippet extraction proves the files themselves are readable.
Impact
Any non-English codebase on Windows (Chinese/Japanese/Korean comments, docstrings, identifiers in strings) silently loses
search_codefor native-language queries — it returns a clean "0 matches" with no error, so users assume the text simply isn't there.