chore(ci): fail CI when unapproved non-English text appears in source files

[lizhengfeng101]

Why

Comments, identifiers and strings in this repository are meant to be English, but nothing enforced it. #861 had to clean up leftovers by hand, and the same drift keeps arriving through generated code and contributions written internally — reviewers cannot reliably catch it by eye.

What

scripts/verify-english-only.go walks the index plus untracked files and reports non-English text across 14 source extensions (.go .ts .tsx .js .cjs .mjs .py .sh .ps1 .css .html .yml .yaml .json) plus Makefile. Markdown is not scanned — the translated READMEs, CONTRIBUTING files and doc pages are legitimately non-English.

Wired into CI next to the existing license and action-pin checks, plus make english-check and make check for local runs. AGENTS.md states the rule in three parts: the rule itself, where translated prose may go (spelled out from what the tree actually holds, since Markdown is exempt by extension while the .ts i18n tables are exempt by prefix), and the two escape hatches.

What counts as non-English

A letter outside ASCII, whichever the writing system. Written English needs no letter beyond the ASCII 26, so anything past that belongs to another language: Cyrillic and Han as much as the diacritics that spell German, French or Vietnamese. Plus CJK and fullwidth punctuation, and combining accents.

Scripts are deliberately not enumerated. A language nobody has contributed in yet is covered on the day it arrives, with no edit to the script.

Three exclusions, each load-bearing:

• Common and Inherited pass. Those two scripts hold the characters belonging to no writing system in particular, and the letterlike symbols among them are letters only by Unicode category — the information source (U+2139, category Ll) that renders as an info icon, the script small l (U+2113), the capitals of the maths alphabets. An earlier revision without this exclusion flagged three ℹ️ lines.
• Symbols are out of scope by construction, because the test is "is a letter" rather than "is non-ASCII". The box drawing, arrows, emoji and maths in the TUI output (1343 instances of ─ alone) are not letters, and neither are the em dashes used throughout the comments.
• Combining accents are in scope, so the decomposed spelling cannot slip through: NFD writes e-acute as e plus U+0301, where the letter is plain ASCII and only the accent carries the language. Variation selectors are combining marks too, but sit outside that block and pass, so ⚠️ is fine.

Letterlike forms that Unicode assigns to a real script stay in scope, so the ohm sign (U+2126 — script Greek, since it is equivalent to U+03A9) is reported like any other Greek letter. A comment spelling sigma or omega as a glyph needs a marker. That is deliberate: exempting Greek to allow maths notation would exempt Greek prose with it.

Not detected: another language spelled entirely in ASCII — a romanised transcription, or German with its umlauts written out (Loeschen der Datei). Telling that from English takes a dictionary rather than a character test, so it stays a matter for review.

Design notes

• Go, not shell. No dependency on the CI container's grep having PCRE, and unicode decides what counts.
• //go:build ignore keeps the script out of ./..., so it does not affect go vet, go build or the 90% coverage threshold.
• Untracked files are included (--others --exclude-standard, ignored paths still excluded) so a new file is checked before it lands rather than the run after.
• git ls-files -z. Without it git quotes and escapes any path that is not plain ASCII — exactly the kind of path internal/diff/git_test.go has fixtures for.

Escape hatches, narrow one preferred

1. allow-non-english: <reason> marker comment on a single line — the rest of the file stays protected. 28 existing lines get one: UTF-8 encoding fixtures, multibyte truncation fixtures, language-switcher labels, heading-ID fixtures, and the fullwidth bar used as a terminal cursor. The colon is part of the marker, so a bare allow-non-english cannot exempt a line without giving a reason.
2. A prefix in allowedPrefixes for a whole tree. Only two entries: pages/src/i18n/ (translated UI copy) and extensions/vscode/, marked TEMPORARY.

Issues the check found

1. The rule was skewed by writing system rather than by language. In pages/src/components/Footer.tsx the zh and ja labels each needed a marker while the adjacent { value: 'ru', label: 'Русский' } passed untouched — a CJK-shaped rule cannot see Cyrillic. Nine lines of Russian sat in the tree unflagged: two language-switcher labels and the heading-ID fixtures in MarkdownRenderer.test.tsx and headingId.test.ts. Contributors writing Chinese had to justify every line; contributors writing Russian had nothing to justify. Widening the rule to "a letter outside ASCII" removes the skew and costs nothing: the whole tree contains four non-ASCII Latin letters, two of which already carry a marker.

2. internal/config/toolsconfig/tools.json advertised the file_read example output as File：path/to/example.go with a fullwidth colon (U+FF1A), while internal/tool/file_read.go:68 emits File: %s. That description is sent to the model on every review, so the example did not match the output it described. A Han-only check would have missed this.

3. The script's own doc comment used Chinese punctuation as an example and stayed green — because it was still untracked and git ls-files only lists tracked files. That is what motivated including untracked files; scripts/verify-license.sh has the same limitation today. It has since caught its own comments three more times during this PR (a 文件.ts path, a sigma/omega glyph, an umlaut), each time correctly.

4. main discarded the error. run() returned wrapped errors that nothing ever printed — main called os.Exit(1) and dropped them, so a git ls-files failure surfaced as a bare exit status with no diagnostic at all. Fixed alongside capturing git's stderr, which would otherwise have been invisible for the same reason.

Also switches action.yml's OCR_LANGUAGE example from 中文 to Chinese, for the same reason as #861: the value is fed to the LLM, and Chinese is what the rest of the project uses.

Verification

• make english-check — clean across 384 scanned files
• Negative tests: Han, kana, Hangul, Cyrillic, Greek, Arabic, Hebrew, Thai and Devanagari all reported; German, French, Turkish and Vietnamese diacritics reported; NFD-decomposed accents reported; symbols, emoji with variation selectors, ℹ️, ℓ and box drawing all pass; a bare allow-non-english without a reason does not exempt; an untracked file with non-English text is reported and exits 1
• make license-check, gofmt -l, go vet ./..., go test ./cmd/... ./internal/... — all pass
• npx tsc --noEmit, npx eslint, npx vitest run in pages/ — no errors
• python3 -m unittest in examples/gerrit_ci/ (61 tests) and node --test scripts/github-actions/*.test.js — pass

Follow-up

extensions/vscode/ still has roughly 450 non-English lines (comments, test names, package.nls.zh-cn.json). Once those are translated, the allowlist entry should be deleted — worth tracking as its own issue so the temporary entry does not become permanent.

[github-actions]

🔍 OpenCodeReview found 2 issue(s) in this PR.

• ✅ Successfully posted inline: 1 comment(s)
• 📝 In summary (no line info): 1 comment(s)

---

[maintainability · low]

📄 .github/workflows/ci.yml

⚠️ GitHub could not post this as an inline comment: No line information provided

When git ls-files fails, .Output() only captures stdout — the stderr content (which typically contains the actual error message from git) is discarded. This makes debugging failures harder. Consider using CombinedOutput() or wrapping with exec.ExitError stderr extraction so the CI log shows the real git error.

💡 Suggested Change

Before:

	out, err := exec.Command("git", "ls-files", "--cached", "--others", "--exclude-standard").Output()
	if err != nil {
		return fmt.Errorf("git ls-files: %w", err)
	}

After:

	out, err := exec.Command("git", "ls-files", "--cached", "--others", "--exclude-standard").CombinedOutput()
	if err != nil {
		return fmt.Errorf("git ls-files: %w\n%s", err, out)
	}

[github-actions]

[bug · medium]
When git ls-files fails (e.g., not a git repository, git not installed, corrupt index), Output() discards stderr. The resulting *exec.ExitError formats only as exit status N, losing the actual diagnostic message from git. This will make CI failures harder to debug.

Consider capturing stderr separately (or using CombinedOutput() if stdout pollution is acceptable here) and including it in the error message. For example:

cmd := exec.Command("git", "ls-files", "--cached", "--others", "--exclude-standard")
out, err := cmd.Output()
if err != nil {
    if exitErr, ok := err.(*exec.ExitError); ok {
        return fmt.Errorf("git ls-files: %w: %s", err, exitErr.Stderr)
    }
    return fmt.Errorf("git ls-files: %w", err)
}

Note: the codebase pattern in cmd/opencodereview/git.go uses CombinedOutput() for the same reason.

[wu21-web]

Suggested change

	out, err := exec.Command("git", "ls-files", "--cached", "--others", "--exclude-standard").Output()
	if err != nil {
	return fmt.Errorf("git ls-files: %w", err)
	}
	var findings []finding
	var scanned int
	for _, file := range strings.Split(strings.TrimSpace(string(out)), "\n") {
	out, err := exec.Command(
	"git", "ls-files", "-z", "--cached", "--others", "--exclude-standard",
	).CombinedOutput()
	if err != nil {
	return fmt.Errorf("git ls-files: %w: %s", err, strings.TrimSpace(string(out)))
	}
	var findings []finding
	var scanned int
	for _, file := range strings.Split(string(out), "\x00") {

[wu21-web]

Suggested change

	unicode.Is(unicode.Katakana, r):
	unicode.Is(unicode.Katakana, r):
	unicode.Is(unicode.Hangul, r):

[wu21-web]

Currently omitting Korean Hangul.

[wu21-web]

Suggested change

	- name: Verify sources are English-only
	- name: Verify sources contain no unapproved CJK text

[wu21-web]

Not quite accurate here, but fine.

[wu21-web]

Suggested change

	- Source files are English-only — comments, identifiers and strings alike. Translated prose belongs in `README.<locale>.md`, `pages/src/content/docs/<locale>/` or an i18n table. `make cjk-check` enforces this in CI; it also rejects fullwidth punctuation (`：`, `（`), which is easy to leave behind in an otherwise English sentence.
	- Source files must not contain unapproved CJK text in comments, identifiers or strings. Translated prose belongs in `README.<locale>.md`, `pages/src/content/docs/<locale>/` or an i18n table. `make cjk-check` enforces this in CI; it also rejects fullwidth punctuation (`：`, `（`), which is easy to leave behind in an otherwise English sentence.

[wu21-web]

This is as same as ci.yml:L35

[wu21-web]

Suggested change

	const exemptMarker = "allow-cjk"
	const exemptMarker = "allow-cjk:"

Maybe why: requiring the colon prevents incomplete allow-cjk substrings from accidentally exempting a line.

[stay-foolish-forever]

A few classes of non-English text slip past the check

I put each of these on its own line in an untracked .ts file and ran make english-check — none of them was reported:

Sample	Code points	Why it passes
⼀⼥⽥	U+2E80–U+2FD5, CJK and Kangxi radicals	script=Han, but category So, so not a letter
﹐﹖︑	U+FE10–U+FE6F, small form and vertical form CJK punctuation	only U+3000–U+303F and U+FF00–U+FFEF are range-checked
٢٠٢٦	U+0660–U+0669, Arabic-Indic digits	script=Arabic, category Nd
ー	U+30FC, Japanese long vowel mark	category Lm, but script=Common, so exclusion 1 clears it
𝛼𝛽	U+1D6A8+, maths-styled Greek	script=Common

Not sure whether the script should defend against these. The check is per-line, so real prose still gets caught by its other letters; it only matters for a line built entirely from the characters above. The U+FE10–U+FE6F row is the one I'd lean towards adding, since its U+FF00 counterparts are already caught and ﹖ reads as correct English punctuation in review.