Skip to content

fix(deep-interview): canonicalize Hangul before hashing and capping prose - #3872

Merged
Yeachan-Heo merged 3 commits into
Yeachan-Heo:devfrom
yazzang-homelab:fix/deep-interview-hangul-normalization
Aug 6, 2026
Merged

fix(deep-interview): canonicalize Hangul before hashing and capping prose#3872
Yeachan-Heo merged 3 commits into
Yeachan-Heo:devfrom
yazzang-homelab:fix/deep-interview-hangul-normalization

Conversation

@yazzang-homelab

@yazzang-homelab yazzang-homelab commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What

Canonicalize deep-interview prose to NFC at the identity and length boundaries: questionHash(), answerHash(), buildAnswerShell() (stored question_text / selected_options / custom_input), and assertDeepInterviewInputWithinLimit().

Why

Fixes the first half of #3871.

Hangul reaches the interview in both composed (NFC) and decomposed (NFD) form — macOS-sourced pastes and several clipboard/IME paths emit NFD — and the two forms are one text for the user but two different JavaScript strings for us:

19 39 false false   # [...nfc].length, [...nfd].length, questionHash equal?, answerHash equal?

That produced three Korean-only failures:

  • appendOrMergeRound() saw a different answer_hash for the same answer, so its documented noop became a duplicate/replaced round — one answer, two round histories.
  • assertDeepInterviewIntentReview() matches approval evidence by answer_hash against recorded answers, so crossing a normalization boundary invalidated the user's own approval.
  • Caps are code-point counted, so decomposed Hangul cost 2–3 units per syllable: MAX_USER_RESPONSE_LENGTH = 10_000 rejected Korean answers at roughly a third of the advertised budget while ASCII prose got the full 10,000.

#3716 asked models for literal UTF-8 in tool inputs, but it is a prompt directive; nothing in the runtime canonicalized user prose. This is the code-level half.

Testing

packages/coding-agent/test/deep-interview-hangul-normalization.test.ts — 5 tests: fixtures are genuinely distinct strings, one question identity across forms, one answer identity for both selected options and custom input, appendOrMergeRound returns noop with a single round when both forms arrive, and cap parity (10,000 composed and decomposed both pass; 10,001 decomposed throws).

bun test packages/coding-agent/test/deep-interview-hangul-normalization.test.ts   # 5 pass (4 fail on dev without the src change)
bun test packages/coding-agent/test/deep-interview-workflow-gates.test.ts \
         packages/coding-agent/test/deep-interview-skill-contract.test.ts \
         packages/coding-agent/test/deep-interview-gate-redteam.test.ts \
         packages/coding-agent/test/sdk-workflow-gate-emitter.test.ts \
         packages/coding-agent/test/agent-session-deep-interview-continuation.test.ts   # 102 pass
bun --cwd=packages/coding-agent run check   # biome + tsc clean

Also verified as not the cause and left untouched: replaying all 173 Korean ask questions from the local session corpus through renderDeepInterviewAskQuestion() / formatDeepInterviewSelectorPrompt() at widths 40–100 loses no Hangul and drifts no padding, so #1996 holds.

GJC verdict

gajae.pr-review-verdict.v1 needs-human sha256:a366111caf273ff2761e9267476ecaa157c8aae8 reviewer:human evidence:rebased onto dev 6c1c8a4798917be42071af02ab3c005ca71817c3 (exact-head CI base gate) + bun test packages/coding-agent/test/*deep-interview*.test.ts (89 pass) + bun test packages/coding-agent/test/gjc-runtime/ (1259 pass) + bun test tools/ask, workflow-gate-redteam, workflow-mutation-guard, workflow-state-command, gjc-skill-state-hooks (189 pass) + tsc -p packages/coding-agent --noEmit clean + biome clean on touched files

  • Target branch is dev
  • bun check passes (bun --cwd=packages/coding-agent run check)
  • Tested locally
  • CHANGELOG updated (if user-facing)
  • Verdict above matches the exact PR head, not an earlier commit

@yazzang-homelab
yazzang-homelab force-pushed the fix/deep-interview-hangul-normalization branch 2 times, most recently from a1c91b9 to a366111 Compare August 5, 2026 13:17
…rose

Hangul reaches the interview in both composed and decomposed form, so the
same Korean answer produced two different answer_hash values: the documented
append-or-merge no-op became a duplicate round, and intent-review approval
evidence stopped matching the user's own recorded answer. Code-point caps
also charged decomposed Hangul two to three units per syllable, so a Korean
user_response was rejected at roughly a third of the advertised budget.

Yeachan-Heo#3716 only asked models for literal UTF-8 in tool inputs; nothing in the
runtime canonicalized user prose. Normalizing at the identity and length
boundaries fixes both without touching stored English transcripts.

Lore-id: 7c41d0a9
Constraint: stored round prose must stay byte-stable for existing NFC state
Rejected: normalize only inside the hash | leaves NFD text in durable state
Rejected: normalize in deepInterviewCharacterCount | hides it from callers
Confidence: high
Scope-risk: narrow
Reversibility: easy
Tested: NFC/NFD question and answer identity, append-or-merge no-op, cap parity
Not-tested: real macOS clipboard capture of decomposed Hangul
Refs: Yeachan-Heo#3871
@yazzang-homelab
yazzang-homelab force-pushed the fix/deep-interview-hangul-normalization branch from a366111 to 392401c Compare August 5, 2026 23:24
@yazzang-homelab

Copy link
Copy Markdown
Contributor Author

Rebased onto current dev (11e48d5bc, #3901) — the previous head was CONFLICTING/DIRTY on the CHANGELOG ## [Unreleased] section. New head 392401cf6; rebase was CHANGELOG-only, no code merges, and the diff is otherwise unchanged.

Re-verified on the new base: bun test packages/coding-agent/test/deep-interview-hangul-normalization.test.ts → 5 pass, 0 fail.

@Yeachan-Heo
Yeachan-Heo merged commit 4bc67a8 into Yeachan-Heo:dev Aug 6, 2026
21 checks passed
@Yeachan-Heo

Copy link
Copy Markdown
Owner

This item is being moved to the Tier 3 review lane. Further review is deferred until after the current emergency stabilization period.


[repo owner's gaebal-gajae (clawdbot) 🦞]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants