Korean/non-ASCII tool calls are rejected instead of resampled: escapedNonAsciiArguments has no recovery path

[probepark]

Symptom

Any tool call carrying non-ASCII text (Korean, em dash, curly quotes, arrows) is frequently rejected before execution with:

Tool call "bash" spelled non-ASCII text as \uXXXX escapes instead of literal UTF-8.
Escaped text cannot be verified - a single wrong hex digit silently becomes a different
character - so the call was not executed. Re-issue it writing every non-ASCII character literally.

The guard itself (findUnnecessaryUnicodeEscape in packages/ai/src/utils/json-parse.ts -> escapedNonAsciiArguments -> rejection in packages/agent/src/agent-loop.ts) is correct: hand-spelled hex parses cleanly but one mistyped nibble decodes to a different, equally valid character, and nothing can repair that after parsing.

The problem is the recovery path, not the detection.

Measured impact

Local session logs, most recent 60 sessions, 7910 tool calls:

	tool calls	non-ASCII present	rejected
all	7910	175	116 (66%)
Hangul-carrying	-	85	66 (78%)

Per model: claude-opus-5 53/54 Hangul calls rejected, claude-sonnet-5 13/13, gpt-5.6-sol/gpt-5.6-terra 0/18. Only the anthropic-messages path is affected.

Not a transport defect

Direct probes against the same proxy and model could not reproduce it: 25+ curl runs varying payload length, thinking on/off, fine-grained-tool-streaming beta on/off, 30 extra tools, cloaked mcp__... tool names, 40-turn filler context, and a primed history containing the rejection error itself - all returned literal UTF-8, zero escapes. Six real gjc -p runs with wire capture (45-76 tools, ~200KB requests, one 929-Hangul-character write call) also produced zero escapes.

It is an intermittent sampling accident that shows up in long sessions. In session logs the flag clusters in runs with 700-1000 assistant messages and fires in consecutive bursts (F@144 F@145 F@146 F@147).

Why the current handling makes it worse

1. A wire-format defect is promoted to a tool error, spending the entire turn.
2. The rejection text re-injects the literal \uXXXX syntax into the context the model samples from next.
3. There is no bounded resample, and no config switch (0 related keys in config.schema.json).

The anthropic.ts thinking-replay repair already establishes the right shape for this class of fault: when the defect is visible before the turn is usable, drop it and re-request within a bounded budget instead of burning the turn.

Fix

Bounded turn resample in the non-managed session path, alongside the existing invalid_prompt and reasoning-content repairs: drop the defective assistant turn from history and re-request, up to twice per turn. The terminal per-call rejection stays as the answer once the budget is spent, and MAX_CONSECUTIVE_MALFORMED_TURNS remains reachable as the loop backstop.

[Yeachan-Heo]

probe 형님, 재현·정량·원인 경계를 깔끔하게 잡아주셔서 바로 전담 lane을 붙였습니다.

Current owner starts from exact dev@cf07293f. 기존 detector/rejection은 유지하고, non-managed agent-loop에서만 malformed assistant turn을 usable history에서 제거한 뒤 logical turn당 최대 2회 bounded resample하는 가설을 검증합니다. 핵심 acceptance는 history에 \\uXXXX rejection text를 재주입하지 않는 것, budget 소진 시 기존 terminal rejection/backstop이 그대로 도달 가능한 것, validation 전에 어떤 tool도 실행되지 않는 것, managed/OpenAI 경로에 회귀가 없는 것입니다.

Deterministic coverage 후 한 PR로 dev merge와 issue close까지 맡겠습니다. 제안된 retry shape가 실제 history/stream ownership과 맞지 않으면 임의 구현하지 않고 exact blocker/evidence를 먼저 공유하겠습니다.

—
[repo owner's gaebal-gajae (clawdbot) 🦞]

[Yeachan-Heo]

Maintainer successor #4491 now owns the current-dev integration of this contribution.

Attribution and intent are preserved: probepark's original authored commit was reconstructed onto exact current dev as signed commit d55549dc5d32a18a05085f6003fdb27b7a1fd5f5; maintainer hardening is signed commit bcfe159d9c8947f38a77bbc13a1de2a4bc5870d9.

Reproduced ancestry/CI evidence:

• PR head: f91f20e3457b5e49de6525b3b62644176a1c7a21
• recorded base: cf07293f7404b35c085dde35013aa5a12d595ad5
• actual PR-head parent/base line: 61f443d9bff05ef24cc7f2e6380da1d23e6a863b
• current dev: f0a44081afe3dbebeba3afb9ab380517ebd6c8d7
• git merge-base --is-ancestor cf07293... f91f20e... fails
• Dev CI affected-path plan failed closed with empty plan digest/source/mode, then evidence/final jobs failed transitively

Local current-head evidence for bcfe159d9c8947f38a77bbc13a1de2a4bc5870d9:

• focused agent + provider regressions: 34 pass
• full packages/agent/test: 751 pass
• packages/agent check/types: pass
• packages/ai check/types: pass
• native build: pass
• Rust workspace check: pass
• coding-agent binary build: pass after native build

The broad AI suite currently has unrelated baseline/environment failures (58 failures, also outside the changed files); no changed-path provider regression failed. Root check:ts is blocked by a pre-existing generated SDK operation-inventory drift reproduced unchanged on exact base f0a44081....

This PR is respectfully superseded rather than force-updated because its head repository is the core repository and maintainer_can_modify is false. Thank you, @probepark, for the measured evidence and focused implementation.

Signed-off-by: Yeachan Heo yeachan.heo@gmail.com

[Yeachan-Heo]

Merged to dev as signed merge commit cf29e8dc82 after exact-head CI success and independent adversarial MERGE_READY on e56f2c1586e54c4705beb8d4ee44a2f110e59193.

Attribution is preserved: probepark's original measured report and authored implementation remain the foundation of the merged series. Maintainer fix-forwards closed public/durable lifecycle ownership, callback continuity, logical-turn tool-choice preservation, and live streaming safety ordering.

Premerge evidence:

• full agent suite: 756 pass
• AI event-stream + changed-provider suites: 25 pass
• agent + AI checks/types: pass
• exact-head Dev CI run 31726560703: success
• all review threads resolved

Thank you, @probepark, for the strong reproduction, quantitative evidence, and focused initial patch.

Signed-off-by: Yeachan Heo yeachan.heo@gmail.com

[Yeachan-Heo]

Postmerge correction: a newer P1 arrived after #4491 merged, so issue #4489 has been reopened and exact-head successor #4500 now owns the metadata-fidelity fix-forward.

#4491 remains the attributed implementation merge. #4500 changes only unmanaged provisional snapshotting to preserve provider-native metadata; managed fallback normalization remains unchanged.

Signed-off-by: Yeachan Heo yeachan.heo@gmail.com

[Yeachan-Heo]

Issue #4489 disposition refresh — hold remains active, tied to PR #4515 exact head 9bf3c2e8d5a5a4af2ab52f1932e3a2e574967280

Refreshed full state (2026-08-14, session gajae-code-issue-4489-nonascii-resample-disposition):

Issue #4489: OPEN/REOPENED. Ownership chain re-verified: #4490 CLOSED → #4491 MERGED (resample core, e56f2c1586e) → #4500 MERGED (provisional metadata, a3c52fa0cd) → #4515 OPEN (sole nonterminal owner; cross-referenced 2026-08-13T19:19:33Z). No other open issue/PR references #4489; no overlapping owner exists.

PR #4515 (fix/issue-4489-escaped-nonascii-resample-successor):

• Head 9bf3c2e8d5 unchanged since the approved review; recorded base 34f5db331b (merge-base of head and dev confirmed 34f5db331b).
• vs current dev e6f8c926dd: mergeable=CONFLICTING / DIRTY — the PR must be rebased by its owner onto current dev before merge; exact-head approval (probepark, 2026-08-14T05:38:48Z) and terminal CI (run 31773753220 on 9bf3c2e8d5: Affected path validation incl. agent-loop-escaped-nonascii-toolcall.test.ts + agent-loop-harmony-leak.test.ts, evidence producer, Virtual integration validation, gjc-state-gates, native-build — all SUCCESS; only Windows/Telegram matrix rows SKIPPED) stand at that head.

Independent acceptance-contract verification (read-only, this lane's dedicated worktree at clean dev e6f8c926dd, no source mutation):

#	Contract item	Verdict	Evidence
1	Detector/rejection retained	✅	findUnnecessaryUnicodeEscape present and unchanged at both dev HEAD and PR head (packages/ai/src/utils/json-parse.ts:131); terminal per-call rejection text preserved verbatim in executeToolCalls
2	Bounded non-managed resample, max 2/turn, drop defective turn from history	✅	MAX_ESCAPED_NONASCII_RESAMPLES = 2; gate !config.fallbackManaged && stopReason∉{error,aborted} && attempt<2 && !committed && hasEscapedNonAsciiToolCall → removeCommittedAssistantMessage + continue (agent-loop.ts ~2094-2104 at PR head). Verified identical in dev HEAD and PR head — dev already contains the merged #4491/#4500 core
3	No \uXXXX rejection-text re-injection into history	✅	Resample path drops the turn and re-requests; no error text is written to context. Test: "preserves all pre-existing history and removes only the defective assistant turn"
4	Budget exhaustion → terminal rejection + MAX_CONSECUTIVE_MALFORMED_TURNS backstop reachable	✅	Post-budget turn falls through to per-call throw; tests "rejects the call once the resample budget is spent" + "still reaches the malformed-turn circuit breaker after persistent escaped calls"
5	No tool executes before validation	✅	Rejection throws inside runInActiveSpan before tool.execute invocation args are built/dispatched; tests "executes no calls from a mixed batch before validating the whole turn", "does not dispatch when an accepted assistant callback aborts during commit"
6	No managed/OpenAI regression	✅	Gate scoped !config.fallbackManaged; tests "leaves managed fallback handling unchanged", "executes literal UTF-8 arguments untouched"; Harmony suite scoped-to-codex/managed-no-retry cases green

Local focused verification on dev e6f8c926dd: bun test packages/agent/test/agent-loop-escaped-nonascii-toolcall.test.ts + agent-loop-harmony-leak.test.ts → 23 pass / 0 fail; full packages/agent suite → 768 pass / 0 fail (1 env-only failure with a local OPENAI_BASE_URL override set, passes with the env unset); packages/agent + packages/ai biome + tsc --noEmit clean.

Coverage gaps found: none. PR #4515 still fully covers the issue acceptance contract at its exact head. Its remaining nonterminal conditions are owner-side only: (a) DIRTY base vs current dev e6f8c926dd — rebase/merge required; note PR #4546 (managed-snapshot diagnostics, +54/−12 in the same agent-loop.ts staging regions: ManagedAttemptSnapshotError sites, #losslessSnapshot, #wouldOverflow) is also OPEN against the same file, so rebase conflict resolution in ManagedAttemptTransaction staging is expected and is the PR owner's to perform.

This lane holds: no duplicate code, no second PR, no source mutation. Issue #4489 stays open until #4515 is merged into dev; on merged-history + product-evidence proof this lane will close #4489 with signed evidence immediately.

—
[repo owner's gaebal-gajae (clawdbot) 🦞]

[probepark]

Managed-fallback sessions still get zero resamples: !config.fallbackManaged is a coverage hole, not a safe no-op

Hit this live on dev@bc63eaaa8 (3 consecutive hard rejections writing Hangul through write and bash, no recovery at all), then traced why the merged #4491/#4500 resample never engaged. This is not a regression in those PRs -- it is a gap the current acceptance contract records as intended.

The claim in the code is false

The resample gate (packages/agent/src/agent-loop.ts ~2107-2114):

if (
	!config.fallbackManaged &&
	message.stopReason !== "error" &&
	message.stopReason !== "aborted" &&
	escapedNonAsciiResampleAttempt < MAX_ESCAPED_NONASCII_RESAMPLES &&
	!escapedToolTransaction?.committed &&
	hasEscapedNonAsciiToolCall(message)
)

The comment directly above it justifies the scoping with: "Managed fallback owns its own retry policy, so this is scoped to the non-managed session path."

That premise does not hold:

1. ManagedAttemptOutcome (packages/agent/src/types.ts:175-187) has exactly three variants -- retryable_discarded, context_overflow_discarded, run_terminal. None of them can encode an escaped-non-ASCII tool-argument defect, so the defect never reaches onManagedAttemptOutcome / #handleManagedAttemptOutcome at all.
2. grep -c escapedNonAscii packages/agent/src/agent.ts -> 0. The managed fallback owner has no code path for this signal whatsoever. Repo-wide, escapedNonAscii appears only in agent-loop.ts, proxy.ts, types.ts, utils/tool-call-healing.ts, and the provider adapters.

So for a managed session the flag goes straight from toolcall_end to the terminal per-call throw in executeToolCalls (agent-loop.ts:3200), with zero recovery attempts -- exactly the pre-#4491 behavior this issue was opened to fix.

The test codifies the gap as correct

packages/agent/test/agent-loop-escaped-nonascii-toolcall.test.ts:660 -- "leaves managed fallback handling unchanged" -- asserts executed is empty and isError is true on the first escaped turn under fallbackManaged: true. Acceptance criterion 6 in the disposition refresh above reads that green result as "No managed/OpenAI regression", but what it actually proves is that managed sessions get no recovery at all.

How many users this silently excludes

packages/coding-agent/src/session/agent-session.ts:17222-17234:

#managedFallbackPromptOptions(): { ... } {
	const controller = this.#defaultFallbackChain();
	if (controller.chain.entries.length < 2) return {};
	return { fallbackManaged: true, ... };
}

fallbackManaged is turned on for any session whose default fallback chain has 2 or more entries -- it is not an opt-in debug mode. So every user with a configured fallback chain runs every turn on the unfixed path.

Combined with this issue's own per-model data (claude-opus-5 53/54 Hangul calls rejected, claude-sonnet-5 13/13, anthropic-messages path only), a two-entry Anthropic fallback chain is simultaneously the worst-affected configuration and the one with zero recovery.

There is also no escape hatch: grep -rn "escapedNonAscii" schemas/config.schema.json -> 0 matches, so an affected user's only workaround is to drop their fallback chain below 2 entries.

Detector is fine -- confirmed again

Not disputing detection. Verified directly against packages/ai/src/utils/json-parse.ts:

literal payload : {"command":"echo hi","note":"<hangul>"}  -> not flagged
escaped payload : {"command":"echo hi","note":"\ud55c..."} -> flagged (\ud55c)
both decode to the identical string

That last line is the whole point: the two payloads are indistinguishable after parsing, so the escape is the only evidence and post-hoc repair is impossible. Detection should stay exactly as is.

Suggested disposition

Either:

• (a) extend recovery to managed -- add a fourth ManagedAttemptOutcome variant (e.g. escaped_arguments_discarded) that #handleManagedAttemptOutcome maps to a bounded retry continuation, so the turn is dropped and re-requested on the same model before the fallback chain advances; or
• (b) declare it out of scope explicitly -- if managed sessions are deliberately excluded, then the code comment ("Managed fallback owns its own retry policy") and acceptance criterion 6 should both be reworded, because as written they assert a policy that does not exist and will mislead the next reader of this gate.

(a) looks correct to me: the defect is a wire-format accident independent of which model in the chain produced it, and advancing the fallback chain is the wrong response to it.

Happy to take whichever direction the owner of #4515 prefers.

[probepark]

Correction to my previous comment, plus a sharper repro signal

Correcting one attribution error in my comment above before it misleads anyone.

What I got wrong: I wrote that my live repro was explained by the managed-fallback gap. It was not. I checked the session config afterwards: modelRoles: {} in ~/.gjc/agent/config.yml, so #defaultFallbackChain() resolves to a single entry, chain.entries.length < 2, and #managedFallbackPromptOptions() returns {}. That session ran on the non-managed path with the #4491 resample fully active. My repro is not evidence for the managed gap.

What still stands: the managed-fallback gap itself is verified from code alone and is unaffected by that error -- ManagedAttemptOutcome has no variant for this defect, agent.ts contains zero references to escapedNonAscii, the gate is !config.fallbackManaged, and #managedFallbackPromptOptions() flips it on for any chain with 2+ entries. The suggested disposition (a)/(b) is unchanged.

The stronger signal: the resample budget did not clear it

What my repro actually shows is more relevant to PR #4515 than the managed gap.

On the non-managed path, with MAX_ESCAPED_NONASCII_RESAMPLES = 2 active, I still got 3 consecutive terminal rejections on Hangul payloads across two different tools (write and bash), back to back, with no successful call in between. Every one of those turns had the resample path available to it.

Two possible readings, and I have not yet isolated which:

1. The defect is deterministic for this payload class, not intermittent. Each rejection burned its 2 resamples and every resample re-emitted the same escaping, so the budget was exhausted 3 times in a row. If so, the issue's framing -- "intermittent sampling accident", "a wire-format accident that resampling clears" -- does not hold for Hangul-heavy payloads, and a fixed budget of 2 is structurally the wrong shape: it converts a hard failure into the same hard failure three samples later.
2. A different gate condition short-circuited the resample. The gate also requires message.stopReason !== "error" && message.stopReason !== "aborted" and !escapedToolTransaction?.committed. If escapedToolTransaction?.committed is true by the time the check runs for these turns, the resample is skipped even on the non-managed path -- which would be a live bug in the merged fix rather than a budget-sizing question.

Distinguishing them is cheap and worth doing before #4515 lands: log (or assert in a test) escapedNonAsciiResampleAttempt and escapedToolTransaction?.committed at the moment of the terminal throw in executeToolCalls. If attempts is 2, it is reading 1; if attempts is 0, it is reading 2.

For reference, the payloads that triggered it were ordinary prose -- Korean interview state written through write, and a gjc deep-interview write --input "$(cat ...)" heredoc through bash. Nothing exotic: no surrogate pairs beyond normal Hangul, no mixed scripts, no emoji.

Apologies for the noise on the first comment; the code-level finding was verified but the repro attribution was not, and I should have checked the chain length before publishing it.

[Yeachan-Heo]

#4489 recovered disposition — implementation owner retained; issue remains open

Fresh read at origin/dev@9d2a2d2f2d8f074333bba92fe1fc4d6902f5b2df confirms PR #4515 is still the sole nonterminal implementation owner: fix/issue-4489-escaped-nonascii-resample-successor@9bf3c2e8d5a5a4af2ab52f1932e3a2e574967280, recorded base 34f5db331b484726c447f6ab026f35c8b391a879, GitHub state OPEN / CONFLICTING / DIRTY, review decision APPROVED. Its old exact-head CI was green (including agent-loop-escaped-nonascii-toolcall.test.ts), but it is stale evidence: the head is not integrated and cannot merge against current dev.

The recovered implementation lane is intact at /home/bellman/Workspace/gajae-code-pr-4490-current-head on that exact branch/head; it is the only lane authorized to rebase and edit the owned source. Local exact-head regression confirmation there: bun test packages/agent/test/agent-loop-escaped-nonascii-toolcall.test.ts → 21 pass, 0 fail, 94 expectations. This disposition worktree made no implementation edits.

Acceptance-contract blocker

Current dev still hard-gates the resample branch with !config.fallbackManaged. The dedicated regression suite explicitly asserts "leaves managed fallback handling unchanged" by expecting the escaped call to end as one error and no execution. That is a documented coverage exclusion, not recovery coverage, and it conflicts with this issue’s recovery outcome for managed-fallback sessions. The correction in the latest probepark report also matters: its live non-managed repro does not prove this managed gap; it remains an independent code-proven gap. The owner must add a deterministic default/non-managed reproduction before attributing that live failure to a different path.

Terminal dependency

Do not close #4489 on approval or historical CI. PR #4515 must rebase onto current dev, resolve its GitHub conflict state, replace the managed-fallback rejection expectation with bounded managed recovery coverage (or explicitly narrow/replace the issue contract through a new signed decision), add the separate default-path repro fixture, then obtain fresh exact-head CI and review. Only after merge to dev should this issue close, with the merge SHA plus fresh-dev product evidence.

Signed-off-by: Yeachan Heo yeachan.heo@gmail.com

— gaebal-gajae