Skip to content

feat: configure model output and Bedrock stream limits - #459

Open
Geonwoo Jeong (geonwoo-jeong) wants to merge 2 commits into
langchain-ai:mainfrom
geonwoo-jeong:codex/configurable-max-output-tokens
Open

feat: configure model output and Bedrock stream limits#459
Geonwoo Jeong (geonwoo-jeong) wants to merge 2 commits into
langchain-ai:mainfrom
geonwoo-jeong:codex/configurable-max-output-tokens

Conversation

@geonwoo-jeong

@geonwoo-jeong Geonwoo Jeong (geonwoo-jeong) commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Closes #474
Closes #485

Context: #458 and #457 cover truncated tool-call detection. This PR stays separate from that failure guard and only exposes model-client runtime limits.

Summary

  • Add an optional OPENWIKI_MAX_OUTPUT_TOKENS setting across supported model providers.
  • Add an optional Bedrock-only OPENWIKI_STREAM_IDLE_TIMEOUT setting for the first/inter-chunk watchdog.
  • Preserve provider defaults when either setting is unset.
  • Keep the Bedrock-only setting inert for other providers, including validation and diagnostics.

Why these controls belong together

OpenWiki currently exposes neither the model output budget nor Bedrock's stream idle watchdog. The two limits are independent: maxTokens controls the allowed response length, while streamIdleTimeout controls how long the Bedrock client waits between chunks.

We observed the 60-second LangChain watchdog after enabling a larger output budget, but this PR does not claim that a larger budget causes first-chunk latency. It exposes both controls so operators can configure the output budget and the watchdog appropriate to their workload without replacing either provider default globally.

Exact behavior

OPENWIKI_MAX_OUTPUT_TOKENS

  • Must be a positive integer.
  • Maps to maxOutputTokens for Google clients.
  • Maps to maxTokens for Anthropic, OpenAI, OpenAI-compatible, OpenRouter, Bedrock, Gemini Enterprise non-Google surfaces, ChatGPT OAuth, and GitHub Copilot.
  • Remains absent when unset.
  • Provider and model limits still apply; unsupported values may be rejected.

OPENWIKI_STREAM_IDLE_TIMEOUT

  • Applies only when the selected provider is Bedrock.
  • Must be an integer from 0 to 2147483647 milliseconds.
  • Maps to ChatBedrockConverse.streamIdleTimeout.
  • 0 uses LangChain's documented watchdog-disable behavior and produces a diagnostic warning that stalled streams may hang indefinitely.
  • Remains absent when unset, preserving the @langchain/aws default.
  • Stale or invalid values are ignored while another provider is active.

Scope

This PR changes model construction, environment management/diagnostics, documentation, and focused tests for these two runtime limits. It does not patch dependencies, alter prompts or workflows, or add truncation detection.

Validation

  • Constructor-level coverage for Bedrock, Gemini, Gemini Enterprise, Anthropic, OpenAI, OpenAI-compatible, OpenRouter, ChatGPT OAuth, and GitHub Copilot.
  • Copilot regression coverage preserves GPT-5 Responses API routing and non-GPT-5 chat-completions routing while applying the configured output limit.
  • Provider-aware regression coverage confirms the Bedrock timeout is ignored by other providers.
  • Validation and diagnostics coverage includes unset, valid, invalid, maximum timer, and watchdog-disabled values.
  • pnpm install --frozen-lockfile
  • pnpm test — 64 files, 792 tests
  • pnpm run build
  • pnpm run typecheck
  • pnpm run lint:check
  • pnpm run format:check
  • pnpm audit --prod — no known vulnerabilities

@geonwoo-jeong
Geonwoo Jeong (geonwoo-jeong) marked this pull request as ready for review July 24, 2026 12:47
@geonwoo-jeong
Geonwoo Jeong (geonwoo-jeong) force-pushed the codex/configurable-max-output-tokens branch from 16766a7 to 119ff55 Compare July 27, 2026 05:41
@geonwoo-jeong Geonwoo Jeong (geonwoo-jeong) changed the title feat: add configurable model output token limit feat: configure model output and Bedrock stream limits Jul 27, 2026
Expose OPENWIKI_STREAM_IDLE_TIMEOUT in milliseconds, accept 0 to disable the LangChain watchdog, and pass the option only to ChatBedrockConverse.

Validate the value against Node's timer range, surface it in diagnostics and debug logs, and preserve the output-limit, truncation guard, and patched @langchain/aws behavior.
@geonwoo-jeong
Geonwoo Jeong (geonwoo-jeong) force-pushed the codex/configurable-max-output-tokens branch from 119ff55 to 41a30d6 Compare July 28, 2026 06:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Make Bedrock stream idle timeout configurable [Feature]: Make model output token limit configurable

1 participant