Skip to content

feat: add configurable TTFT timeout fallback#576

Open
NaNomicon wants to merge 9 commits intodecolua:masterfrom
NaNomicon:agent/sisyphus/ttft-upstream-pr
Open

feat: add configurable TTFT timeout fallback#576
NaNomicon wants to merge 9 commits intodecolua:masterfrom
NaNomicon:agent/sisyphus/ttft-upstream-pr

Conversation

@NaNomicon
Copy link
Copy Markdown

@NaNomicon NaNomicon commented Apr 13, 2026

Summary

  • add configurable ttftTimeoutMs and ttftCooldownMs settings so slow streaming requests can abort before the first token and fall back to the next eligible account or combo member
  • recognize ttft_timeout in fallback handling, race request-start to first-byte arrival in chatCore, and defer stream success bookkeeping until the first chunk actually arrives
  • expose TTFT timeout/cooldown controls in the dashboard settings UI and surface TTFT timeout fallback details in request details

Why

Some upstream providers can hold a streaming request open for too long before the first byte arrives. When that happens today, fallback is delayed even if healthier accounts or combo members are available. This change lets operators cap time-to-first-token and recover automatically instead of waiting on a slow first response.

Notes

  • scope stays per-account, reusing the existing cooldown/fallback flow
  • 0 keeps TTFT timeout disabled
  • no per-provider or per-model TTFT overrides are introduced
  • branch was cherry-picked onto upstream master and excludes local planning artifacts

@NaNomicon
Copy link
Copy Markdown
Author

Verification note: I validated the feature behavior on the fork branches before splitting these PRs. When re-verifying the cherry-picked upstream branches locally, the existing upstream build script (next build --webpack) failed immediately because the installed Next CLI in that environment rejected the --webpack flag. That failure appears to come from the current upstream/base environment rather than this TTFT feature diff itself, but I wanted to call it out explicitly for reviewers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant