Skip to content

[FEATURE] Model canary: smoke-test every model in the context-window tables nightly #5634

Description

@yonib05

Human Overview

SDK Language

Both / Not language-specific

Problem Statement

The nightly integration run of main (#5503) includes the per-SDK model smoke tables added in #5452, so every model in those tables gets a real tool-using turn and a context-window check each night. The tables only cover the October 2026 catalog batch. Models already in the context-window tables before that batch, including the Claude, Nova and GPT families, have no row, so a provider-side change to one of them is found by users rather than by the canary.

Proposed Solution

Add a smoke row for every model in _CONTEXT_WINDOW_LIMITS (Python) and CONTEXT_WINDOW_LIMITS (TypeScript) that the integration accounts can reach, and add a check that fails when a model is in the context-window table without a smoke row. Failures surface through the canary issues from #5577.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-modelRelated to models or model providersenhancementNew feature or request

    Fields

    Language

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions