Skip to content

Classify gpt-6-astra as reasoning + text/image-capable in OpenAI model metadata #54

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Copilot wants to merge 3 commits into
base: trunk
Choose a base branch
from
+107 −9
Draft

Classify gpt-6-astra as reasoning + text/image-capable in OpenAI model metadata #54

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6723ad7
Initial plan
Copilot Sep 24, 2026
e0e9705
Fix GPT-6 Astra metadata classification
Copilot Sep 24, 2026
96449a6
Update src/Metadata/OpenAiModelMetadataDirectory.php
jeffpaul Sep 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 51 additions & 9 deletions src/Metadata/OpenAiModelMetadataDirectory.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,17 +33,19 @@ class OpenAiModelMetadataDirectory extends AbstractOpenAiCompatibleModelMetadata
/**
* Regular expression matching the model ID prefixes of OpenAI reasoning models.
*
* Reasoning models (codex-mini-latest, the versioned GPT-5 family, and the verified o1, o3,
* and o4 families) must be classified separately from standard GPT models. Only o-families
* whose behavior has been verified are recognized; new o-families require documentation and
* tests before they are added. GPT-5 chat aliases share a reasoning-family prefix but are
* non-reasoning models and are handled separately by self::isNonReasoningChatModel().
* Reasoning models (codex-mini-latest, the versioned GPT-5 family, GPT-6 Astra, and the
* verified o1, o3, and o4 families) must be classified separately from standard GPT models.
* Only o-families whose behavior has been verified are recognized; new o-families require
* documentation and tests before they are added. GPT-5 chat aliases share a
* reasoning-family prefix but are non-reasoning models and are handled separately by
* self::isNonReasoningChatModel().
*
* @since 1.1.0
*
* @var string
*/
private const REASONING_MODEL_ID_PATTERN = '/^(?:codex-mini-latest|gpt-5(?:\.\d+)?|o(?:1|3|4))(?:-|$)/';
private const REASONING_MODEL_ID_PATTERN =
'/^(?:codex-mini-latest|gpt-5(?:\.\d+)?|gpt-6-astra(?:-\d{4}-\d{2}-\d{2})?|o(?:1|3|4))(?:-|$)/';

@saarnilauri saarnilauri Sep 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The (?:-\d{4}-\d{2}-\d{2})? group added after gpt-6-astra has no effect here. The alternation is followed by (?:-|$), so gpt-6-astra on its own already matches anything that starts with gpt-6-astra-, including gpt-6-astra-2026-09-01. I checked, and classification is the same with or without the group.

It also makes the pattern look like only bare and dated Astra IDs count as reasoning models. That's the assumption behind the mismatch I mentioned on line 424, so the group is misleading as well as redundant.

For comparison, the same group is needed in EFFORT_NONE_DEFAULT_MODEL_ID_PATTERN (line 61) because that pattern ends in $. And gpt-5(?:\.\d+)? is needed here because . isn't -, so (?:-|$) wouldn't match gpt-5.2.

I'd drop the group, which also lets the constant fit on one line again:

private const REASONING_MODEL_ID_PATTERN = '/^(?:codex-mini-latest|gpt-5(?:\.\d+)?|gpt-6-astra|o(?:1|3|4))(?:-|$)/';

(If you go with the generation-level constant from line 424, this line would reference that instead.)


/**
* Regular expression matching the IDs of reasoning models that use reasoning effort `none` by default.
Expand Down Expand Up @@ -157,6 +159,26 @@ protected function parseResponseToModelMetadataList(Response $response): array
),
new SupportedOption(OptionEnum::outputModalities(), [[ModalityEnum::text()]]),
]);
$gptTextAndImageInputOptions = array_merge($gptBaseOptions, $gptSamplingOptions, [

@saarnilauri saarnilauri Sep 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please read this together with my comment on line 424 before deleting anything.

As the PR stands, this array is never selected. For every ID where supportsTextAndImageInputOnly() returns true:

  • isReasoningModel() is true (line 48 now includes gpt-6-astra)
  • isNonReasoningChatModel() is false (/^gpt-5-chat-latest$/)
  • hasDefaultReasoningEffortNone() is false (/^gpt-(?:5\.[12]|5\.4(?:-(?:mini|nano))?)…$/)

so supportsSamplingOptions() is always false, and the ternary on lines 354–356 always picks $gptReasoningTextAndImageInputOptions. The data provider in this PR asserts the same thing on lines 98–99 ('gpt-6-astra' => false).

It's unreachable because the PR only handles Astra, which is the one GPT-6 model that can't use reasoning_effort: none. According to the docs, Sol and Luna can. Once the generation is covered as suggested on line 424, a text + image model that also advertises sampling options is exactly what this array is for. I checked locally, and Luna and Sol end up here.

So there are two consistent options:

  • Cover the generation on line 424: keep this array. The ternary on lines 354–356 then actually does something.
  • Keep the PR scoped to Astra only: delete this array (162–171), its use capture (line 296), and the ternary, and assign $modelOptions = $gptReasoningTextAndImageInputOptions; directly. Unreachable code suggests behavior the code doesn't have.

The current state, where the array looks intentional but nothing can reach it, is the one to avoid.

Separately: this method now has three pairs of option arrays with the same structure ($gptOptions/$gptReasoningOptions, $gptMultimodalInputOptions/$gptReasoningMultimodalInputOptions, and the new pair). They differ only in whether $gptSamplingOptions is merged in and which modality list is used, and the use clause is up to 17 variables. This PR doesn't need to fix that, but if the modality list came from a per-family helper, each pair would collapse into one expression:

array_merge(
    $gptBaseOptions,
    self::supportsSamplingOptions($modelId) ? $gptSamplingOptions : [],
    [
        new SupportedOption(OptionEnum::inputModalities(), self::inputModalitiesFor($modelId)),
        new SupportedOption(OptionEnum::outputModalities(), [[ModalityEnum::text()]]),
    ]
);

new SupportedOption(
OptionEnum::inputModalities(),
[
[ModalityEnum::text()],
[ModalityEnum::text(), ModalityEnum::image()],
]
),
new SupportedOption(OptionEnum::outputModalities(), [[ModalityEnum::text()]]),
]);
$gptReasoningTextAndImageInputOptions = array_merge($gptBaseOptions, [
new SupportedOption(
OptionEnum::inputModalities(),
[
[ModalityEnum::text()],
[ModalityEnum::text(), ModalityEnum::image()],
]
),
new SupportedOption(OptionEnum::outputModalities(), [[ModalityEnum::text()]]),
]);
$gptMultimodalSpeechOutputOptions = array_merge($gptBaseOptions, $gptSamplingOptions, [
new SupportedOption(
OptionEnum::inputModalities(),
Expand Down Expand Up @@ -271,6 +293,8 @@ static function (array $modelData) use (
$gptReasoningOptions,
$gptMultimodalInputOptions,
$gptReasoningMultimodalInputOptions,
$gptTextAndImageInputOptions,
$gptReasoningTextAndImageInputOptions,
$gptMultimodalSpeechOutputOptions,
$gptSearchOptions,
$imageCapabilities,
Expand Down Expand Up @@ -325,7 +349,12 @@ static function (array $modelData) use (
&& !str_contains($modelId, '-realtime')
&& !str_contains($modelId, '-transcribe')
) {
if (self::supportsMultimodalTextInput($modelId)) {
if (self::supportsTextAndImageInputOnly($modelId)) {

@saarnilauri saarnilauri Sep 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only matters once line 424 is widened, so please treat it as part of that change.

This branch runs before the supportsMultimodalTextInput() branch, which means it also runs before the -audio / -search handling nested at lines 361–364. Right now that's fine, because the $ anchor on line 424 keeps any suffixed ID from reaching this point.

If the pattern is widened to the prefix form suggested on line 424, a GPT-6 ID with a -search or -audio suffix would match here first and get $gptReasoningTextAndImageInputOptions, skipping $gptSearchOptions / $gptMultimodalSpeechOutputOptions. I'm not saying such an ID exists today. Lines 361–364 exist because other families ship those suffixes, and the handling should apply regardless of family.

Since the suffix checks apply across families, they could move ahead of the family dispatch instead of being nested inside one branch:

if (str_contains($modelId, '-audio')) {
    // ... existing audio handling
} elseif (str_contains($modelId, '-search')) {
    // ... existing search handling
} elseif (self::supportsTextAndImageInputOnly($modelId)) {
    // ... GPT-6
} elseif (self::supportsMultimodalTextInput($modelId)) {
    // ... existing multimodal handling
}

A regression test with a synthetic -search GPT-6 ID would keep this from coming back.

$modelCaps = $gptCapabilities;
$modelOptions = self::supportsSamplingOptions($modelId)
? $gptTextAndImageInputOptions
: $gptReasoningTextAndImageInputOptions;
} elseif (self::supportsMultimodalTextInput($modelId)) {
$modelCaps = $gptCapabilities;
$modelOptions = $gptMultimodalInputOptions;
// New multimodal output model for audio generation.
Expand Down Expand Up @@ -382,12 +411,25 @@ private static function supportsMultimodalTextInput(string $modelId): bool
);
}

/**
* Checks whether an OpenAI text generation model supports text and image input only.
*
* @since n.e.x.t
*
* @param string $modelId The model ID.
* @return bool True if the model supports text and image input only, false otherwise.
*/
private static function supportsTextAndImageInputOnly(string $modelId): bool
{
return (bool) preg_match('/^gpt-6-astra(?:-\d{4}-\d{2}-\d{2})?$/', $modelId);

@saarnilauri saarnilauri Sep 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking. This matches a single model ID, and the rest of the generation is left out. /v1/models currently lists three GPT-6 models, and this PR's parser gives:

model id input modalities sampling options
gpt-6-astra text | text+image no
gpt-6-luna text yes
gpt-6-sol text yes

Luna and Sol end up where Astra was before this PR. With text-only input, a client that requires text + image filters them out. And since neither matches REASONING_MODEL_ID_PATTERN, both advertise temperature/top_p/logprobs/top_logprobs.

According to OpenAI's latest-model guide, both of those are wrong:

"All latest OpenAI models support text and image input, text output, multilingual capabilities, and vision."

"GPT-6 Astra does not support the none reasoning effort" … "GPT-6 Sol and Luna do."

All three are reasoning models with text + image input. The PR handles Astra correctly, including leaving out sampling options since Astra can't use reasoning_effort: none. Luna and Sol need the same classification.

The PR description calls the root cause "model-ID patterning that did not include GPT-6 Astra in multimodal/reasoning capability paths." I'd describe it as the patterning not including GPT-6 at all. Astra is just the one that was reported.

Suggestion: a generation-level constant shared by isReasoningModel() and the modality helper:

/**
 * Regular expression matching the model IDs of the GPT-6 generation.
 *
 * @since n.e.x.t
 *
 * @var string
 */
private const GPT6_MODEL_ID_PATTERN = '/^gpt-6-(?:astra|luna|sol)(?:-|$)/';

With this applied locally, all three resolve to text | text+image, and Astra still has no sampling options.

Open question. The docs say Sol and Luna support reasoning_effort: none. They don't say either one defaults to it. EFFORT_NONE_DEFAULT_MODEL_ID_PATTERN is specifically about the default ("use reasoning effort none by default"), so I can't tell whether Sol and Luna belong there, and that's what decides whether they advertise sampling options. It would be good to confirm this before merge. It's the only claim in this comment I don't have a source for.

Also on this line. This pattern is exact-anchored ($), while REASONING_MODEL_ID_PATTERN on line 48, also edited in this PR, is prefix-anchored ((?:-|$)). Line 48 treats any gpt-6-astra-* as a reasoning model; line 424 only treats the bare and dated IDs as Astra. No suffixed Astra IDs exist today, so nothing breaks yet, but the two patterns should agree on what the family is.

The existing test data suggests the prefix form: gpt-5-mini, gpt-5.4-mini, gpt-5-pro, gpt-5.2-pro, gpt-5.4-nano, gpt-5.2-codex, gpt-5-chat-latest, and gpt-5.4-mini-2026-03-17. That last one is a variant plus a date, and an exact anchor like this one wouldn't match it.

Related: gpt-6-astra-2026-09-01 isn't in the live listing, so the dated-snapshot handling here (and its test case on line 99) is speculative for now. It does no harm, but the date group on line 48 is redundant either way (see my comment there).

If you relax this anchor, please also look at my comment on line 352. That change needs to go in at the same time.

}

/**
* Checks whether an OpenAI text generation model is a reasoning model.
*
* Reasoning model families include codex-mini-latest, versioned GPT-5 models (e.g. `gpt-5`,
* `gpt-5.5`), and the verified o1, o3, and o4 families. GPT-5 chat aliases are non-reasoning
* models and are handled separately; see {@see self::isNonReasoningChatModel()} and
* `gpt-5.5`), GPT-6 Astra, and the verified o1, o3, and o4 families. GPT-5 chat aliases are
* non-reasoning models and are handled separately; see {@see self::isNonReasoningChatModel()} and
* {@see self::supportsSamplingOptions()}.
*
* @since 1.1.0
Expand Down
56 changes: 56 additions & 0 deletions tests/Metadata/OpenAiModelMetadataDirectoryTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,8 @@ public static function samplingOptionSupportProvider(): array
'gpt-5.2-chat-latest (live API rejected temperature)' => ['gpt-5.2-chat-latest', false],
'fine-tuned gpt-5.2' => ['ft:gpt-5.2:example-org:example-model', true],
'fine-tuned gpt-5.2-codex' => ['ft:gpt-5.2-codex:example-org:example-model', false],
'gpt-6-astra (reasoning always enabled)' => ['gpt-6-astra', false],
'gpt-6-astra dated snapshot (reasoning always enabled)' => ['gpt-6-astra-2026-09-01', false],
'codex-mini-latest (reasoning always enabled)' => ['codex-mini-latest', false],
'o3 (reasoning always enabled)' => ['o3', false],
'o4-mini (reasoning always enabled)' => ['o4-mini', false],
Expand All @@ -103,6 +105,33 @@ public static function samplingOptionSupportProvider(): array
];
}

/**
* Tests that GPT-6 Astra advertises text and image input only.
*/
public function testGpt6AstraInputModalities(): void
{
$modelMetadata = $this->parseSingleModelMetadata('gpt-6-astra');
$inputModalities = $this->getSupportedOptionValues($modelMetadata, OptionEnum::inputModalities());

$modalityCombinations = array_map(
static function (array $modalities): string {
return implode(
'+',
array_map(
static function ($modality): string {
return $modality->value;
},
$modalities
)
);
},
$inputModalities
);
sort($modalityCombinations);

$this->assertSame(['text', 'text+image'], $modalityCombinations);
}

/**
* Tests that non-sampling base options remain supported for reasoning models.
*/
Expand Down Expand Up @@ -166,4 +195,31 @@ static function (SupportedOption $supportedOption): string {
$modelMetadata->getSupportedOptions()
);
}

/**
* Returns the values for the given supported option.
*
* @param ModelMetadata $modelMetadata The model metadata.
* @param OptionEnum $optionName The option name.
* @return array<mixed> The option values.
*/
private function getSupportedOptionValues(ModelMetadata $modelMetadata, OptionEnum $optionName): array
{
foreach ($modelMetadata->getSupportedOptions() as $supportedOption) {
if ($supportedOption->getName()->value !== $optionName->value) {
continue;
}

$reflection = new \ReflectionObject($supportedOption);
foreach ($reflection->getProperties() as $property) {
$property->setAccessible(true);
$value = $property->getValue($supportedOption);
if (is_array($value)) {
return $value;
}
}
}

$this->fail(sprintf('Supported option "%s" was not found.', $optionName->value));
}
}