Skip to content

feat(gemini): add 3.7 Flash with Grok 4.6 and Gemini evals - #238

Merged
teamchong merged 1 commit into
mainfrom
eval/grok-4.6-high
Aug 18, 2026
Merged

feat(gemini): add 3.7 Flash with Grok 4.6 and Gemini evals#238
teamchong merged 1 commit into
mainfrom
eval/grok-4.6-high

Conversation

@teamchong

Copy link
Copy Markdown
Owner

Problem

Gemini 2.5/3.x rejects replayed multi-turn tool calls when a preceding functionCall loses its thoughtSignature. Long sessions also fail to compact reliably: signed turns become opaque barriers, history collapse can stop at latestPlainUser, and oversized image history can abort collapse instead of fitting the image budget. We also lacked current Grok 4.6 and Gemini 3.7 evaluation receipts for these paths.

Mechanism / Fix

  • Add a measured Gemini 3.7 Flash profile using the existing 1568x728 image geometry and 1,078-token image cost.
  • Preserve signed Gemini turns during replay and stamp signatures onto unsigned sibling calls.
  • Normalize Google turn roles on exits and passthrough paths.
  • Let history collapse traverse prior compressed tool results and closed signed turns while keeping the live tail native.
  • Clamp collapsed history to maxImages instead of abandoning compression.
  • Add Grok 4.6 and Gemini 3.7 evaluation runners, receipts, and supporting model/profile updates.

Before vs After

// Before: a replayed sibling call could lose Gemini's required signature.
const plan = await planGoogleHistory(contents, modelName, options);
// Result: 400 errors, or no collapse when signed history blocked traversal.

// After: preserve native signed turns and stamp unsigned siblings.
const normalized = stampGeminiThoughtSignatures(
  normalizeGoogleTurnRoles(contents),
);
// Result: valid replay plus bounded history-image compaction.

Eval Results

Grok 4.6

MODEL=grok-4.6 through the Codex Responses provider
quality: 100/100
novel-arithmetic: 100/100
verbatim-hex: 100/100
gist recall: 17/18

The live grok-4.6 profile resolved with high reasoning effort. Dense hex recall remained 0.4 on the shared corpus, matching the prior Grok 4.5 result.

Gemini 3.7 Flash

MODEL=gemini-3.7-flash LIVE=1

The Gemini runners were added and exercised, but the configured Cloudflare AI Gateway catalog did not expose gemini-3.7-flash as a Google AI Studio model. The similarly named Workers AI model is a GLM text-only endpoint, so it cannot provide a valid image-model receipt. Existing Gemini 3.6 receipts remain unchanged.

Verify

pnpm test
Test Files 72 passed (72)
Tests 1167 passed (1167)

pnpm typecheck
exit 0
  • Rebased on current main
  • pnpm test and pnpm typecheck pass
  • No raw prompts, credentials, session files, or machine identifiers

@teamchong
teamchong merged commit 8509717 into main Aug 18, 2026
5 of 6 checks passed
@teamchong
teamchong deleted the eval/grok-4.6-high branch August 18, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant