feat(providers/google): improve cachedContent, expose rich token metadata and pass mediaResolution #6256
+492
−27
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This pull request improves support for some of the features of Google's Generative AI and Vertex AI. See the list below.
tools
,toolConfig
, andsystemInstruction
whencachedContent
is provided. This aligns with Google's requirements for utilizing cached content effectively.providerMetadata
now includes a comprehensive set of token counts and details from Google's API response for both streaming and non-streaming operations:cachedContentTokenCount
thoughtsTokenCount
(defaults to 0 ifthinkingBudget
is explicitly set to 0)promptTokensDetails
(array, with modality and token count)cacheTokensDetails
(array, with modality and token count)candidatesTokensDetails
(array, with modality and token count)toolUsePromptTokensDetails
(array, with modality and token count)mediaResolution
option has been added toGoogleGenerativeAISettings
and is now passed to the Google API, allowing users to specify media resolution for vision capabilities.cachedContent
parameters and the accurate extraction and exposure of all new token metadata fields.Background
Previously, support for Google's caching features and detailed token reporting in the Vercel AI SDK was limited. For
cachedContent
to function correctly, specific parameters must be omitted from the request. This PR implements this as conditional logic. It also exposes some of the additional Google-specific token metadata by modality, which provides users with deeper insights into their token consumption, and model interactions. ThemediaResolution
setting allows images uploaded to the models to be interpreted in a higher, or lower resolution.Verification
cachedContent
is used.providerMetadata
fordoGenerate
anddoStream
.@ai-sdk/google
and Vertex provider packages locally into an active project to confirm end-to-end functionality ofcachedContent
improvements and metadata exposure.Tasks
cachedContent
.providerMetadata
(e.g.,cachedContentTokenCount
,thoughtsTokenCount
,promptTokensDetails
, etc.).mediaResolution
setting.minor
changeset for relevant packages has been added (for bug fixes / features - runpnpm changeset
in the project root)pnpm prettier-fix
in the project root)Future Work
cachedContents
. This would simplify cache creation for Vercel AI SDK users (see Google Vertex AI Docs for CachedContents).GenerateContentResponse
inproviderMetadata
(see Google Vertex AI Docs for GenerateContentResponse).Related Issues