[WIP] feat(vertexai): Gemini multimodal output

[dlarocque]

No description provided.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 4f7f1ec

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[github-actions]

Vertex AI Mock Responses Check ⚠️

A newer major version of the mock responses for Vertex AI unit tests is available. update_vertexai_responses.sh should be updated to clone the latest version of the responses: v10.0

[google-oss-bot]

Size Report ¹

Affected Products

• @firebase/vertexai

  Type	Base (4e0f630)	Merge (3a54191)	Diff
  browser	34.7 kB	35.9 kB	+1.18 kB (+3.4%)
  main	35.7 kB	36.9 kB	+1.20 kB (+3.4%)
  module	34.7 kB	35.9 kB	+1.18 kB (+3.4%)

• firebase

  Type	Base (4e0f630)	Merge (3a54191)	Diff
  firebase-vertexai.js	28.3 kB	29.2 kB	+958 B (+3.4%)

Test Logs

• Base (4e0f630): https://github.com/firebase/firebase-js-sdk/actions/runs/14364226510
• Merge (3a54191): https://github.com/firebase/firebase-js-sdk/actions/runs/14390892637

1. https://storage.googleapis.com/firebase-sdk-metric-reports/lZxgwrhmwp.html

[google-oss-bot]

Size Analysis Report ¹

Affected Products

• @firebase/vertexai

  • ChatSession

    Size

    Type	Base (4e0f630)	Merge (3a54191)	Diff
    size	15.8 kB	16.7 kB	+852 B (+5.4%)
    size-with-ext-deps	34.7 kB	35.6 kB	+858 B (+2.5%)

    Dependency

    Type	Base (4e0f630)	Merge (3a54191)	Diff
    functions	21 dependencies addHelpers aggregateResponses assignRoleToPartsAndValidateSendMessageRequest constructRequest createEnhancedContentResponse formatBlockErrorMessage formatNewContent generateContent generateContentStream generateResponseSequence getClientHeaders getFunctionCalls getHeaders getResponsePromise getResponseStream getText hadBadFinishReason makeRequest processStream registerVertex validateChatHistory	22 dependencies addHelpers aggregateResponses assignRoleToPartsAndValidateSendMessageRequest constructRequest createEnhancedContentResponse formatBlockErrorMessage formatNewContent generateContent generateContentStream generateResponseSequence getClientHeaders getData getFunctionCalls getHeaders getResponsePromise getResponseStream getText hadBadFinishReason makeRequest processStream registerVertex validateChatHistory	+ getData

  • GenerativeModel

    Size

    Type	Base (4e0f630)	Merge (3a54191)	Diff
    size	18.9 kB	19.8 kB	+852 B (+4.5%)
    size-with-ext-deps	37.9 kB	38.7 kB	+858 B (+2.3%)

    Dependency

    Type	Base (4e0f630)	Merge (3a54191)	Diff
    functions	24 dependencies addHelpers aggregateResponses assignRoleToPartsAndValidateSendMessageRequest constructRequest countTokens createEnhancedContentResponse formatBlockErrorMessage formatGenerateContentInput formatNewContent formatSystemInstruction generateContent generateContentStream generateResponseSequence getClientHeaders getFunctionCalls getHeaders getResponsePromise getResponseStream getText hadBadFinishReason makeRequest processStream registerVertex validateChatHistory	25 dependencies addHelpers aggregateResponses assignRoleToPartsAndValidateSendMessageRequest constructRequest countTokens createEnhancedContentResponse formatBlockErrorMessage formatGenerateContentInput formatNewContent formatSystemInstruction generateContent generateContentStream generateResponseSequence getClientHeaders getData getFunctionCalls getHeaders getResponsePromise getResponseStream getText hadBadFinishReason makeRequest processStream registerVertex validateChatHistory	+ getData

  • ResponseModality

    Size

    Type	Base (4e0f630)	Merge (3a54191)	Diff
    size	?	3.46 kB	? (?)
    size-with-ext-deps	?	21.4 kB	? (?)

    Dependency

    Type	Base (4e0f630)	Merge (3a54191)	Diff
    functions	?	registerVertex	?
    classes	?	VertexAIService	?
    variables	?	19 dependencies BlockReason DEFAULT_LOCATION FinishReason FunctionCallingMode HarmBlockMethod HarmBlockThreshold HarmCategory HarmProbability HarmSeverity ImagenAspectRatio ImagenPersonFilterLevel ImagenSafetyFilterLevel Modality ResponseModality SchemaType Task VERTEX_TYPE name version	?
    enums	?		?

    External Dependency

    Module	Base (4e0f630)	Merge (3a54191)	Diff
    @firebase/app	?	_registerComponent registerVersion	?
    @firebase/component	?	Component	?
    @firebase/logger	?	Logger	?

  • getGenerativeModel

    Size

    Type	Base (4e0f630)	Merge (3a54191)	Diff
    size	19.1 kB	19.9 kB	+852 B (+4.5%)
    size-with-ext-deps	38.1 kB	38.9 kB	+858 B (+2.3%)

    Dependency

    Type	Base (4e0f630)	Merge (3a54191)	Diff
    functions	25 dependencies addHelpers aggregateResponses assignRoleToPartsAndValidateSendMessageRequest constructRequest countTokens createEnhancedContentResponse formatBlockErrorMessage formatGenerateContentInput formatNewContent formatSystemInstruction generateContent generateContentStream generateResponseSequence getClientHeaders getFunctionCalls getGenerativeModel getHeaders getResponsePromise getResponseStream getText hadBadFinishReason makeRequest processStream registerVertex validateChatHistory	26 dependencies addHelpers aggregateResponses assignRoleToPartsAndValidateSendMessageRequest constructRequest countTokens createEnhancedContentResponse formatBlockErrorMessage formatGenerateContentInput formatNewContent formatSystemInstruction generateContent generateContentStream generateResponseSequence getClientHeaders getData getFunctionCalls getGenerativeModel getHeaders getResponsePromise getResponseStream getText hadBadFinishReason makeRequest processStream registerVertex validateChatHistory	+ getData

Test Logs

• Base (4e0f630): https://github.com/firebase/firebase-js-sdk/actions/runs/14364226510
• Merge (3a54191): https://github.com/firebase/firebase-js-sdk/actions/runs/14390892637

1. https://storage.googleapis.com/firebase-sdk-metric-reports/4AZW25UCsE.html