I can not use google search retrieval feature

[ensia96]

Description of the bug:

Code

import { Injectable } from "@nestjs/common";
import config from "config";
import {
  DynamicRetrievalMode,
  GoogleGenerativeAI,
  GoogleGenerativeAIError,
} from "@google/generative-ai";
import { AI_CONSTANT } from "../ai/constant";
import { AIType } from "../ai/type";
import { AIVO } from "../ai/vo";
import { CommonVO } from "../common/vo";

export const GEMINI = Symbol("GEMINI");

@Injectable()
export class Gemini {
  API_KEY: string;
  api: GoogleGenerativeAI;

  constructor() {
    this.API_KEY = config.get<string>("gemini.apiKey");
    this.api = new GoogleGenerativeAI(this.API_KEY);
  }

  async generateAnswer(messages: AIType.Message[]) {
    try {
      const [message, ...conversation] = messages.reverse();
      const [systemInstruction, ...history] = conversation.reverse();
      const answer = await this.api
        .getGenerativeModel({
          model: AI_CONSTANT.MODEL.GEMINI,
          systemInstruction: systemInstruction.content,
          /*
          tools: [
            {
              googleSearchRetrieval: {
                dynamicRetrievalConfig: {
                  mode: DynamicRetrievalMode.MODE_DYNAMIC,
                },
              },
            },
          ],
          */
        })
        .startChat({ history: history.map((message) => new AIVO.Message(message).gemini) })
        .sendMessage(message.content);
      console.log("answer :", answer);
      return {
        id: new CommonVO.UUID().value,
        createdAt: new Date(),
        fromAI: answer.response.text(),
        model: AI_CONSTANT.MODEL.GEMINI,
        // model: answer.response.modelVersion,
        raw: answer,
      };
    } catch (error) {
      if (error instanceof GoogleGenerativeAIError) {
        console.error("error :", error);
        return {
          code: error.name,
          message: error.message,
          statusCode: (error as unknown as { status: number }).status,
          // statusCode: error.status,
        } as AIType.Error;
      }
      throw error;
    }
  }
}

1. I can't use the Google Search Retrieval.

• It seems like your API and library source code are in conflict.

error : GoogleGenerativeAIFetchError: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent: [400 Bad Request] Unable to submit request because Please use google_search field instead of google_search_retrieval field.. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini
    at handleResponseNotOk (< project >/node_modules/@google/generative-ai/dist/index.js:434:11)
    at processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async makeRequest (< project >/node_modules/@google/generative-ai/dist/index.js:403:9)
    at async generateContent (< project >/node_modules/@google/generative-ai/dist/index.js:867:22)
    at async ChatSession.sendMessage (< project >/node_modules/@google/generative-ai/dist/index.js:1205:9) {
  status: 400,
  statusText: 'Bad Request',
  errorDetails: undefined
}

• So, I commented code.

          /*
          tools: [
            {
              googleSearchRetrieval: {
                dynamicRetrievalConfig: {
                  mode: DynamicRetrievalMode.MODE_DYNAMIC,
                },
              },
            },
          ],
          */

2. I can't use response.modelVersion.

• It seems like response object has modelVersion. (below is test result without google search retrieval tool)

answer : {
  response: {
    candidates: [ [Object] ],
    usageMetadata: {
      promptTokenCount: 172,
      candidatesTokenCount: 85,
      totalTokenCount: 257,
      promptTokensDetails: [Array],
      candidatesTokensDetails: [Array]
    },
    modelVersion: 'gemini-2.0-flash',
    text: [Function (anonymous)],
    functionCall: [Function (anonymous)],
    functionCalls: [Function (anonymous)]
  }
}

• But, I got an error.

Property 'modelVersion' does not exist on type 'EnhancedGenerateContentResponse'. (tsserver 2339)

• So, I commented and replaced with constant.

        model: AI_CONSTANT.MODEL.GEMINI,
        // model: answer.response.modelVersion,

3. I can't use error status code.

• It seems like error object has modelVersion.

error : GoogleGenerativeAIFetchError: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent: [400 Bad Request] Unable to submit request because Please use google_search field instead of google_search_retrieval field.. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini
    at handleResponseNotOk (< project >/node_modules/@google/generative-ai/dist/index.js:434:11)
    at processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async makeRequest (< project >/node_modules/@google/generative-ai/dist/index.js:403:9)
    at async generateContent (< project >/node_modules/@google/generative-ai/dist/index.js:867:22)
    at async ChatSession.sendMessage (< project >/node_modules/@google/generative-ai/dist/index.js:1205:9) {
  status: 400,
  statusText: 'Bad Request',
  errorDetails: undefined
}

• But, I got an error.

Property 'status' does not exist on type 'GoogleGenerativeAIError'. (tsserver 2339)

• So, I commented and add code for assertion.

          statusCode: (error as unknown as { status: number }).status,
          // statusCode: error.status,

Actual vs expected behavior:

expected

1. I can use the Google Search Retrieval.
2. I can use response.modelVersion.
3. I can use error status code.

actual

1. I can't use the Google Search Retrieval. I got error response.
2. I can't use response.modelVersion. I got typescript error.
3. I can't use error status code. I got typescript error.

Any other information you'd like to share?

No response

[imperorrp]

Yeah there's a similar issue with the legacy Python SDK when it comes to Gemini 2 models, at least for the Google Search Retrieval. I found this issue there talking about the same problem: google-gemini/deprecated-generative-ai-python#667

There's a gist linked there that shows how to use search with Gemini 2 models with the new Python SDK though.

EDIT:
Okay apparently there's an example for using "Search as a tool" for Gemini 2 models in samples/ on this repo- https://github.com/google-gemini/generative-ai-js/blob/main/samples/search_grounding.js

But although mentioned in the docs, there's no example for this shown there yet- https://ai.google.dev/gemini-api/docs/grounding?lang=node#search-suggestions

[ensia96]

@imperorrp

I checked the sample code in the link you provided, and there wasn't much difference in the implementation.

The issue in the link you provided seems to have been resolved by replacing it with the new library.

Hmm.. I guess I'll have to submit a pull request to fix this..

Thanks for sharing the information!

[ensia96]

Oops... there is an issue mentioning the js version of genai.

If anyone is having the same problem as me, please refer to this issue.

[imperorrp]

@imperorrp

I checked the sample code in the link you provided, and there wasn't much difference in the implementation.

The issue in the link you provided seems to have been resolved by replacing it with the new library.

Hmm.. I guess I'll have to submit a pull request to fix this..

Thanks for sharing the information!

Not necessary to use the new js library though. Did you check the samples/ examples on this repo (https://github.com/google-gemini/generative-ai-js/blob/main/samples/search_grounding.js#L55-L77)? Looks like replacing googleSearchRetrieval with googleSearch should work for Gemini 2 models while still using this SDK. This example just isn't in the official docs site.