Dynamic retrieval score do not work on Gemini 1.5

[zmlee0514]

Description

I used to use this code to set the dynamic_threshold and it works well. However, it can not stop Gemini from using the grounding tool to search anymore, even I only input 'hello' or set the threshold to '1'. Furthermore, the Gemini does also not respond retrieval_metadata, where I could see the google_search_dynamic_retrieval_score in the past.

Then I found the SDK update(#4644) and rewrote it to the code example below, but it still does not work. How could I effectively set the dynamic_threshold now?

vertexai.init(project=self.project_id, location=self.region)
self.generation_config = generative_models.GenerationConfig(
    temperature=self.prompt_setup["parameters"].get("temperature"),
    top_p=self.prompt_setup["parameters"].get("top_p"),
    max_output_tokens=self.prompt_setup["parameters"].get("max_output_token"),
    response_schema=self.prompt_setup["parameters"].get("response_schema"),
    response_mime_type="application/json",
)
self.dynamic_retrieval_config = gapic_tool_types.DynamicRetrievalConfig(
    mode=gapic_tool_types.DynamicRetrievalConfig.Mode.MODE_DYNAMIC,
    dynamic_threshold=self.prompt_setup["parameters"].get("searching_threshold")
)
self.google_search_tool = generative_models.Tool.from_google_search_retrieval(
    generative_models.grounding.GoogleSearchRetrieval()
)
self.google_search_tool._raw_tool.google_search_retrieval.dynamic_retrieval_config = self.dynamic_retrieval_config
self.model = generative_models.GenerativeModel(
    model_name=self.gemini_version,
    generation_config=self.generation_config,
    system_instruction=self.prompt_setup.get("system"),
    tools=[self.google_search_tool],
)

Environment details

• OS type and version: Ubuntu 22.01.1 6.8.0-48-generic
• Python version: 3.10.14
• pip version: pip 24.0
• google-cloud-aiplatform version: 1.75.0

Code example

import vertexai
from vertexai.generative_models import GenerativeModel, Tool, grounding

vertexai.init(project="project-name", location="asia-east1")
dynamic_retrieval_config = grounding.DynamicRetrievalConfig(
    mode=grounding.DynamicRetrievalConfig.Mode.MODE_DYNAMIC,
    dynamic_threshold=1
)
tool = Tool.from_google_search_retrieval(grounding.GoogleSearchRetrieval(dynamic_retrieval_config=dynamic_retrieval_config))
model = GenerativeModel("gemini-1.5-flash-002", tools=[tool])
response = model.generate_content("hello")
print(response)

Response

candidates {
  content {
    role: "model"
    parts {
      text: "Hello!  The word \"hello\" is a common English greeting.  Its meaning is simple: it\'s used to greet someone, whether in person, on the phone, or even to express surprise.  The word first appeared in writing around 1826.  Beyond its basic function as a greeting, \"hello\" has also found its way into popular culture, such as in Adele\'s famous 2015 song of the same name.  There is even a magazine called *HELLO!* focused on celebrity news.\n"
    }
  }
  avg_logprobs: -0.23691466047957138
  finish_reason: STOP
  grounding_metadata {
    web_search_queries: "hello"
    search_entry_point {
      rendered_content: "<style>\n.container {\n  align-items: center;\n  border-radius: 8px;\n  display: flex;\n  font-family: Google Sans, Roboto, sans-serif;\n  font-size: 14px;\n  line-height: 20px;\n  padding: 8px 12px;\n}\n.chip {\n  display: inline-block;\n  border: solid 1px;\n  border-radius: 16px;\n  min-width: 14px;\n  padding: 5px 16px;\n  text-align: center;\n  user-select: none;\n  margin: 0 8px;\n  -webkit-tap-highlight-color: transparent;\n}\n.carousel {\n  overflow: auto;\n  scrollbar-width: none;\n  white-space: nowrap;\n  margin-right: -12px;\n}\n.headline {\n  display: flex;\n  margin-right: 4px;\n}\n.gradient-container {\n  position: relative;\n}\n.gradient {\n  position: absolute;\n  transform: translate(3px, -9px);\n  height: 36px;\n  width: 9px;\n}\n@media (prefers-color-scheme: light) {\n  .container {\n    background-color: #fafafa;\n    box-shadow: 0 0 0 1px #0000000f;\n  }\n  .headline-label {\n    color: #1f1f1f;\n  }\n  .chip {\n    background-color: #ffffff;\n    border-color: #d2d2d2;\n    color: #5e5e5e;\n    text-decoration: none;\n  }\n  .chip:hover {\n    background-color: #f2f2f2;\n  }\n  .chip:focus {\n    background-color: #f2f2f2;\n  }\n  .chip:active {\n    background-color: #d8d8d8;\n    border-color: #b6b6b6;\n  }\n  .logo-dark {\n    display: none;\n  }\n  .gradient {\n    background: linear-gradient(90deg, #fafafa 15%, #fafafa00 100%);\n  }\n}\n@media (prefers-color-scheme: dark) {\n  .container {\n    background-color: #1f1f1f;\n    box-shadow: 0 0 0 1px #ffffff26;\n  }\n  .headline-label {\n    color: #fff;\n  }\n  .chip {\n    background-color: #2c2c2c;\n    border-color: #3c4043;\n    color: #fff;\n    text-decoration: none;\n  }\n  .chip:hover {\n    background-color: #353536;\n  }\n  .chip:focus {\n    background-color: #353536;\n  }\n  .chip:active {\n    background-color: #464849;\n    border-color: #53575b;\n  }\n  .logo-light {\n    display: none;\n  }\n  .gradient {\n    background: linear-gradient(90deg, #1f1f1f 15%, #1f1f1f00 100%);\n  }\n}\n</style>\n<div class=\"container\">\n  <div class=\"headline\">\n    <svg class=\"logo-light\" width=\"18\" height=\"18\" viewBox=\"9 9 35 35\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\">\n      <path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M42.8622 27.0064C42.8622 25.7839 42.7525 24.6084 42.5487 23.4799H26.3109V30.1568H35.5897C35.1821 32.3041 33.9596 34.1222 32.1258 35.3448V39.6864H37.7213C40.9814 36.677 42.8622 32.2571 42.8622 27.0064V27.0064Z\" fill=\"#4285F4\"/>\n      <path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M26.3109 43.8555C30.9659 43.8555 34.8687 42.3195 37.7213 39.6863L32.1258 35.3447C30.5898 36.3792 28.6306 37.0061 26.3109 37.0061C21.8282 37.0061 18.0195 33.9811 16.6559 29.906H10.9194V34.3573C13.7563 39.9841 19.5712 43.8555 26.3109 43.8555V43.8555Z\" fill=\"#34A853\"/>\n      <path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M16.6559 29.8904C16.3111 28.8559 16.1074 27.7588 16.1074 26.6146C16.1074 25.4704 16.3111 24.3733 16.6559 23.3388V18.8875H10.9194C9.74388 21.2072 9.06992 23.8247 9.06992 26.6146C9.06992 29.4045 9.74388 32.022 10.9194 34.3417L15.3864 30.8621L16.6559 29.8904V29.8904Z\" fill=\"#FBBC05\"/>\n      <path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M26.3109 16.2386C28.85 16.2386 31.107 17.1164 32.9095 18.8091L37.8466 13.8719C34.853 11.082 30.9659 9.3736 26.3109 9.3736C19.5712 9.3736 13.7563 13.245 10.9194 18.8875L16.6559 23.3388C18.0195 19.2636 21.8282 16.2386 26.3109 16.2386V16.2386Z\" fill=\"#EA4335\"/>\n    </svg>\n    <svg class=\"logo-dark\" width=\"18\" height=\"18\" viewBox=\"0 0 48 48\" xmlns=\"http://www.w3.org/2000/svg\">\n      <circle cx=\"24\" cy=\"23\" fill=\"#FFF\" r=\"22\"/>\n      <path d=\"M33.76 34.26c2.75-2.56 4.49-6.37 4.49-11.26 0-.89-.08-1.84-.29-3H24.01v5.99h8.03c-.4 2.02-1.5 3.56-3.07 4.56v.75l3.91 2.97h.88z\" fill=\"#4285F4\"/>\n      <path d=\"M15.58 25.77A8.845 8.845 0 0 0 24 31.86c1.92 0 3.62-.46 4.97-1.31l4.79 3.71C31.14 36.7 27.65 38 24 38c-5.93 0-11.01-3.4-13.45-8.36l.17-1.01 4.06-2.85h.8z\" fill=\"#34A853\"/>\n      <path d=\"M15.59 20.21a8.864 8.864 0 0 0 0 5.58l-5.03 3.86c-.98-2-1.53-4.25-1.53-6.64 0-2.39.55-4.64 1.53-6.64l1-.22 3.81 2.98.22 1.08z\" fill=\"#FBBC05\"/>\n      <path d=\"M24 14.14c2.11 0 4.02.75 5.52 1.98l4.36-4.36C31.22 9.43 27.81 8 24 8c-5.93 0-11.01 3.4-13.45 8.36l5.03 3.85A8.86 8.86 0 0 1 24 14.14z\" fill=\"#EA4335\"/>\n    </svg>\n    <div class=\"gradient-container\"><div class=\"gradient\"></div></div>\n  </div>\n  <div class=\"carousel\">\n    <a class=\"chip\" href=\"https://vertexaisearch.cloud.google.com/grounding-api-redirect/AYygrcQ6tHsTx07C-VrZnCXSWKHS9tXRdsX1z6ww3nGQM14O5a2fR7erBXXO0VGwBfLFoIMhBUItiwJsl8RKGGGivGqc_Iko4uVHoDLff5ydOzng7Jyfl19X7waN7t53JHx85_mo2qJRl7Kvi7yhulWVPlXko-YJKwFAOvBzlgR7rssXrR6knEoSh1uY\">hello</a>\n  </div>\n</div>\n"
    }
    grounding_chunks {
      web {
        uri: "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AYygrcQDUaNdhu9yvutmEelZjPPP1uOlRjrR-Q8aBLOKDG4NzTsCsE0SAPlw1uN7dMBzbvVxXKC3REuUOJNfGAYrGfc5lTJUSpHhJ4upA1o9pueNAyx6bMUIyv0r3ZNIu8s="
        title: "wikipedia.org"
      }
    }
    grounding_chunks {
      web {
        uri: "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AYygrcRAuwpGKPC4x4s0UijmqnkwYhxAh6xaMc8rryv5cK4RJJG28nKzd1c-1uCnG4g810Pj9xIeY5jMBLBqqz07BKfckgypb3BYU0HDoclfbkjBXMQkKzW5ayXe"
        title: "hellomagazine.com"
      }
    }
    grounding_supports {
      segment {
        start_index: 170
        end_index: 217
        text: "The word first appeared in writing around 1826."
      }
      grounding_chunk_indices: 0
      confidence_scores: 0.958493412
    }
    grounding_supports {
      segment {
        start_index: 407
        end_index: 435
        text: "* focused on celebrity news."
      }
      grounding_chunk_indices: 1
      confidence_scores: 0.82979691
    }
  }
}
model_version: "gemini-1.5-flash-002"
usage_metadata {
  prompt_token_count: 1
  candidates_token_count: 111
  total_token_count: 112
}

[shiba-hiro]

I’m encountering the same issue.
Based on my testing, setting the location value to "us-central1" seems to make dynamic retrieval work as expected.
However, when specifying "asia-northeast1", grounding is always executed instead.

[shiba-hiro]

Update:

Today, I was able to confirm that dynamic retrieval works as expected when specifying the "asia-northeast1" location.