feat: add asr with gemini models #787
Conversation
📝 WalkthroughWalkthroughAdds Gemini 2.5 ASR support in daras_ai_v2/asr.py by introducing three models (flash_lite, flash, pro), defining GEMINI_SUPPORTED languages, mapping model IDs, wiring supported languages, and adding a Gemini branch in run_asr that builds a language-aware prompt and calls call_gemini_api with audio fileData, temperature 0.0, and max_output_tokens 16384. In daras_ai_v2/language_model.py, renames _call_gemini_api to call_gemini_api and updates call sites. In scripts/init_llm_pricing.py, seeds pricing entries for multiple Gemini 2.5/2.0/1.5 models under Google and google/ prefixed IDs. Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
Tip 🔌 Remote MCP (Model Context Protocol) integration is now available!Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats. ✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
daras_ai_v2/language_model.py (1)
2169-2216: Record exact token usage from Vertex instead of char counts.Vertex returns usageMetadata; using it avoids billing drift. Fallback to char lengths if absent.
Apply:
@@ - raise_for_status(r) - ret = "".join( - parts[0]["text"] - for msg in r.json()["candidates"] - if (parts := msg.get("content", {}).get("parts")) - ) + raise_for_status(r) + out = r.json() + ret = "".join( + parts[0]["text"] + for msg in out.get("candidates", []) + if (parts := msg.get("content", {}).get("parts")) + ) @@ - record_cost_auto( - model=model_id, - sku=ModelSku.llm_prompt, - quantity=sum( - len(part.get("text") or "") for item in contents for part in item["parts"] - ), - ) - record_cost_auto( - model=model_id, - sku=ModelSku.llm_completion, - quantity=len(ret), - ) + usage = out.get("usageMetadata") or {} + prompt_tokens = usage.get("promptTokenCount") + completion_tokens = usage.get("candidatesTokenCount") + if prompt_tokens is None: + prompt_tokens = sum(len(part.get("text") or "") for item in contents for part in item["parts"]) + if completion_tokens is None: + completion_tokens = len(ret) + record_cost_auto(model=model_id, sku=ModelSku.llm_prompt, quantity=prompt_tokens) + record_cost_auto(model=model_id, sku=ModelSku.llm_completion, quantity=completion_tokens)
🧹 Nitpick comments (2)
scripts/init_llm_pricing.py (1)
653-699: Avoid duplication between unprefixed andgoogle/-prefixed entries.You’re mirroring rows for both Vertex (unprefixed) and OpenAI-compatible (
google/…) IDs. Consider a small helper to seed both variants together to reduce drift.Apply this refactor:
@@ - # duplicate: because model_id is prefixed with "google/" with the OpenAI-compatible API - llm_pricing_create( - model_id="google/gemini-2.5-flash-lite", - model_name=LargeLanguageModels.gemini_2_5_flash_lite.name, - unit_cost_input=0.1, - unit_cost_output=0.4, - unit_quantity=10**6, - provider=ModelProvider.google, - pricing_url="https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-flash-lite", - ) + # duplicate: because model_id is prefixed with "google/" with the OpenAI-compatible API + for mid, name, cin, cout, url in [ + ("gemini-2.5-flash-lite", LargeLanguageModels.gemini_2_5_flash_lite.name, 0.1, 0.4, "https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-flash-lite"), + ("gemini-2.5-pro", LargeLanguageModels.gemini_2_5_pro.name, 1.25, 10, "https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-pro"), + ("gemini-2.5-flash", LargeLanguageModels.gemini_2_5_flash.name, 0.30, 2.5, "https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-flash"), + ]: + for variant in (mid, f"google/{mid}"): + llm_pricing_create( + model_id=variant, + model_name=name, + unit_cost_input=cin, + unit_cost_output=cout, + unit_quantity=10**6, + provider=ModelProvider.google, + pricing_url=url, + ) @@ - llm_pricing_create( - model_id="google/gemini-2.5-pro-preview-03-25", - model_name=LargeLanguageModels.gemini_2_5_pro_preview.name, - unit_cost_input=1.25, # actually 2.5 when len(input) >= 200K - unit_cost_output=10, # actually 15 when len(input) >= 200K - unit_quantity=10**6, - provider=ModelProvider.google, - pricing_url="https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-pro-preview", - ) + llm_pricing_create( + model_id="google/gemini-2.5-pro-preview-03-25", + model_name=LargeLanguageModels.gemini_2_5_pro_preview.name, + unit_cost_input=1.25, # see notes + unit_cost_output=10, # see notes + unit_quantity=10**6, + provider=ModelProvider.google, + pricing_url="https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-pro-preview", + notes="Costs may increase to $2.5 (input) and $15 (output) when input length >= 200K tokens.", + ) @@ - llm_pricing_create( - model_id="google/gemini-2.5-flash", - model_name=LargeLanguageModels.gemini_2_5_flash.name, - unit_cost_input=0.30, - unit_cost_output=2.5, - unit_quantity=10**6, - provider=ModelProvider.google, - pricing_url="https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-flash", - ) - llm_pricing_create( - model_id="google/gemini-2.5-flash-preview-04-17", - model_name=LargeLanguageModels.gemini_2_5_flash_preview.name, - unit_cost_input=0.15, - unit_cost_output=3.5, # thinking cost, non-thinking is 0.6 - unit_quantity=10**6, - provider=ModelProvider.google, - pricing_url="https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-flash", - ) + llm_pricing_create( + model_id="google/gemini-2.5-flash", + model_name=LargeLanguageModels.gemini_2_5_flash.name, + unit_cost_input=0.30, + unit_cost_output=2.5, + unit_quantity=10**6, + provider=ModelProvider.google, + pricing_url="https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-flash", + ) + llm_pricing_create( + model_id="google/gemini-2.5-flash-preview-04-17", + model_name=LargeLanguageModels.gemini_2_5_flash_preview.name, + unit_cost_input=0.15, + unit_cost_output=3.5, # thinking cost; non-thinking can be lower + unit_quantity=10**6, + provider=ModelProvider.google, + pricing_url="https://ai.google.dev/gemini-api/docs/pricing#gemini-2.5-flash", + notes="Preview tier; thinking-enabled outputs may be priced higher than non-thinking.", + )daras_ai_v2/asr.py (1)
113-121: Normalize Hebrew code: use “he” instead of deprecated “iw”.“iw” is obsolete BCP-47; langcodes may not recognize it cleanly. Replace with “he”.
Apply:
- "fr-CH", "gl-ES", "ka-GE", "de-AT", "de-DE", "de-CH", "el-GR", "gu-IN", "iw-IL", "hi-IN", "hu-HU", "is-IS", "id-ID", + "fr-CH", "gl-ES", "ka-GE", "de-AT", "de-DE", "de-CH", "el-GR", "gu-IN", "he-IL", "hi-IN", "hu-HU", "is-IS", "id-ID",And in GEMINI_SUPPORTED:
- "hmn", "hu", "is", "ig", "id", "ga", "it", "ja", "jv", "kn", "kk", "km", "ko", "kri", "ku", "ky", "lo", "la", "lv", + "hmn", "hu", "is", "ig", "id", "ga", "it", "ja", "jv", "kn", "kk", "km", "ko", "kri", "ku", "ky", "lo", "la", "lv", @@ - "pl", "pt", "pa", "ro", "ru", "sm", "gd", "sr", "st", "sn", "sd", "si", "sk", "sl", "so", "es", "su", "sw", "sv", + "pl", "pt", "pa", "ro", "ru", "sm", "gd", "sr", "st", "sn", "sd", "si", "sk", "sl", "so", "es", "su", "sw", "sv",(Replace any “iw” with “he” where present.)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (3)
daras_ai_v2/asr.py(5 hunks)daras_ai_v2/language_model.py(3 hunks)scripts/init_llm_pricing.py(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
daras_ai_v2/asr.py (1)
daras_ai_v2/language_model.py (1)
call_gemini_api(2169-2216)
scripts/init_llm_pricing.py (2)
daras_ai_v2/language_model.py (1)
LargeLanguageModels(82-994)usage_costs/models.py (1)
ModelProvider(53-66)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: Analyze (javascript-typescript)
- GitHub Check: Analyze (actions)
- GitHub Check: Analyze (python)
- GitHub Check: test (3.10.12, 1.8.3)
🔇 Additional comments (9)
scripts/init_llm_pricing.py (4)
626-652: Gemini 2.5 (unprefixed) pricing entries look consistent with Vertex calls.These IDs match the unprefixed model_ids used by call_gemini_api (e.g., "gemini-2.5-pro"), so usage-cost recording from Vertex requests will resolve correctly to these pricing rows.
700-716: 2.0 flash-lite entry aligns with enums; LGTM.
717-735: 1.5 (unprefixed) entries ensure backward-compat billing for Gemini v1.5 paths.Good to keep these since LLMApis.gemini uses unprefixed model_ids.
691-699: Verify Google Gemini 2.5-Flash pricing
The automated check didn’t locate “2.5-flash-preview” or “2.5-flash” entries on the official pricing page (https://ai.google.dev/gemini-api/docs/pricing). Manually confirm the current per-million output-token rates (thinking vs. non-thinking) and updateunit_cost_outputin scripts/init_llm_pricing.py (lines 691–699) as needed.daras_ai_v2/language_model.py (2)
2158-2165: Vision helper now uses call_gemini_api; LGTM.
2121-2128: No references to_call_gemini_apifound; rename verifieddaras_ai_v2/asr.py (3)
284-287: New ASR model enums for Gemini: LGTM.
351-354: Model ID mapping matches Vertex (unprefixed) IDs used by call_gemini_api.
383-386: Language coverage wiring is correct.
| elif selected_model in { | ||
| AsrModels.gemini_2_5_flash_lite, | ||
| AsrModels.gemini_2_5_flash, | ||
| AsrModels.gemini_2_5_pro, | ||
| }: | ||
| from daras_ai_v2.language_model import CHATML_ROLE_USER, call_gemini_api | ||
|
|
||
| if language: | ||
| lobj = langcodes.Language.get(language.strip()) | ||
| prompt = f"Transcribe this audio without translation. The spoken language is {lobj.display_name()}." | ||
| else: | ||
| prompt = "Transcribe this audio." | ||
|
|
||
| return call_gemini_api( | ||
| model_id=asr_model_ids[selected_model], | ||
| contents=[ | ||
| { | ||
| "role": CHATML_ROLE_USER, | ||
| "parts": [ | ||
| { | ||
| "fileData": { | ||
| "fileUri": audio_url, | ||
| "mimeType": "audio/wav", | ||
| } | ||
| }, | ||
| {"text": prompt}, | ||
| ], | ||
| } | ||
| ], | ||
| max_output_tokens=16384, | ||
| temperature=0.0, | ||
| ) |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Vertex fileUri likely needs a GCS URI; also add translation support.
- Vertex fileData.fileUri usually expects a GCS URI. Use gs_url_to_uri(audio_url) for consistency with image handling.
- Honor speech_translation_target by adjusting the prompt.
Apply:
@@
- elif selected_model in {
+ elif selected_model in {
AsrModels.gemini_2_5_flash_lite,
AsrModels.gemini_2_5_flash,
AsrModels.gemini_2_5_pro,
}:
from daras_ai_v2.language_model import CHATML_ROLE_USER, call_gemini_api
- if language:
- lobj = langcodes.Language.get(language.strip())
- prompt = f"Transcribe this audio without translation. The spoken language is {lobj.display_name()}."
- else:
- prompt = "Transcribe this audio."
+ if language:
+ lobj = langcodes.Language.get(language.strip())
+ prompt = f"Transcribe this audio without translation. The spoken language is {lobj.display_name()}."
+ else:
+ prompt = "Transcribe this audio."
+ if speech_translation_target:
+ tgt = langcodes.Language.get(speech_translation_target.strip())
+ prompt = f"Transcribe this audio and then translate the transcript into {tgt.display_name()}. Return only the translated text."
return call_gemini_api(
model_id=asr_model_ids[selected_model],
contents=[
{
"role": CHATML_ROLE_USER,
"parts": [
{
"fileData": {
- "fileUri": audio_url,
+ "fileUri": gs_url_to_uri(audio_url),
"mimeType": "audio/wav",
}
},
{"text": prompt},
],
}
],
max_output_tokens=16384,
temperature=0.0,
)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| elif selected_model in { | |
| AsrModels.gemini_2_5_flash_lite, | |
| AsrModels.gemini_2_5_flash, | |
| AsrModels.gemini_2_5_pro, | |
| }: | |
| from daras_ai_v2.language_model import CHATML_ROLE_USER, call_gemini_api | |
| if language: | |
| lobj = langcodes.Language.get(language.strip()) | |
| prompt = f"Transcribe this audio without translation. The spoken language is {lobj.display_name()}." | |
| else: | |
| prompt = "Transcribe this audio." | |
| return call_gemini_api( | |
| model_id=asr_model_ids[selected_model], | |
| contents=[ | |
| { | |
| "role": CHATML_ROLE_USER, | |
| "parts": [ | |
| { | |
| "fileData": { | |
| "fileUri": audio_url, | |
| "mimeType": "audio/wav", | |
| } | |
| }, | |
| {"text": prompt}, | |
| ], | |
| } | |
| ], | |
| max_output_tokens=16384, | |
| temperature=0.0, | |
| ) | |
| elif selected_model in { | |
| AsrModels.gemini_2_5_flash_lite, | |
| AsrModels.gemini_2_5_flash, | |
| AsrModels.gemini_2_5_pro, | |
| }: | |
| from daras_ai_v2.language_model import CHATML_ROLE_USER, call_gemini_api | |
| if language: | |
| lobj = langcodes.Language.get(language.strip()) | |
| prompt = f"Transcribe this audio without translation. The spoken language is {lobj.display_name()}." | |
| else: | |
| prompt = "Transcribe this audio." | |
| if speech_translation_target: | |
| tgt = langcodes.Language.get(speech_translation_target.strip()) | |
| prompt = f"Transcribe this audio and then translate the transcript into {tgt.display_name()}. Return only the translated text." | |
| return call_gemini_api( | |
| model_id=asr_model_ids[selected_model], | |
| contents=[ | |
| { | |
| "role": CHATML_ROLE_USER, | |
| "parts": [ | |
| { | |
| "fileData": { | |
| "fileUri": gs_url_to_uri(audio_url), | |
| "mimeType": "audio/wav", | |
| } | |
| }, | |
| {"text": prompt}, | |
| ], | |
| } | |
| ], | |
| max_output_tokens=16384, | |
| temperature=0.0, | |
| ) |
🤖 Prompt for AI Agents
daras_ai_v2/asr.py around lines 1305 to 1336: Vertex's fileData.fileUri should
be a GCS URI and the prompt must honor speech_translation_target; replace
audio_url with gs_url_to_uri(audio_url) when building fileUri (import
gs_url_to_uri if not already imported) and adjust the prompt construction to
include translation when speech_translation_target is set (e.g., if
speech_translation_target: prompt should instruct to translate the audio to that
language, otherwise keep "Transcribe this audio" or "Transcribe this audio
without translation" depending on current behavior); ensure the call_gemini_api
contents use the converted GCS URI and that speech_translation_target is
referenced safely (strip/validate) when composing the prompt.
|
We tried this before for openai models and sometimes the transcription came out as a response to the question asked in the audio file instead of the transcription, hence would not recommend this approach unless they have a explicit transcription api. Also it should use the request.input_prompt from the asr recipe. for evals we can use /copilot directly i think |
Q/A checklist
How to check import time?
You can visualize this using tuna:
To measure import time for a specific library:
To reduce import times, import libraries that take a long time inside the functions that use them instead of at the top of the file:
Legal Boilerplate
Look, I get it. The entity doing business as “Gooey.AI” and/or “Dara.network” was incorporated in the State of Delaware in 2020 as Dara Network Inc. and is gonna need some rights from me in order to utilize my contributions in this PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Dara Network Inc can use, modify, copy, and redistribute my contributions, under its choice of terms.